Jump to content
Sign in to follow this  
Shanheavel

HTTP & gZip encoding

Recommended Posts

Shanheavel

Hello,

I've made program which uses TCP and HTTP protocols for browsing webpages.

TCPStartup()
$IP = TCPNameToIP("google.pl")
$ServerSocket = TCPConnect($IP, 80)
TCPSend($ServerSocket, $Request)
...

Sometimes received packets are gZiped I know that I can remove "Accept-Encoding: gzip,deflate" from request, but I want use it).

I tried lots of Zip UDFs but they don't work.

...

I tested all.

Example:

Received data from server:

HTTP/1.1 200 OK

Date: Sun, 14 Aug 2011 10:38:55 GMT

Server: Apache

Cache-Control: no-cache

Vary: Accept-Encoding

Content-Encoding: gzip

Content-Length: 982

Keep-Alive: timeout=15, max=100

Connection: Keep-Alive

Content-Type: text/plain; charset=UTF-8

‹ ťUMoŰF=G€˙Ă– *ęQÔ‡ÓP&ŔR´·:§\Ś!9"W"w‰ÝĄ(˛čˇF‹ü zń©±ţWwEZ¦0r!¸óńŢěçńdp&#AET] o)Ü)g

[h­Vp2xeÇ<*sdj4qmŻJ)ʱG䏓Á‹WöđeY€”ĂŃxu!PĘcŤ#ó‚®lW)ŤRßwg#’ń„2{´Ôľ?Íóđ8sZ^M«1Ý’(Ó¸ľf/©ÂśHUgčC6‰ŕ%‹˝Rd6ÍGAx©1ÇKFĂŕĚŃÉA‡Ac_{

`“Ôoť-ÁŘXBôČôMQ[[ű ôŻ€Ő5l¸wČ

s

‚Á™jf“vHÇ#e˘DFĄş-ż˘±J=w1)vşT^(s—‚dbć[熑FV00.ÓçÎqQ˛R‚=ĎGŮBVęy˝y;™ś.¬ŕÎuć´I:ŰąEn‘î‘|ÂĽfđ-G7źÍµ?ěŻ1Ů_ý˛żÚ€íVw”Óéb1˙É

žŚ|fż”

« mĄíÎzśóééۉk=ß3Y~Q>Ćň;Č©í¦w4‹Ét:›[AßŮ㹏đ>Śb\„bŰU=”Ůdćj”‡O#mW©JU©JŰ

ŕčŮŢw?wľ\4Ź\ű#Ę6ŔîMt6źÎgş»}ç4NűQ·{uŘ••€ő”Jqv´vG΢ŚFx0~Đ‘öhHĎ2ľEáO'fUś6¸[ŰRhö­Tw(íYëFă48"°CôVQ‘"ň-ÇH…Ƹ4

â±°î6|Ĺą:|Źţ(Î3ůP€¤WľőNÜřg+šXÁ˙-żNĄ€h݉ôSRĄ

Ďq

žóhśH8Ă|\de1îĆEZĽ„ş”)Ż~ÔÓô]÷Íű˘©CŮ Ů_AlôŠ"IDý ľ­h%#qČúDÎ+$űżH¤ŕzÖśŐQÚ%š†|Ű—Z–$ä"FáM‹‘<Ł1/IÄ3.Ľő¤Ş%Yq¦^KÚ çΊݒǔ%ž9B|¬řÉFr˛F-™

hšýµľ9ŮŠűë›~0Ý"­BwőŔ]Ł«'÷aÝ!…őţJ?±)(FŤ†ĘKIeCJ© ˘7źIeoGA‰¬ľ~YőźŚ`4#”$\עżÇ

7Ę7>ś7µ‚5Qş ˘{Ú•SYdP{”e”á’?ęëX÷Q€Ů'OwEë2ĄĚ›ż{ÜÓůÄ

#1őńv¸˙[Ş

ĚČƇ?j·Ý÷ř?|Ž°ň

Get binary body:

$ServerRecv = TCPRecv($ServerSocket, 1024^2)
$HeaderEnd = StringInStr($ServerRecv, @CRLF & @CRLF)
$Body = StringTrimLeft($ServerRecv, $HeaderEnd + 3)
$Binary = StringToBinary($Body)

And I use for example:

_Zlib_UncompressBinary($Binary, BinaryLen($Binary))

Maybe I do something wrong?

Edited by Adrian777

Share this post


Link to post
Share on other sites
guinness

How about using 7-Zip? >>


UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites
Shanheavel

How about using 7-Zip? >>

I've tried it day ago. If I save body of received packet (as string) to "file.txt", rename to "file.zip" and extract (by 7zip) I've "file" and error:

"CRC failed in <filename>. The file is corrupt."

I can rename it again to "file.txt", open in notepad and I've correct packet but this errors interfere with my "browser".

Edited by Adrian777

Share this post


Link to post
Share on other sites
smartee

@Adrian777: Post a runnable snippet of code that demonstrates your problem. Then we can offer something other than guesses and links you've already tried :mellow:

Share this post


Link to post
Share on other sites
guinness

This is what I use to decrypt a compressed file using 7-Zip (must be supported by 7-Zip.) As smartee said please post some reproducible code and/or website that you're having the problem with, because it could be the simplest of errors you've somehow overlooked :mellow:

Func _7Zip_Decrypt($sZipFile, $sDestinationFolder = @ScriptDir, $sPassword = "")
    If FileExists($sZipFile) = 0 Then
        Return SetError(1, 0, 0)
    EndIf
    If StringRight($sDestinationFolder, 1) <> "\" Then
        $sDestinationFolder &= "\"
    EndIf
    If FileExists($sDestinationFolder) = 0 Then
        DirCreate($sDestinationFolder)
    EndIf
    If $sPassword <> "" Then
        $sPassword = "-p" & '"' & $sPassword & '" '
    EndIf
    If FileExists(@ScriptDir & "\" & "7za.exe") = 0 Then
        FileInstall("7za.exe", @ScriptDir & "\" & "7za.exe", 0)
    EndIf
    Return RunWait('7za.exe' & ' x "' & $sZipFile & '" ' & $sPassword & "-y -o" & '"' & $sDestinationFolder & '"', "", @SW_HIDE)
EndFunc   ;==>_7Zip_Decrypt
Edited by guinness

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites
Shanheavel

@guinness

With normal ziped files (compressed by program) 7-Zip returns 0. With my received packet it returns 2.

@My code:

Send.au3:

; --- Config ---
Global Const $IP = "87.98.235.245" ; server IP
Global Const $RequestFile = "request.txt" ; file with request to send
Global Const $RecvFile = "recv.txt" ; file to write received data
; --- End ---
Global $ServerRecv = ""

; --- System ---
TCPStartup()

$ServerSocket = TCPConnect($IP, 80)
If @error Then MsgBox(16, 1, @error)
ConsoleWrite("Connected with " & $IP & @CR)
TCPSend($ServerSocket, FileRead($RequestFile))
ConsoleWrite("Send ... " & @CR)
Sleep(200)
$ServerRecv = TCPRecv($ServerSocket, 1024 ^ 2, 1)
If @error Then MsgBox(16, 2, @error)
If $ServerRecv <> Binary("") Then
    ConsoleWrite("Recv... " & @CR)
    MsgBox(32, StringLen($ServerRecv), $ServerRecv)
    FileWrite($RecvFile, $ServerRecv)
EndIf
TCPCloseSocket($ServerSocket)
TCPShutdown()
; -------

I think it's simple! =)

My request:

POST /ajax/logon.php?t=login HTTP/1.1

Host: www.margonem.pl

User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0.1) Gecko/20100101 Firefox/5.0.1

Accept: text/html, */*

Accept-Language: pl,en-us;q=0.7,en;q=0.3

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-2,utf-8;q=0.7,*;q=0.7

Connection: keep-alive

Content-Type: application/x-www-form-urlencoded; charset=UTF-8

X-Requested-With: XMLHttpRequest

Referer: http://www.margonem.pl/

Content-Length: 26

Cookie: ads=0-1; __utma=1.1328332501.1313174251.1313174151.1313174151.1; __utmb=1.1.10.1313174151; __utmc=1; __utmz=1.1313174151.1.7.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)

Pragma: no-cache

Cache-Control: no-cache

l=grimreaper&p=devilmaycry

This is example login request. After send we should received packet with ziped body. Edited by Adrian777

Share this post


Link to post
Share on other sites
guinness

Please read the Forum rules as the website you specified is for a game >> http://www.autoitscript.com/forum/forum-2/announcement-13-forum-rules/

Edited by guinness

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites
bogQ

probably

http://www.margonem.pl/


TCP server and client - Learning about TCP servers and clients connection
Au3 oIrrlicht - Irrlicht project
Au3impact - Another 3D DLL game engine for autoit. (3impact 3Drad related)



460px-Thief-4-temp-banner.jpg
There are those that believe that the perfect heist lies in the preparation.
Some say that it’s all in the timing, seizing the right opportunity. Others even say it’s the ability to leave no trace behind, be a ghost.

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.