trancexx

Alternative data compression

21 posts in this topic




That's pretty clever trancexx.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

trancexx,

About 50% compression when I tested - clever. :thumbsup:

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

That's a brilliant idea to use PNG compression algorithm for any kind of binary data.

Cool.

Br,

UEZ

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

Can you explain me the idea of the rotation of the image before I try analysing and understand it? I know it works only with value 6 (A 180-degree rotation followed by a horizontal flip (No rotation and a vertical flip))...

Further I can see in many progams that there is an option to set the compression rate also for PNG....

Thanks,

UEZ

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

Can you explain me the idea of the rotation of the image before I try analysing and understand it? I know it works only with value 6 (A 180-degree rotation followed by a horizontal flip (No rotation and a vertical flip))...

Further I can see in many progams that there is an option to set the compression rate also for PNG....

Thanks,

UEZ

Yes, of course I can explain flipping.

If you look at BMP data line by line you will see that first line of data is drawn at the bottom of he pic, and last line is drawn at the top. For example try compressing this data:

$sData = "0x" & _
"000000" & "000000" & "000000" & "000000" & _
"000000" & "000000" & "000000" & "000000" & _
"000000" & "000000" & "000000" & "000000" & _
"FFFFFF" & "FFFFFF" & "FFFFFF" & "FFFFFF" & _
"FFFFFF" & "FFFFFF" & "FFFFFF" & "FFFFFF" & _
"FFFFFF" & "FFFFFF" & "FFFFFF" & "FFFFFF"

... I've organized it so that every line is exactly a line inside BMP and every chunk is one pixel represented by RGB value. Considering visually RGB out of (0x000000) is black color and 0xFFFFFF is white one could expect that the image will be black up and white down. Something like (every word is one pixel):

BLACK BLACK BLACK BLACK
BLACK BLACK BLACK BLACK
BLACK BLACK BLACK BLACK
WHITE  WHITE  WHITE  WHITE 
WHITE  WHITE  WHITE  WHITE 
WHITE  WHITE  WHITE  WHITE

But if you look at the picture you will see it's other way around - what's up that's down and vice versa.

First line is black-ish and that's the padding I was commenting on in the code. It holds the original data size information too. Ignore it here.

GdipCreateBitmapFromScan0 creates top to down image and if I want correct data I must flip it to be down to top otherwise it's all mixed up.

I didn't try setting PNG compression level, I know about JPG, but if you know how's that done post it.


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

Ah yes, I can remember it again that the bitmap data is set from bottom to top - now it makes sense.

Regarding compression settings - I don't now yet how to set any compression value for PNG but e.g. in IrfanView you can set it (0-9). My assumption was that it must be possible somehow. Maybe it uses any 3rd party dll for that.

Br,

UEZ


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

PNG compression does not seem to be possible by GDI+.

http://stackoverflow.com/questions/4418454/c-seeking-png-compression-algorithm-library

Share this post


Link to post
Share on other sites

According to MSDN:

GDI+ has built-in encoders and decoders that support the following file types:

  • BMP
  • GIF
  • JPEG
  • PNG
  • TIFF
and this is easy to show it in trancexx example. I wrote on console all supported encoders using GdipGetImageEncoders and I get this:

*.BMP;*.DIB;*.RLE

*.JPG;*.JPEG;*.JPE;*.JFIF

*.GIF

*.TIF;*.TIFF

*.PNG


When the words fail... music speaks

Share this post


Link to post
Share on other sites

@Andreik: of course PNG is supported but it has only a default setting for PNG compression whereas you can set compression strength for JPG in GDI+ but not for PNG format.

Br,

UEZ


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

Would it be possible to process the data directly from variables instead of saving then loading a file? This would be much faster for multiple compression operations in a row.

Share this post


Link to post
Share on other sites

#14 ·  Posted (edited)

Would it be possible to process the data directly from variables instead of saving then loading a file? This would be much faster for multiple compression operations in a row.

Yes it would. Honestly, I didn't see much sense in that, that's why it's not there.

But I guess it could be useful if data is exchanged between entities, over TCP connection for example. In that case if you want to compress and decompress data this way, new function could be written that would load "Image" from the variable holding compressed data. That function could be something like this:

; _GDIPlus_Startup() must be called before usage
Func GdipImageLoadFromVariable($vImage)
    Local Const $sIID_IStream = "{0000000C-0000-0000-C000-000000000046}"
    Local Const $tagIStreamReadWrite = "Read hresult(struct*;dword;dword*); Write hresult(struct*;dword;dword*);"
    ; Create stream object
    Local $oStream = ObjCreateInterface(CreateStreamOnHGlobal(), $sIID_IStream, $tagIStreamReadWrite)
    ; Check for errors
    If @error Then Return SetError(1, 0, 0)
    ; Allocate space for binary
    Local $tBinary = DllStructCreate("byte[" & BinaryLen($vImage) & "]")
    DllStructSetData($tBinary, 1, $vImage)
    ; Write to stream
    Local $iWritten
    $oStream.Write($tBinary, DllStructGetSize($tBinary), $iWritten)
    ; Load Image from stream
    Local $hImage = GdipLoadImageFromStream($oStream())
    If @error Then Return SetError(2, 0, 0)
    Return $hImage
EndFunc

Func GdipLoadImageFromStream($pStream)
    Local $aCall = DllCall($ghGDIPDll, "dword", "GdipLoadImageFromStream", _
            "ptr", $pStream, _
            "ptr*", 0)
    If @error Or $aCall[0] Then Return SetError(1, 0, 0)
    Return $aCall[2]
EndFunc
Edited by trancexx

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

@trancexx yes, the idea was for transport over sockets, thanks for the example!

After testing the addition of this compression method, the data was actually bloated, and the send rate slower, as compared to using just LZMA alone. I was able to achieve faster data processing, AND faster data transfer by using LZMA + QuickLZ at different points, but eventually they began to overlap and corrupt data, I was looking for an alternative secondary compression algorithm, but this just seems a bit too bulky for the size of data I'm transfer, but perhaps it could be more useful on operation or transfers involving larger amounts of data?

Either way, nice code!

Edit: Also would there be a way to make this have more width and less length, I think this would be a neat way to print out data, bit it would be very difficult to reconstruct an image that is so tall and thin.

Edited by nullschritt

Share this post


Link to post
Share on other sites

#16 ·  Posted (edited)

create png okey

but

create gif

SaveAsPic($sData, $sPic,"gif")

ConsoleWrite(BinaryToString(ReadFromPic($sPic)) & @CRLF) dont work

why?

Edited by mesale0077

Share this post


Link to post
Share on other sites

Because GIF format allows up to 8bpp and that is a problem because suitable initial BMP should be made to match the scheme.

To understand the issue fully you should really learn more about these formats. Don't because only freaks do that. Freaks and me.

1 person likes this

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

Thank you for sharing this script, can you tell me if the compression GDI limits or if it is me who is doing it wrong.
Here is a reduced easier to show you my problem script:

#include <ScreenCapture.au3>
#include "_Compress.au3"
#include "Bitmap_String.au3"

While(True)
    Local $hBmb = _ScreenCapture_Capture("")
    Local $img = _GDIPlus_BitmapCreateFromHBITMAP($hBmb)
    Local $binString = _Bitmap2BinaryString($img, 100)
    Local $imgCompress = Compress($binString)
    ConsoleWrite(BinaryLen($binString) & " -> " & BinaryLen($imgCompress) & @LF)
    _WinAPI_DeleteObject($hBmb)
    _GDIPlus_ImageDispose($img)
    Sleep(10)
WEnd

In running, we can see that the compression returns an empty string from time to time.

 
I see this problem only appears with certain images, here's a script and a clear picture that reproduces the bug:
#include "_Compress.au3"

Local $binString = FileRead(@ScriptDir & "\a.png.txt")
Local $imgCompress = Compress($binString)
ConsoleWrite(BinaryLen($binString) & " -> " & BinaryLen($imgCompress) & @LF)
In "GdipImageSaveToVariable", GdipSaveImageToStream return 7 -> "Win32Error"
 
Would you have any idea why?
 

Thank you.

_Compress.au3

Bitmap_String.au3

a.png.txt

Edited by TommyDDR

_GUIRegisterMsg (Register more than 1 time the same Msg), _Resize_Window (GUICtrlSetResizing for children windows), _GUICtrlSetOnHover (Link a function when mouse go on, left, clic down, clic up, on a control), _InputHeure (Create an input for hour contain), _GUICtrlCalendar (Make a complete calendar), _GUICtrlCreateGraphic3D (Create a 3D graph), _ArrayEx.au3 (Array management), _GUIXViewEx.au3 (List/Tree View management).

Share this post


Link to post
Share on other sites

#19 ·  Posted (edited)

Good job but I can't run it. Please update for AutoIt 3.3.12.0.

Edited by Adele

Share this post


Link to post
Share on other sites

$ghGDIPDll  to $__g_hGDIPDll change please :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now