Memory compression

[trancexx 1,021]

Appears that memory compression is in these days. So, to be in, here is another method. This one is called "Native API Compression".

Apparently it's LZ1(LZ77) compression algorithm or something very similar and it's available through one or two dll calls. Of course that files can be compressed too by this method.

Functions with small example:

expandcollapse popup

#NoTrayIcon

$sString = "lzwlzwlzwlzwlzwlzwlzwlzwlzwlzwlzwlzw"
ConsoleWrite("Before compression: " & Binary($sString) & @CRLF)

$BC = _LZNTCompress($sString)
ConsoleWrite("Compressed: " & $BC & @CRLF)

$CB = _LZNTDecompress($BC)
ConsoleWrite("Decompressed: " & $CB & @CRLF)






; #FUNCTION# ;===============================================================================
;
; Name...........: _LZNTDecompress
; Description ...: Decompresses input data.
; Syntax.........: _LZNTDecompress ($bBinary)
; Parameters ....: $vInput - Binary data to decompress.
; Return values .: Success - Returns decompressed binary data.
;                          - Sets @error to 0
;                  Failure - Returns empty string and sets @error:
;                  |1 - Error decompressing.
; Author ........: trancexx
; Related .......: _LZNTCompress
; Link ..........; http://msdn.microsoft.com/en-us/library/bb981784.aspx
;
;==========================================================================================
Func _LZNTDecompress($bBinary)

    $bBinary = Binary($bBinary)

    Local $tInput = DllStructCreate("byte[" & BinaryLen($bBinary) & "]")
    DllStructSetData($tInput, 1, $bBinary)

    Local $tBuffer = DllStructCreate("byte[" & 16 * DllStructGetSize($tInput) & "]") ; initially oversizing buffer

    Local $a_Call = DllCall("ntdll.dll", "int", "RtlDecompressBuffer", _
            "ushort", 2, _
            "ptr", DllStructGetPtr($tBuffer), _
            "dword", DllStructGetSize($tBuffer), _
            "ptr", DllStructGetPtr($tInput), _
            "dword", DllStructGetSize($tInput), _
            "dword*", 0)

    If @error Or $a_Call[0] Then
        Return SetError(1, 0, "") ; error decompressing
    EndIf

    Local $tOutput = DllStructCreate("byte[" & $a_Call[6] & "]", DllStructGetPtr($tBuffer))

    Return SetError(0, 0, DllStructGetData($tOutput, 1))

EndFunc   ;==>_LZNTDecompress



; #FUNCTION# ;===============================================================================
;
; Name...........: _LZNTCompress
; Description ...: Compresses input data.
; Syntax.........: _LZNTCompress ($vInput [, $iCompressionFormatAndEngine])
; Parameters ....: $vInput - Data to compress.
;                  $iCompressionFormatAndEngine - Compression format and engine type. Default is 2 (standard compression). Can be:
;                  |2 - COMPRESSION_FORMAT_LZNT1 | COMPRESSION_ENGINE_STANDARD
;                  |258 - COMPRESSION_FORMAT_LZNT1 | COMPRESSION_ENGINE_MAXIMUM
; Return values .: Success - Returns compressed binary data.
;                          - Sets @error to 0
;                  Failure - Returns empty string and sets @error:
;                  |1 - Error determining workspace buffer size.
;                  |2 - Error compressing.
; Author ........: trancexx
; Related .......: _LZNTDecompress
; Link ..........; http://msdn.microsoft.com/en-us/library/bb981783.aspx
;
;==========================================================================================
Func _LZNTCompress($vInput, $iCompressionFormatAndEngine = 2)

    If Not ($iCompressionFormatAndEngine = 258) Then 
        $iCompressionFormatAndEngine = 2
    EndIf   
    
    Local $bBinary = Binary($vInput)

    Local $tInput = DllStructCreate("byte[" & BinaryLen($bBinary) & "]")
    DllStructSetData($tInput, 1, $bBinary)

    Local $a_Call = DllCall("ntdll.dll", "int", "RtlGetCompressionWorkSpaceSize", _
            "ushort", $iCompressionFormatAndEngine, _
            "dword*", 0, _
            "dword*", 0)

    If @error Or $a_Call[0] Then
        Return SetError(1, 0, "") ; error determining workspace buffer size
    EndIf

    Local $tWorkSpace = DllStructCreate("byte[" & $a_Call[2] & "]") ; workspace is needed for compression

    Local $tBuffer = DllStructCreate("byte[" & 16 * DllStructGetSize($tInput) & "]") ; initially oversizing buffer

    Local $a_Call = DllCall("ntdll.dll", "int", "RtlCompressBuffer", _
            "ushort", $iCompressionFormatAndEngine, _
            "ptr", DllStructGetPtr($tInput), _
            "dword", DllStructGetSize($tInput), _
            "ptr", DllStructGetPtr($tBuffer), _
            "dword", DllStructGetSize($tBuffer), _
            "dword", 4096, _
            "dword*", 0, _
            "ptr", DllStructGetPtr($tWorkSpace))

    If @error Or $a_Call[0] Then
        Return SetError(2, 0, "") ; error compressing
    EndIf

    Local $tOutput = DllStructCreate("byte[" & $a_Call[7] & "]", DllStructGetPtr($tBuffer))

    Return SetError(0, 0, DllStructGetData($tOutput, 1))

EndFunc   ;==>_LZNTCompress

Available in Microsoft Windows XP and later versions of all Windows operating systems.

There is $tBuffer there if you look. I'm oversizing it plenty initially (saw this bad example in c++). Probably should be some better way to do that but documentation for this functions on MSDN is not over yet apparently

Two compression rates are available. Default COMPRESSION_ENGINE_STANDARD is lightening speedy. Better compression is gained by COMPRESSION_ENGINE_MAXIMUM which is slower.

[smashly 12]

Thank You for sharing and nice work trancexx,

I just tried it on a 12KB script as standard and the output file is 6KB..

50% compression of my script... nice

Set compression as max and I get 5KB filesize.

The compression isn't as high as 3rd party programs, but since it's using the native windows to do it's thing I love the idea.

I'm off to play with it bit more

Cheers

Edited January 10, 2009 by smashly

[trancexx 1,021]

Thank You for sharing and nice work trancexx,

I just tried it on a 12KB script as standard and the output file is 6KB..

50% compression of my script... nice

Set compression as max and I get 5KB filesize.

The compression isn't as high as 3rd party programs, but since it's using the native windows to do it's thing I love the idea.

I'm off to play with it bit more

Cheers

Ok, enough playing with it! You're gonna break something.

[Splash 1]

Thanks! =D

[knightz 0]

thank you

this script is very useful

[netegg 0]

so sad, would you like to explain how to use it? Why i can't get the result as above?

[trancexx 1,021]

It's very hard to explain.You just do.

Understanding internal workings might help; 150 sentences about the working.

Edited March 9, 2011 by trancexx

[AdmiralAlkex 127]

This UDF is great. I use it for the "CompressPlaylist"-option in ShiftER, end result is around a third/sixth that of the non-compressed file.

Thank you trancexx