Resize a file with null chars

[Starcom]

Hi,

I need to enlarge a file with "Null" chars but the size of my file stay the same.

This is a little script:

#include <String.au3>
$Max = 524288
$Size = FileGetSize("V:/dev/test.bin")
$handle = FileOpen("V:/dev/test.bin",1)
$Size = $Max - $Size
FileWrite("V:/dev/test.bin",_StringRepeat(CHR(00),$Size))
FileClose($handle)
Exit

Thanks

Edited June 18, 2007 by Starcom

[The Kandie Man]

Yeah, I worked up a little something for you. I got everything working right and then I got angry at the speed of string repeat. It pissed me off so much that I just decided to write my own _HighSpeedStringRepeat function. My function takes less than a second to execute while the older traditional StringRepeat function took so long that I just canceled the script before it finished.

#include <String.au3>; you can remove this if you use the high speed string repeat
$s_FileName = "V:/dev/test.bin"
;~ $s_FileName = "test.bin"
$i_Max = 524288
$i_Size = FileGetSize($s_FileName)
$h_File = FileOpen($s_FileName,1 + 16);Open file for appending and writing as binary
$i_Size = $i_Max - $i_Size
;~ FileWrite($h_File,_StringRepeat(CHR(0),$i_Size));use this for slowness
FileWrite($h_File,_HighSpeedStringRepeat(CHR(0),$i_Size));use this for high speed
FileClose($h_File)
Exit

Func _HighSpeedStringRepeat($s_String,$i_Count)
    ;Written by The Kandie Man because he was pissed off with the slow speed of StringRepeat, so he made an exponential HighSpeedStringRepeat
    While StringLen($s_String) < $i_Count
        $s_String = $s_String & $s_String & $s_String
    WEnd
    Return StringTrimRight($s_String,StringLen($s_String) - $i_Count)
EndFunc

It worked perfectly for me, I hope it works for you.

- The Kandie Man ;-)

[Starcom]

It looks very nice.

I'll try this tonight.

I see you have resolve the speed problem too (my firsts tries used a For...Next loop and add char to char... very slooowlyy ).

Thank you for your job. I'll return to write a feed back.

[Richard Robertson]

I love that exponential character addition. How'd you think that up?

[ChrisL]

I need to enlarge a file with "Null" chars but the size of my file stay the same.

Just curious but why do you need to enlarge a file?

[The Kandie Man]

I love that exponential character addition. How'd you think that up?

Yeah, I knew what things could be done fast and what things could be done slow. I knew that concentanating strings, regardless of size, takes a very small amount of time. With that in mind, i just told AutoIt to concentenate strings furiously by using an exponential base of 3. Each time the loop runs, it adds an additional power to the original base 3. First loop there are 3 null characters, second loop there are 9 null characters, third loop there are 27 null characters, etc. By doing this, I significantly reduce the number of loops needed to perform the string repeat function. This does add a problem though, the exponential growth will ultimately be larger than what I need. This is trivial as todays computers have large amounts of RAM and the StringTrimRight and StringTrimLeft functions can be used to cut off any excess characters. The StringTrimRight and StringTrimLeft functions are extremely fast and can trim off extremely large amounts of extremely large strings practically instantly.

The post above uses a high speed string repeat function that will only work effectively for a one character string. I have since then written another _HighSpeedStringRepeat() function that is capable of repeating a multi-character string. This function uses a more conservative exponential base of 2:

$s_FileName = "TestString.bin"
$h_File = FileOpen($s_FileName,2 + 16);Open file for erase contents and writing as binary
FileWrite($h_File,_HighSpeedStringRepeat("ASDF;",1000000));use this for high speed
FileClose($h_File)

;Note, this function uses the same @ERROR flags and return value as _StringRepeat()
Func _HighSpeedStringRepeat($s_String,$i_Count)
    ;Written by The Kandie Man because he was pissed off with the slow speed of StringRepeat, so he made an exponential HighSpeedStringRepeat
    Select
        Case Not StringIsInt($i_Count)
            SetError(1)
            Return ""
        Case StringLen($s_String) < 1
            SetError(1)
            Return ""
        Case $i_Count <= 0
            SetError(1)
            Return ""
        Case Else
            Local $i_BaseStringLen = StringLen($s_String)
            Local $i_TotalStringLen = $i_BaseStringLen * $i_Count
            While StringLen($s_String) < $i_TotalStringLen
                $s_String &= $s_String
            WEnd
            Return StringTrimRight($s_String,StringLen($s_String) - $i_TotalStringLen)
    EndSelect
EndFunc

The above script used 56,192 KB of RAM to perform the operation. When I increased the number of repeats from 1,000,000 to 10,000,000 it used 827,208 KB of RAM to perform the operation. I therefore don't recommend you use it for anything higher than 10,000,000 because if the program continues at the exponential rate, AutoIt will give a fatal unable to allocate memory error. This is because AutoIt was probably trying to allocate something like 2400 MB of RAM to perform the next exponential string concentenation operation.

I may add to this function so that it only loops a certain number of times to prevent AutoIt memory allocation errors. It could then be used on releatively large files as well. Right now i can replicate 50MB of null characters in a couple seconds and safely write them to a file without running out of memory.

I will work on this function some more if there is any interest for it.

- The Kandie Man ;-)

EDIT: Used Randallc's suggested &= operator which is %15 faster

Edited June 19, 2007 by The Kandie Man

[The Kandie Man]

Just curious but why do you need to enlarge a file?

There could be many reasons. The most likely reason is that a program that interprets the file requires the file to have a certain length regardless of its contents in order to be correctly read. A perfect example is the format that Microsoft uses for its word documents. Create a blank document and save it. You will notice the word document is an amazing 20.0 KB size. Why the hell does it have a 20KB size if there is absolutely nothing in the documents? Simple, the .doc format has a bunch of header and formatting information that is included in the saved file. When a blank document is saved, it is saved with 20KB of formatting and heading data. Now, I am not saying that Null characters are heading data, but sometimes programs read certain settings from a configuration file by byte location. For example, a setting for the location of a menu or something may be located in the file 300 bytes in. The application then opens the file and reads byte 300 to 302 because the setting has a 3 byte length reserved to it. The value for that may be 20, but there may be a maximum value of 999 for that setting. As a result, 3 bytes of space are reserved in that file for that setting. The last byte of the 3 byte setting would be NULL since the current value is 20.

I hope that made sense.

- The Kandie Man ;-)

[randallc]

Hi,

I think you can just set the EndOfFile to larger; 10x quicker if speed an issue;

;apitry3.au3
#include "APITailRW.au3"
Local $sDeskFile = @DesktopDir & "\Prairie Wind.bmp", $s_File = @WindowsDir & "\Prairie Wind.bmp"
FileCopy($s_File,$sDeskFile,1)
local $i_FileSize = FileGetSize($sDeskFile), $TimerStamp1=TimerInit(),$i_Max = 524288,$i_Size = $i_Max - $i_FileSize
ConsoleWrite("$i_FileSize="&$i_FileSize&@LF)
;1  ======_FileSetEndAPI======================================
$h_File = _FileOpenAPI ($sDeskFile) ;_APIFileOpen
_FileSetPosAPI ($h_File, $i_FileSize+$i_Size)
_FileSetEndAPI($h_File)
_FileCloseAPI ($h_File);_APIFileClose
$i_FileSize2 = FileGetSize($sDeskFile)
ConsoleWrite("Time _FileSetEndAPI="&round(TimerDiff($TimerStamp1))&"msec"&@LF)
;~ ShellExecuteWait($sDeskFile)
ConsoleWrite("$i_FileSize2="&FileGetSize($sDeskFile)&@LF)
;======Read parts or tail of binary file =====in any order=================================
local $h_File = _FileOpenAPI ($sDeskFile), $i_FileSize2 = FileGetSize($sDeskFile);_APIFileOpen
$b_dataLast100 = _FileReadAPI ($h_File, 100, $i_FileSize2 -100, 1)  ; read last 100 bin chars
_FileCloseAPI ($h_File);_APIFileClose
ConsoleWrite("$b_dataLast100="&$b_dataLast100&@LF)

Time _FileSetEndAPI=8msec

Time _HighSpeedStringRepeat1=74msec

Time _HighSpeedStringRepeat2=64msec

Best, randall [cf use &= for 15% speed increase in _HighSpeedStringRepeat2 and original _StringRepeat] Edited June 19, 2007 by randallc

[Starcom]

Nice idea !

But it is usefull only in my case (Null chars filling).

Thanks, I'will try it too.

Just curious but why do you need to enlarge a file?

As The Kandie Man wrote, I need to make a file compatible with a program that uses fixed size files.

[Starcom]

Well, I've tested both solution and finally decide to use the Kandie Man's one because the files size I need to use is less than 2 Mo, so the speed is acceptable.

Second reason is that his solution doesn't requires any includes.

API solution will be a good solution to compute big files as logfiles, DVDImage, Movies, etc...

[randallc]

Well, I've tested both solution and finally decide to use the Kandie Man's one because the files size I need to use is less than 2 Mo, so the speed is acceptable.

Second reason is that his solution doesn't requires any includes.

API solution will be a good solution to compute big files as logfiles, DVDImage, Movies, etc...

OK, good luck.

Best, Randall

[Starcom]

OK, good luck.

Best, Randall

Is your APITailRW.au3 able to truncate X byte at the start of the file ?

[randallc]

Is your APITailRW.au3 able to truncate X byte at the start of the file ?

I think so; just set _filepos using that func, or as parameter in filereader func, for start; then set length to read.

best, Randall

[Starcom]

Thanks. I'll try it.

[Starcom]

I bet i did something wrong...

;apitry3.au3
#include "F:\Test\APITailRW.au3"

Local $sDeskFile =  "F:\Test\test.txt"
local $i_FileSize = FileGetSize($sDeskFile), $i_Max = 524288, $i_start = 4, $i_Size = $i_Max - $i_FileSize, $i_sizeXXL = $i_Max + $i_start


ConsoleWrite("$i_FileSize=" & $i_FileSize & @LF)

;1  ======_FileSetEndAPI======================================
;Enlarge the file to the max size
$h_File = _FileOpenAPI ($sDeskFile) ;_APIFileOpen
_FileSetPosAPI ($h_File, $i_sizeXXL)
_FileSetEndAPI($h_File)
_FileCloseAPI ($h_File);_APIFileClose

;Read file from start position to size of final file
$h_File = _FileOpenAPI ($sDeskFile) ;_APIFileOpen
$DatasRead = _FileReadAPI ($h_File, $i_max, $i_start) ;_APIFileOpen
_FileCloseAPI ($h_File);_APIFileClose

;Write final file starting at begin of the FileChangeDir
$h_File = _FileOpenAPI ($sDeskFile) ;_APIFileOpen
_FileWriteAPI ($h_File, $DatasRead)
_FileCloseAPI ($h_File);_APIFileClose

;Set end of the file at needs length
$h_File = _FileOpenAPI ($sDeskFile) ;_APIFileOpen
_FileSetPosAPI ($h_File, $i_Max)
_FileSetEndAPI($h_File)
_FileCloseAPI ($h_File);_APIFileClose

$i_FileSize2 = FileGetSize($sDeskFile)

;~ ShellExecuteWait($sDeskFile)
ConsoleWrite("$i_FileSize2="&FileGetSize($sDeskFile)&@LF)

1. I'm not sure that works in binary.

2. Start datas are repeated next.

[randallc]

Hi,

1. There's no point setting final size before starting? - only at end

2. You did not set FilePos parameter in file write to zero to overwrite at start.

3. You will see still 2 caracters on next line repeated as you are not overwriting them with any data

4. You don't need to keep opening/ closinf file handle as you are using the same one.

5. there's a fourth otional parameter in _FileReadAPI ;",1" sets binary

Best, randall

;apitry4.au3
#include "APITailRW.au3"
Local $sDeskFile =  @DesktopDir&"\Answer2.txt",$sFileRead =  @ScriptDir&"\Answer.txt"
FileCopy($sFileRead,$sDeskFile,1)
local $i_FileSize = FileGetSize($sDeskFile), $i_Max = 524288, $i_start = 4, $i_Size = $i_Max - $i_FileSize, $i_sizeXXL = $i_Max + $i_start
ConsoleWrite("$i_FileSize=" & $i_FileSize & @LF)
$h_File = _FileOpenAPI ($sDeskFile);_FileOpenAPI;Read file from start position to size of  file
$DatasRead = _FileReadAPI ($h_File, $i_FileSize, $i_start);[******* ",1" next parameter if you want binary]
_FileWriteAPI ($h_File, $DatasRead,0);[,0 param to set pos;Write final file starting at begin of the FileChangePos
_FileSetPosAPI ($h_File, $i_Max); else you will still have the first _FileSetPosAPI and be writing at the last set pos position]
_FileSetEndAPI($h_File);Set end of the file at needs length
_FileCloseAPI ($h_File);_FileCloseAPI
$i_FileSize2 = FileGetSize($sDeskFile)
ConsoleWrite("$i_FileSize2="&FileGetSize($sDeskFile)&@LF)

Best, randall Edited June 28, 2007 by randallc