Sign in to follow this  
Followers 0
JFX

[solved] FileSetPos and FileWrite speed on hugh files

23 posts in this topic

#1 ·  Posted (edited)

Hi, I'm trying to create a big file and append data to the end of it.

$hFile = FileOpen("D:\test.bin", 16+1)
FileSetPos($hFile, 8 * 1024 * 1024 * 1024, 0)
FileWrite($hFile, 'footer')
FileClose($hFile)

But it takes over 2 minutes. is there anyway to make this faster?

Edited by JFX

Share this post


Link to post
Share on other sites



Hi JFX,

I wonder why you want to set the Position if you use append mode on fileopen?


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

Hi Hannes,

the file does not exist and i need to write to that offset.

What i want is to create a fixed VHD the fast way like this tool does vhdtool

I mostly have finished creation of the vhd footer, but the problem is how to create the big file and append the 512 Bytes to it. :)

Share this post


Link to post
Share on other sites

Try this:

;coded by UEZ 2011
#Include <WinAPI.au3>

Global $bin, $sbin
For $i = 1 To 512 ;create 512 bytes of data
    $sbin &= $i
Next
$bin = Binary($sbin) ;convert is to binary

$nBytes = 0
$bytes = 10 * 1024^2 ;create a 10 MB file
$tBuffer = DllStructCreate("byte[" & $bytes & "]") ;create struct
$hFile = _WinAPI_CreateFile("Huge.bin", 1, 4, 2) ;create 10 MB file
_WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $bytes, $nBytes) ;fill 10 MB with zeros
_WinAPI_SetFilePointer($hFile, -512, 2) ;set file position to end of file minus 512
DllStructSetData($tBuffer, 1, $bin) ;fill 512 byte of data
_WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), 512, $nBytes) ;write 512 bytes of data to the end of the file
_WinAPI_FlushFileBuffers($hFile) ;flush to disk
_WinAPI_CloseHandle($hFile) ;close file handle
$tBuffer = 0 ;release struct memory

Br,

UEZ


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

Thanks UEZ,

but this would give me memory allocation error with bigger sizes

and would not be faster because it also fills the file with zeros.

Share this post


Link to post
Share on other sites

That is the disadvantage of this method when you really want to create huge files, e.g. 5 GB and you don't have the memory for it.

Let me try to find out a workaround to create huge files without allocating huge memory!

Br,

UEZ


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

hmm, this seems to quick create a 8GB + 512 Bytes file in a second.

SetFileSize("D:\test.bin", 8 * 1024^3 + 512)


Func SetFileSize($FileName, $newSize)
    $fileHandle = _WinAPI_CreateFile($FileName, 1, 4, 4)
    _WinAPI_SetFilePointerEx($fileHandle, $newSize)
    DllCall("kernel32.dll", "long", "SetEndOfFile", "long", $fileHandle)
    _WinAPI_FlushFileBuffers($fileHandle)
    _WinAPI_CloseHandle($fileHandle)
EndFunc



Func _WinAPI_SetFilePointerEx($hFile, $iPos, $iMethod = 0)
    Local $aResult
    $aResult = DllCall("kernel32.dll", "dword", "SetFilePointerEx", "hwnd", $hFile, "uint64", $iPos, "uint64*", 0, "dword", $iMethod)
    If @error Then Return SetError(1, 0, -1)
    Return $aResult[0]
EndFunc

but i still need to write in the last 512 bytes nad this takes minutes again.

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Try this one:

;coded by UEZ 2011
#Include <WinAPI.au3>

Global $bin, $sbin
For $i = 1 To 512 ;create 512 bytes of data
    $sbin &= $i
Next
$bin = Binary($sbin) ;convert is to binary

$nBytes = 0
$bytes = 10 * 1024^2 ;create a 10 MB file
$cbytes = 1024
$restbytes = Mod($bytes, $cbytes)
$tBuffer = DllStructCreate("byte[" & $cbytes & "]") ;create struct
$hFile = _WinAPI_CreateFile("Huge.bin", 1, 4, 2)

$bench = TimerInit()
For $i = 0 To Floor($bytes / $cbytes) Step $cbytes ;create 10 MB file in 1MB steps
    _WinAPI_SetFilePointer($hFile, $i * $cbytes, 0) ;set file position to end of file minus 512
    _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $cbytes, $nBytes)
Next
If $restbytes Then
    _WinAPI_SetFilePointer($hFile, 0, 2)
    _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $restbytes, $nBytes)
EndIf
ConsoleWrite(Round(TimerDiff($bench), 2) & " ms to create " & $bytes & " bytes" & @CRLF)

$bench = TimerInit()
_WinAPI_SetFilePointer($hFile, -512, 2) ;set file position to end of file minus 512
DllStructSetData($tBuffer, 1, $bin) ;fill 512 bytes of data
_WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), 512, $nBytes) ;write 512 bytes of data to the end of the file
ConsoleWrite(Round(TimerDiff($bench), 2) & " ms to add 512 bytes of data to the end of the file." & @CRLF)
_WinAPI_FlushFileBuffers($hFile) ;flush to disk
_WinAPI_CloseHandle($hFile) ;close file handle
$tBuffer = 0 ;release struct memory

My results:

272.88 ms to create 10485760 bytes

0.28 ms to add 512 bytes of data to the end of the file.

Br,

UEZ

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

You cannot write to an uninitialzed file, so Windows fills it with zeros which takes some time.


*GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes

Share this post


Link to post
Share on other sites

@UEZ,

_WinAPI_SetFilePointer seems to have a 2GB limit and even with _WinAPI_SetFilePointerEx this will be also very slow when using with GB sizes.

@ProgAndy

Interesting, do you know a way to avoid this?

The VHDTool here seems to do this somehow http://archive.msdn.microsoft.com/vhdtool

Share this post


Link to post
Share on other sites

Interesting, do you know a way to avoid this?

No. Also, this is just something I concluded from testing and I have no documented proof for that.

*GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

Okay Problem solved,

fsutil file setvaliddata saved the day

Edited by JFX

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

This function exists as DLLCall, too: SetFileValidData in kernel32.dll

http://msdn.microsoft.com/en-us/library/aa365544%28v=vs.85%29.aspx

PS: It will only work when the file is on an NTFS formatted partition.

Edited by ProgAndy

*GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes

Share this post


Link to post
Share on other sites

Thanks I will use a check if file system is not NTFS.

Better to keep fsutil, dllcall is to weird for me. :)

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000

But what is the sense of fsutil file setvaliddata <size>?

Func _WinAPI_SetFileValidData($hFile, $datalength)
    Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength)
    If @error Or Not $aRes[0] Then SetError(1, 0, 0)
    Return $aRes[0]
EndFunc

Br,

UEZ

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000

But what is the sense of fsutil file setvaliddata <size>?

Func _WinAPI_SetFileValidData($hFile, $datalength)
    Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength)
    If @error Or Not IsArray($aRes) Then SetError(1, 0, 0)
    Return SetError(0, 0, $aRes[0])
EndFunc

Br,

UEZ

Just as a side note, what's the sense in:

If @error Or Not IsArray($aRes) Then SetError(1, 0, 0)

...line of your function?

If there is an error then there will be no array and if $aRes is not array then there is an error set. It's like saying "if wrong or not right".

This would be much more correct here:

If @error Or Not $aRes[0] Then...

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

It is senseless, I created only the function and played with it to find out the functionality but could not find any usage for it!

Br,

UEZ


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

It is senseless, I created only the function and played with it to find out the functionality but could not find any usage for it!

I corrected it. :)

Br,

UEZ

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000

But what is the sense of fsutil file setvaliddata <size>?

Func _WinAPI_SetFileValidData($hFile, $datalength)
    Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength)
    If @error Or Not $aRes[0] Then SetError(1, 0, 0)
    Return SetError(0, 0, $aRes[0])
EndFunc

Br,

UEZ

Just as a side note, what's the sense in:

Return SetError(0, 0, $aRes[0])

...line of your function?

When entering a function @error is set to 0. If there is no error additionally set then there will be no error set (how smart), meaning error is 0 after the function returns.

This would be much more correct here:

Return $aRes[0]

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

Of course it makes not sense to set error = 0 when there has no error occured but it it's just a habit like doing some things with the right hand... :)

Br,

UEZ


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0