[solved] FileSetPos and FileWrite speed on hugh files

[JFX]

Hi, I'm trying to create a big file and append data to the end of it.

$hFile = FileOpen("D:\test.bin", 16+1)
FileSetPos($hFile, 8 * 1024 * 1024 * 1024, 0)
FileWrite($hFile, 'footer')
FileClose($hFile)

But it takes over 2 minutes. is there anyway to make this faster?

Edited March 28, 2011 by JFX

[hannes08]

Hi JFX,

I wonder why you want to set the Position if you use append mode on fileopen?

[JFX]

Hi Hannes,

the file does not exist and i need to write to that offset.

What i want is to create a fixed VHD the fast way like this tool does vhdtool

I mostly have finished creation of the vhd footer, but the problem is how to create the big file and append the 512 Bytes to it.

[UEZ]

Try this:

;coded by UEZ 2011
#Include <WinAPI.au3>

Global $bin, $sbin
For $i = 1 To 512 ;create 512 bytes of data
    $sbin &= $i
Next
$bin = Binary($sbin) ;convert is to binary

$nBytes = 0
$bytes = 10 * 1024^2 ;create a 10 MB file
$tBuffer = DllStructCreate("byte[" & $bytes & "]") ;create struct
$hFile = _WinAPI_CreateFile("Huge.bin", 1, 4, 2) ;create 10 MB file
_WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $bytes, $nBytes) ;fill 10 MB with zeros
_WinAPI_SetFilePointer($hFile, -512, 2) ;set file position to end of file minus 512
DllStructSetData($tBuffer, 1, $bin) ;fill 512 byte of data
_WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), 512, $nBytes) ;write 512 bytes of data to the end of the file
_WinAPI_FlushFileBuffers($hFile) ;flush to disk
_WinAPI_CloseHandle($hFile) ;close file handle
$tBuffer = 0 ;release struct memory

Br,

UEZ

[JFX]

Thanks UEZ,

but this would give me memory allocation error with bigger sizes

and would not be faster because it also fills the file with zeros.

[UEZ]

That is the disadvantage of this method when you really want to create huge files, e.g. 5 GB and you don't have the memory for it.

Let me try to find out a workaround to create huge files without allocating huge memory!

Br,

UEZ

[JFX]

hmm, this seems to quick create a 8GB + 512 Bytes file in a second.

SetFileSize("D:\test.bin", 8 * 1024^3 + 512)


Func SetFileSize($FileName, $newSize)
    $fileHandle = _WinAPI_CreateFile($FileName, 1, 4, 4)
    _WinAPI_SetFilePointerEx($fileHandle, $newSize)
    DllCall("kernel32.dll", "long", "SetEndOfFile", "long", $fileHandle)
    _WinAPI_FlushFileBuffers($fileHandle)
    _WinAPI_CloseHandle($fileHandle)
EndFunc



Func _WinAPI_SetFilePointerEx($hFile, $iPos, $iMethod = 0)
    Local $aResult
    $aResult = DllCall("kernel32.dll", "dword", "SetFilePointerEx", "hwnd", $hFile, "uint64", $iPos, "uint64*", 0, "dword", $iMethod)
    If @error Then Return SetError(1, 0, -1)
    Return $aResult[0]
EndFunc

but i still need to write in the last 512 bytes nad this takes minutes again.

[UEZ]

Try this one:

;coded by UEZ 2011
#Include <WinAPI.au3>

Global $bin, $sbin
For $i = 1 To 512 ;create 512 bytes of data
    $sbin &= $i
Next
$bin = Binary($sbin) ;convert is to binary

$nBytes = 0
$bytes = 10 * 1024^2 ;create a 10 MB file
$cbytes = 1024
$restbytes = Mod($bytes, $cbytes)
$tBuffer = DllStructCreate("byte[" & $cbytes & "]") ;create struct
$hFile = _WinAPI_CreateFile("Huge.bin", 1, 4, 2)

$bench = TimerInit()
For $i = 0 To Floor($bytes / $cbytes) Step $cbytes ;create 10 MB file in 1MB steps
    _WinAPI_SetFilePointer($hFile, $i * $cbytes, 0) ;set file position to end of file minus 512
    _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $cbytes, $nBytes)
Next
If $restbytes Then
    _WinAPI_SetFilePointer($hFile, 0, 2)
    _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $restbytes, $nBytes)
EndIf
ConsoleWrite(Round(TimerDiff($bench), 2) & " ms to create " & $bytes & " bytes" & @CRLF)

$bench = TimerInit()
_WinAPI_SetFilePointer($hFile, -512, 2) ;set file position to end of file minus 512
DllStructSetData($tBuffer, 1, $bin) ;fill 512 bytes of data
_WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), 512, $nBytes) ;write 512 bytes of data to the end of the file
ConsoleWrite(Round(TimerDiff($bench), 2) & " ms to add 512 bytes of data to the end of the file." & @CRLF)
_WinAPI_FlushFileBuffers($hFile) ;flush to disk
_WinAPI_CloseHandle($hFile) ;close file handle
$tBuffer = 0 ;release struct memory

My results:

272.88 ms to create 10485760 bytes

0.28 ms to add 512 bytes of data to the end of the file.

Br,

UEZ

Edited March 28, 2011 by UEZ

[ProgAndy]

You cannot write to an uninitialzed file, so Windows fills it with zeros which takes some time.

[JFX]

@UEZ,

_WinAPI_SetFilePointer seems to have a 2GB limit and even with _WinAPI_SetFilePointerEx this will be also very slow when using with GB sizes.

@ProgAndy

Interesting, do you know a way to avoid this?

The VHDTool here seems to do this somehow http://archive.msdn.microsoft.com/vhdtool

[ProgAndy]

Interesting, do you know a way to avoid this?

No. Also, this is just something I concluded from testing and I have no documented proof for that.

[JFX]

Okay Problem solved,

fsutil file setvaliddata saved the day

Edited March 28, 2011 by JFX

[ProgAndy]

This function exists as DLLCall, too: SetFileValidData in kernel32.dll

http://msdn.microsoft.com/en-us/library/aa365544%28v=vs.85%29.aspx

PS: It will only work when the file is on an NTFS formatted partition.

Edited March 28, 2011 by ProgAndy

[JFX]

Thanks I will use a check if file system is not NTFS.

Better to keep fsutil, dllcall is to weird for me.

[UEZ]

fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000

But what is the sense of fsutil file setvaliddata <size>?

Func _WinAPI_SetFileValidData($hFile, $datalength)
    Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength)
    If @error Or Not $aRes[0] Then SetError(1, 0, 0)
    Return $aRes[0]
EndFunc

Br,

UEZ

Edited March 28, 2011 by UEZ

[trancexx]

fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000

But what is the sense of fsutil file setvaliddata <size>?

Func _WinAPI_SetFileValidData($hFile, $datalength)
    Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength)
    If @error Or Not IsArray($aRes) Then SetError(1, 0, 0)
    Return SetError(0, 0, $aRes[0])
EndFunc

Br,

UEZ

Just as a side note, what's the sense in:

If @error Or Not IsArray($aRes) Then SetError(1, 0, 0)

...line of your function?

If there is an error then there will be no array and if $aRes is not array then there is an error set. It's like saying "if wrong or not right".

This would be much more correct here:

If @error Or Not $aRes[0] Then...

[UEZ]

It is senseless, I created only the function and played with it to find out the functionality but could not find any usage for it!

Br,

UEZ

[UEZ]

It is senseless, I created only the function and played with it to find out the functionality but could not find any usage for it!

I corrected it.

Br,

UEZ

Edited March 28, 2011 by UEZ

[trancexx]

fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000

But what is the sense of fsutil file setvaliddata <size>?

Func _WinAPI_SetFileValidData($hFile, $datalength)
    Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength)
    If @error Or Not $aRes[0] Then SetError(1, 0, 0)
    Return SetError(0, 0, $aRes[0])
EndFunc

Br,

UEZ

Just as a side note, what's the sense in:

Return SetError(0, 0, $aRes[0])

...line of your function?

When entering a function @error is set to 0. If there is no error additionally set then there will be no error set (how smart), meaning error is 0 after the function returns.

This would be much more correct here:

Return $aRes[0]

[UEZ]

Of course it makes not sense to set error = 0 when there has no error occured but it it's just a habit like doing some things with the right hand...

Br,

UEZ