JFX Posted March 28, 2011 Share Posted March 28, 2011 (edited) Hi, I'm trying to create a big file and append data to the end of it. $hFile = FileOpen("D:\test.bin", 16+1) FileSetPos($hFile, 8 * 1024 * 1024 * 1024, 0) FileWrite($hFile, 'footer') FileClose($hFile) But it takes over 2 minutes. is there anyway to make this faster? Edited March 28, 2011 by JFX Link to comment Share on other sites More sharing options...
hannes08 Posted March 28, 2011 Share Posted March 28, 2011 Hi JFX, I wonder why you want to set the Position if you use append mode on fileopen? Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler] Link to comment Share on other sites More sharing options...
JFX Posted March 28, 2011 Author Share Posted March 28, 2011 Hi Hannes,the file does not exist and i need to write to that offset.What i want is to create a fixed VHD the fast way like this tool does vhdtoolI mostly have finished creation of the vhd footer, but the problem is how to create the big file and append the 512 Bytes to it. Link to comment Share on other sites More sharing options...
UEZ Posted March 28, 2011 Share Posted March 28, 2011 Try this:;coded by UEZ 2011 #Include <WinAPI.au3> Global $bin, $sbin For $i = 1 To 512 ;create 512 bytes of data $sbin &= $i Next $bin = Binary($sbin) ;convert is to binary $nBytes = 0 $bytes = 10 * 1024^2 ;create a 10 MB file $tBuffer = DllStructCreate("byte[" & $bytes & "]") ;create struct $hFile = _WinAPI_CreateFile("Huge.bin", 1, 4, 2) ;create 10 MB file _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $bytes, $nBytes) ;fill 10 MB with zeros _WinAPI_SetFilePointer($hFile, -512, 2) ;set file position to end of file minus 512 DllStructSetData($tBuffer, 1, $bin) ;fill 512 byte of data _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), 512, $nBytes) ;write 512 bytes of data to the end of the file _WinAPI_FlushFileBuffers($hFile) ;flush to disk _WinAPI_CloseHandle($hFile) ;close file handle $tBuffer = 0 ;release struct memoryBr,UEZ Please don't send me any personal message and ask for support! I will not reply! Selection of finest graphical examples at Codepen.io The own fart smells best! ✌Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!¯\_(ツ)_/¯ ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ Link to comment Share on other sites More sharing options...
JFX Posted March 28, 2011 Author Share Posted March 28, 2011 Thanks UEZ, but this would give me memory allocation error with bigger sizes and would not be faster because it also fills the file with zeros. Link to comment Share on other sites More sharing options...
UEZ Posted March 28, 2011 Share Posted March 28, 2011 That is the disadvantage of this method when you really want to create huge files, e.g. 5 GB and you don't have the memory for it. Let me try to find out a workaround to create huge files without allocating huge memory! Br, UEZ Please don't send me any personal message and ask for support! I will not reply! Selection of finest graphical examples at Codepen.io The own fart smells best! ✌Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!¯\_(ツ)_/¯ ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ Link to comment Share on other sites More sharing options...
JFX Posted March 28, 2011 Author Share Posted March 28, 2011 hmm, this seems to quick create a 8GB + 512 Bytes file in a second. SetFileSize("D:\test.bin", 8 * 1024^3 + 512) Func SetFileSize($FileName, $newSize) $fileHandle = _WinAPI_CreateFile($FileName, 1, 4, 4) _WinAPI_SetFilePointerEx($fileHandle, $newSize) DllCall("kernel32.dll", "long", "SetEndOfFile", "long", $fileHandle) _WinAPI_FlushFileBuffers($fileHandle) _WinAPI_CloseHandle($fileHandle) EndFunc Func _WinAPI_SetFilePointerEx($hFile, $iPos, $iMethod = 0) Local $aResult $aResult = DllCall("kernel32.dll", "dword", "SetFilePointerEx", "hwnd", $hFile, "uint64", $iPos, "uint64*", 0, "dword", $iMethod) If @error Then Return SetError(1, 0, -1) Return $aResult[0] EndFunc but i still need to write in the last 512 bytes nad this takes minutes again. Link to comment Share on other sites More sharing options...
UEZ Posted March 28, 2011 Share Posted March 28, 2011 (edited) Try this one: ;coded by UEZ 2011 #Include <WinAPI.au3> Global $bin, $sbin For $i = 1 To 512 ;create 512 bytes of data $sbin &= $i Next $bin = Binary($sbin) ;convert is to binary $nBytes = 0 $bytes = 10 * 1024^2 ;create a 10 MB file $cbytes = 1024 $restbytes = Mod($bytes, $cbytes) $tBuffer = DllStructCreate("byte[" & $cbytes & "]") ;create struct $hFile = _WinAPI_CreateFile("Huge.bin", 1, 4, 2) $bench = TimerInit() For $i = 0 To Floor($bytes / $cbytes) Step $cbytes ;create 10 MB file in 1MB steps _WinAPI_SetFilePointer($hFile, $i * $cbytes, 0) ;set file position to end of file minus 512 _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $cbytes, $nBytes) Next If $restbytes Then _WinAPI_SetFilePointer($hFile, 0, 2) _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), $restbytes, $nBytes) EndIf ConsoleWrite(Round(TimerDiff($bench), 2) & " ms to create " & $bytes & " bytes" & @CRLF) $bench = TimerInit() _WinAPI_SetFilePointer($hFile, -512, 2) ;set file position to end of file minus 512 DllStructSetData($tBuffer, 1, $bin) ;fill 512 bytes of data _WinAPI_WriteFile($hFile, DllStructGetPtr($tBuffer), 512, $nBytes) ;write 512 bytes of data to the end of the file ConsoleWrite(Round(TimerDiff($bench), 2) & " ms to add 512 bytes of data to the end of the file." & @CRLF) _WinAPI_FlushFileBuffers($hFile) ;flush to disk _WinAPI_CloseHandle($hFile) ;close file handle $tBuffer = 0 ;release struct memory My results: 272.88 ms to create 10485760 bytes 0.28 ms to add 512 bytes of data to the end of the file. Br, UEZ Edited March 28, 2011 by UEZ Please don't send me any personal message and ask for support! I will not reply! Selection of finest graphical examples at Codepen.io The own fart smells best! ✌Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!¯\_(ツ)_/¯ ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ Link to comment Share on other sites More sharing options...
ProgAndy Posted March 28, 2011 Share Posted March 28, 2011 You cannot write to an uninitialzed file, so Windows fills it with zeros which takes some time. *GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes Link to comment Share on other sites More sharing options...
JFX Posted March 28, 2011 Author Share Posted March 28, 2011 @UEZ,_WinAPI_SetFilePointer seems to have a 2GB limit and even with _WinAPI_SetFilePointerEx this will be also very slow when using with GB sizes.@ProgAndyInteresting, do you know a way to avoid this?The VHDTool here seems to do this somehow http://archive.msdn.microsoft.com/vhdtool Link to comment Share on other sites More sharing options...
ProgAndy Posted March 28, 2011 Share Posted March 28, 2011 Interesting, do you know a way to avoid this?No. Also, this is just something I concluded from testing and I have no documented proof for that. *GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes Link to comment Share on other sites More sharing options...
JFX Posted March 28, 2011 Author Share Posted March 28, 2011 (edited) Okay Problem solved,fsutil file setvaliddata saved the day Edited March 28, 2011 by JFX Link to comment Share on other sites More sharing options...
ProgAndy Posted March 28, 2011 Share Posted March 28, 2011 (edited) This function exists as DLLCall, too: SetFileValidData in kernel32.dll http://msdn.microsoft.com/en-us/library/aa365544%28v=vs.85%29.aspx PS: It will only work when the file is on an NTFS formatted partition. Edited March 28, 2011 by ProgAndy *GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes Link to comment Share on other sites More sharing options...
JFX Posted March 28, 2011 Author Share Posted March 28, 2011 Thanks I will use a check if file system is not NTFS. Better to keep fsutil, dllcall is to weird for me. Link to comment Share on other sites More sharing options...
UEZ Posted March 28, 2011 Share Posted March 28, 2011 (edited) fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000 But what is the sense of fsutil file setvaliddata <size>? Func _WinAPI_SetFileValidData($hFile, $datalength) Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength) If @error Or Not $aRes[0] Then SetError(1, 0, 0) Return $aRes[0] EndFunc Br, UEZ Edited March 28, 2011 by UEZ Please don't send me any personal message and ask for support! I will not reply! Selection of finest graphical examples at Codepen.io The own fart smells best! ✌Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!¯\_(ツ)_/¯ ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ Link to comment Share on other sites More sharing options...
trancexx Posted March 28, 2011 Share Posted March 28, 2011 fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000 But what is the sense of fsutil file setvaliddata <size>? Func _WinAPI_SetFileValidData($hFile, $datalength) Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength) If @error Or Not IsArray($aRes) Then SetError(1, 0, 0) Return SetError(0, 0, $aRes[0]) EndFunc Br, UEZ Just as a side note, what's the sense in: If @error Or Not IsArray($aRes) Then SetError(1, 0, 0) ...line of your function? If there is an error then there will be no array and if $aRes is not array then there is an error set. It's like saying "if wrong or not right". This would be much more correct here: If @error Or Not $aRes[0] Then... ♡♡♡ . eMyvnE Link to comment Share on other sites More sharing options...
UEZ Posted March 28, 2011 Share Posted March 28, 2011 It is senseless, I created only the function and played with it to find out the functionality but could not find any usage for it! Br, UEZ Please don't send me any personal message and ask for support! I will not reply! Selection of finest graphical examples at Codepen.io The own fart smells best! ✌Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!¯\_(ツ)_/¯ ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ Link to comment Share on other sites More sharing options...
UEZ Posted March 28, 2011 Share Posted March 28, 2011 (edited) It is senseless, I created only the function and played with it to find out the functionality but could not find any usage for it! I corrected it. Br, UEZ Edited March 28, 2011 by UEZ Please don't send me any personal message and ask for support! I will not reply! Selection of finest graphical examples at Codepen.io The own fart smells best! ✌Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!¯\_(ツ)_/¯ ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ Link to comment Share on other sites More sharing options...
trancexx Posted March 28, 2011 Share Posted March 28, 2011 fsutil can create also larger than 2 GB files: e.g. fsutil file createnew c:\test.bin 3145728000 But what is the sense of fsutil file setvaliddata <size>? Func _WinAPI_SetFileValidData($hFile, $datalength) Local $aRes = DllCall ("kernel32.dll", "int", "SetFileValidData", "handle", $hFile, "int64", $datalength) If @error Or Not $aRes[0] Then SetError(1, 0, 0) Return SetError(0, 0, $aRes[0]) EndFunc Br, UEZ Just as a side note, what's the sense in: Return SetError(0, 0, $aRes[0]) ...line of your function? When entering a function @error is set to 0. If there is no error additionally set then there will be no error set (how smart), meaning error is 0 after the function returns. This would be much more correct here: Return $aRes[0] ♡♡♡ . eMyvnE Link to comment Share on other sites More sharing options...
UEZ Posted March 28, 2011 Share Posted March 28, 2011 Of course it makes not sense to set error = 0 when there has no error occured but it it's just a habit like doing some things with the right hand... Br, UEZ Please don't send me any personal message and ask for support! I will not reply! Selection of finest graphical examples at Codepen.io The own fart smells best! ✌Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!¯\_(ツ)_/¯ ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now