BitByteBit Posted February 25, 2010 Share Posted February 25, 2010 For some bizarre reason I wanted to generate the numbers from 10000000 to 99999999 and save them in a text file(with line breaks in between), now what would be the quickest way to do this I asked myself, I've tried a few methods but can't help wondering if there is a quicker way than what I have come up with. I tried using a for to loop, where I made a variable = previous numbers, the current number, and a line break. After this I thought about writing this huge variable to the text file, in hindsight that is a lot of data to write and probably not the best option. So for me the compromise was to append the file with the new number within the loop, but is this really the quickest way? Code will explain all =]. expandcollapse popup$Num = 10000000 $TimerStart = TimerInit() $FileHwnd = FileOpen(@DesktopDir & "\10 Million to 20 Million.txt", 1+8) For $Num = 10000000 To 20000000 $FileData = FileWrite($FileHwnd, $Num & @CRLF) ;ToolTip($Num) Next FileClose($FileHwnd) ;ToolTip("") $Timer = TimerDiff($TimerStart) $Timer = Round($Timer) $Timer /= 1000 $Timer /= 60 MsgBox(0,"",$Timer) #cs Method 2 $Data = "" $Num = 10000000 $begin = TimerInit() For $Num = 10000000 To 20000000 $Data = $Data & $Num & @CRLF Next $Timer = TimerDiff($begin) $Timer = Round($Timer) $Timer /= 1000 $Timer /= 60 MsgBox(0, "", $Timer) $TimerStart = TimerInit() $FileHwnd = FileOpen(@DesktopDir & "\10 Million to 20 Million.txt", 1 + 8) FileWrite($FileHwnd, $Data) FileClose($FileHwnd) $Timer = TimerDiff($TimerStart) $Timer = Round($Timer) $Timer /= 1000 $Timer = Round($Timer) $Timer /= 60 MsgBox(0, "", $Timer) #ce Link to comment Share on other sites More sharing options...
dani Posted February 25, 2010 Share Posted February 25, 2010 (edited) The for-loop you use seems perfect to me, that's what for-loops are meant to do right, loop through values known in advance How much quicker can it get? The only thing you do basically is open the file and then write many lines, which is exactly what you want. There is no need to save the result of FileWrite into $FileData though, right?~ editAs you have timed it yourself, I wonder which is faster:1) building a giant string of numbers and linebreaks & write it after the loop2) writing number & linebreak one by one during the loopHave you tested this? Edited February 25, 2010 by d4ni Link to comment Share on other sites More sharing options...
BitByteBit Posted February 25, 2010 Author Share Posted February 25, 2010 (edited) The for-loop you use seems perfect to me, that's what for-loops are meant to do right, loop through values known in advance How much quicker can it get? The only thing you do basically is open the file and then write many lines, which is exactly what you want. There is no need to save the result of FileWrite into $FileData though, right?~ editAs you have timed it yourself, I wonder which is faster:1) building a giant string of numbers and linebreaks & write it after the loop2) writing number & linebreak one by one during the loopHave you tested this?Good point about $FileData.No, I've never let it reach the end of the loop, it devours cpu cycles and I saw little things I could change.It seemed like it would take one or more hours to go to 20000000(think about going all the way up to 99999999)But lets both run a little test and see what we come up with?Are the calculations for the timer correct? Like I said, never reached the end of the loop xD Edited February 25, 2010 by BitByteBit Link to comment Share on other sites More sharing options...
danielkza Posted February 25, 2010 Share Posted February 25, 2010 (edited) At first I thought buffering would help, but seems no matter what you do, the hard drive will always be the bottleneck. Changing the buff. size from 32KB to 2MB or even 64MB made no difference whatsoever. Maybe somebody with a SSD can help you out with testing (not me as of now). Global $hFile = FileOpen(@DesktopDir & "\10 Million to 20 Million.txt", 1+8) Global $timerStart = TimerInit() Global $buffer = "" Global Const $BUFFER_LIMIT = 1024*1024 ; 2MB @ 2bytes per char For $n = 10000000 To 20000000 If StringLen($buffer) > $BUFFER_LIMIT Then ;ConsoleWrite("Buffer flushed (n="& $n & ")" & @CRLF) FileWrite($hFile, $buffer) $buffer = "" EndIf $buffer &= $n & @CRLF Next FileWrite($hFile, $buffer) FileClose($hFile) Global $timerEnd = TimerDiff($timerStart) MsgBox(0, @ScriptName, StringFormat("Elapsed time: %ds", $timerEnd/1000)) Edited February 25, 2010 by danielkza Link to comment Share on other sites More sharing options...
dani Posted February 25, 2010 Share Posted February 25, 2010 (edited) It only takes about 1 minute on my PC. It creates a 97MB .txt file, lol Takes my Scite ~ 7 seconds to open it.. $t = TimerInit() $hFile = FileOpen("test.txt", 1) For $Num = 10000000 To 20000000 FileWrite($hFile, $Num & @CRLF) Next FileClose($hFile) ConsoleWrite(TimerDiff($t)/1000 & " sec" & @CR) Output: 62.9018096533064 sec I'll try if it is faster when I first build the giant string, then use 1 FileWrite. ~ edit Much faster: $timerStart = TimerInit() $hFile = FileOpen("test.txt", 2) $OneGiantString = "" For $Num = 10000000 To 20000000 $OneGiantString &= $Num & @CRLF Next FileWrite($hFile, $OneGiantString) FileClose($hFile) $timerEnd = TimerDiff($timerStart) $roundedTime = Round($timerEnd/1000, 2) ConsoleWrite($roundedTime & " sec" & @CR) Output (besides the 97MB .txt file ): 21.78 sec It therefore seems it's all the FileWrite-actions that cause real delay. Try to avoid those, so first just build the string Edited February 25, 2010 by d4ni Link to comment Share on other sites More sharing options...
FinalVersion Posted February 25, 2010 Share Posted February 25, 2010 Also the speed of this, would depend on your CPU Speed. [center][+] Steam GUI [+][+] Clipboard Tool [+][+] System :: Uptime [+][+] StarCraft II Mouse Trap [+][/center] Link to comment Share on other sites More sharing options...
dani Posted February 25, 2010 Share Posted February 25, 2010 (edited) I also ran till 99999999, takes quite a while longer but I can probably speed it up a bit by reducing the writes: $timerStart = TimerInit() $hFile = FileOpen("test.txt", 2) $OneGiantString = "" For $Num = 10000000 To 99999999 $OneGiantString &= $Num & @CRLF If Mod($Num, 100000) = 0 Then FileWrite($hFile, $OneGiantString) $OneGiantString = "" EndIf Next FileWrite($hFile, $OneGiantString) FileClose($hFile) $timerEnd = TimerDiff($timerStart) $roundedTime = Round($timerEnd/1000, 2) ConsoleWrite($roundedTime & " sec" & @CR) 331.65 sec It now writes every 10000 numbers, but I will increase this to 10000000. See if it matters. The file is now 878 MB SciTE crashes on load ~ edit Well that didn't matter. When I used If Mod($Num, 10000000) = 0 it took roughly the same amount of time... Anyway, as no editor can open the file this seems quite useless Edited February 25, 2010 by d4ni Link to comment Share on other sites More sharing options...
BitByteBit Posted February 25, 2010 Author Share Posted February 25, 2010 (edited) Guess one day I will write the numbers to individual files, maybe going up in 10 of millions, who knows. Thank you very much for your time and input. About halfway through this exercise, I wanted to calculate roughly how long it would take. I figured I could do that by timing how long it took to do X cycles in the loop, then multiplying that time by the amount of times I would be doing it, what do you think? Edited February 25, 2010 by BitByteBit Link to comment Share on other sites More sharing options...
danielkza Posted February 25, 2010 Share Posted February 25, 2010 (edited) The following code takes 2-3s on my machine: the AutoIt version does no better than 20s.EDIT: If it helps you: 10000000 to 99999999 ~= 30s#define _CRT_SECURE_NO_WARNINGS 1 #include <cstdio> #include <cstdlib> #include <ctime> int main(int argc, char** argv) { if(argc < 4) return 1; unsigned long a = atoi(argv[1]), b = atoi(argv[2]); if(b < a) { unsigned long n = a; a = b; b = n; } time_t timeStart, timeEnd; time(&timeStart); FILE *hf = fopen(argv[3], "w"); if(hf) { for(unsigned long i=a; i<=b;++i) { fprintf(hf, "%d\n",i); } } fclose(hf); time(&timeEnd); printf("Elapsed time: %us\n", timeEnd-timeStart); return 0; } Edited February 25, 2010 by danielkza Link to comment Share on other sites More sharing options...
MvGulik Posted February 25, 2010 Share Posted February 25, 2010 (edited) whatever Edited February 7, 2011 by MvGulik "Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions.""The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014) "Believing what you know ain't so" ... Knock Knock ... Link to comment Share on other sites More sharing options...
jchd Posted February 25, 2010 Share Posted February 25, 2010 I wonder what can be a "bizarre reason" for doing that, even more for placing every number in an individual file. Really. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt) Link to comment Share on other sites More sharing options...
AgentSmith15 Posted February 25, 2010 Share Posted February 25, 2010 Have you guys looked at this topic? AutoIT Speed Tester http://www.autoitscript.com/forum/index.php?showtopic=51604 [center][/center] Link to comment Share on other sites More sharing options...
MvGulik Posted February 25, 2010 Share Posted February 25, 2010 (edited) whatever Edited February 7, 2011 by MvGulik "Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions.""The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014) "Believing what you know ain't so" ... Knock Knock ... Link to comment Share on other sites More sharing options...
jchd Posted February 25, 2010 Share Posted February 25, 2010 That is outdated "benchmark". It may have some value, but operations may see their internal implementation vary widely accross releases so what's true today may be wrong one beta later. I'm not against an ad hoc timing for fine tuning a critical part from time to time, but more important is to select the right data structure, code organization and algorithms. From then, doing $var /= 1 or $var = $var / 1 or nothing at all should be a reflex. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt) Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now