Sign in to follow this  
Followers 0
BitByteBit

Count to 99999999

14 posts in this topic

For some bizarre reason I wanted to generate the numbers from 10000000 to 99999999 and save them in a text file(with line breaks in between), now what would be the quickest way to do this I asked myself, I've tried a few methods but can't help wondering if there is a quicker way than what I have come up with.

I tried using a for to loop, where I made a variable = previous numbers, the current number, and a line break.

After this I thought about writing this huge variable to the text file, in hindsight that is a lot of data to write and probably not the best option.

So for me the compromise was to append the file with the new number within the loop, but is this really the quickest way?

Code will explain all =].

$Num = 10000000
$TimerStart = TimerInit()
$FileHwnd = FileOpen(@DesktopDir & "\10 Million to 20 Million.txt", 1+8)

For $Num = 10000000 To 20000000
    $FileData = FileWrite($FileHwnd, $Num & @CRLF)
    ;ToolTip($Num)
Next

FileClose($FileHwnd)
;ToolTip("")
$Timer = TimerDiff($TimerStart)
$Timer = Round($Timer)
$Timer /= 1000
$Timer /= 60
MsgBox(0,"",$Timer)


#cs Method 2
$Data = ""
$Num = 10000000
$begin = TimerInit()

For $Num = 10000000 To 20000000
    $Data = $Data & $Num & @CRLF
Next

$Timer = TimerDiff($begin)
$Timer = Round($Timer)
$Timer /= 1000
$Timer /= 60
MsgBox(0, "", $Timer)
$TimerStart = TimerInit()

$FileHwnd = FileOpen(@DesktopDir & "\10 Million to 20 Million.txt", 1 + 8)
FileWrite($FileHwnd, $Data)
FileClose($FileHwnd)

$Timer = TimerDiff($TimerStart)
$Timer = Round($Timer)
$Timer /= 1000
$Timer = Round($Timer)
$Timer /= 60
MsgBox(0, "", $Timer)
#ce

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

The for-loop you use seems perfect to me, that's what for-loops are meant to do right, loop through values known in advance :mellow: How much quicker can it get? The only thing you do basically is open the file and then write many lines, which is exactly what you want. There is no need to save the result of FileWrite into $FileData though, right?

~ edit

As you have timed it yourself, I wonder which is faster:

1) building a giant string of numbers and linebreaks & write it after the loop

2) writing number & linebreak one by one during the loop

Have you tested this?

Edited by d4ni

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

The for-loop you use seems perfect to me, that's what for-loops are meant to do right, loop through values known in advance :mellow: How much quicker can it get? The only thing you do basically is open the file and then write many lines, which is exactly what you want. There is no need to save the result of FileWrite into $FileData though, right?

~ edit

As you have timed it yourself, I wonder which is faster:

1) building a giant string of numbers and linebreaks & write it after the loop

2) writing number & linebreak one by one during the loop

Have you tested this?

Good point about $FileData.

No, I've never let it reach the end of the loop, it devours cpu cycles and I saw little things I could change.

It seemed like it would take one or more hours to go to 20000000(think about going all the way up to 99999999)

But lets both run a little test and see what we come up with?

Are the calculations for the timer correct? Like I said, never reached the end of the loop xD

Edited by BitByteBit

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

At first I thought buffering would help, but seems no matter what you do, the hard drive will always be the bottleneck. Changing the buff. size from 32KB to 2MB or even 64MB made no difference whatsoever. Maybe somebody with a SSD can help you out with testing (not me as of now).

Global $hFile = FileOpen(@DesktopDir & "\10 Million to 20 Million.txt", 1+8)

Global $timerStart = TimerInit()
Global $buffer = ""

Global Const $BUFFER_LIMIT = 1024*1024 ; 2MB @ 2bytes per char

For $n = 10000000 To 20000000
    If StringLen($buffer) > $BUFFER_LIMIT Then
        ;ConsoleWrite("Buffer flushed (n="& $n & ")" & @CRLF)
        FileWrite($hFile, $buffer)
        $buffer = ""
    EndIf

    $buffer &= $n & @CRLF
Next
FileWrite($hFile, $buffer)
FileClose($hFile)

Global $timerEnd = TimerDiff($timerStart)
MsgBox(0, @ScriptName, StringFormat("Elapsed time: %ds", $timerEnd/1000))
Edited by danielkza

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

It only takes about 1 minute on my PC. It creates a 97MB .txt file, lol :lol: Takes my Scite ~ 7 seconds to open it..

$t = TimerInit()
$hFile = FileOpen("test.txt", 1)
For $Num = 10000000 To 20000000
    FileWrite($hFile, $Num & @CRLF)
Next
FileClose($hFile)
ConsoleWrite(TimerDiff($t)/1000 & " sec" & @CR)

Output:

62.9018096533064 sec

I'll try if it is faster when I first build the giant string, then use 1 FileWrite.

~ edit

Much faster:

$timerStart = TimerInit()
$hFile = FileOpen("test.txt", 2)
$OneGiantString = ""
For $Num = 10000000 To 20000000
    $OneGiantString &= $Num & @CRLF
Next
FileWrite($hFile, $OneGiantString)
FileClose($hFile)

$timerEnd = TimerDiff($timerStart)
$roundedTime = Round($timerEnd/1000, 2)
ConsoleWrite($roundedTime & " sec" & @CR)

Output (besides the 97MB .txt file :():

21.78 sec

It therefore seems it's all the FileWrite-actions that cause real delay. Try to avoid those, so first just build the string :mellow:

Edited by d4ni

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

I also ran till 99999999, takes quite a while longer but I can probably speed it up a bit by reducing the writes:

$timerStart = TimerInit()
$hFile = FileOpen("test.txt", 2)
$OneGiantString = ""
For $Num = 10000000 To 99999999
    $OneGiantString &= $Num & @CRLF
    If Mod($Num, 100000) = 0 Then
        FileWrite($hFile, $OneGiantString)
        $OneGiantString = ""
    EndIf
Next
FileWrite($hFile, $OneGiantString)
FileClose($hFile)

$timerEnd = TimerDiff($timerStart)
$roundedTime = Round($timerEnd/1000, 2)
ConsoleWrite($roundedTime & " sec" & @CR)

331.65 sec

It now writes every 10000 numbers, but I will increase this to 10000000. See if it matters. The file is now 878 MB :mellow: SciTE crashes on load :lol:

~ edit

Well that didn't matter. When I used If Mod($Num, 10000000) = 0 it took roughly the same amount of time... Anyway, as no editor can open the file this seems quite useless :(

Edited by d4ni

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Guess one day I will write the numbers to individual files, maybe going up in 10 of millions, who knows.

Thank you very much for your time and input.

About halfway through this exercise, I wanted to calculate roughly how long it would take.

I figured I could do that by timing how long it took to do X cycles in the loop, then multiplying that time by the amount of times I would be doing it, what do you think?

Edited by BitByteBit

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

The following code takes 2-3s on my machine: the AutoIt version does no better than 20s.

EDIT: If it helps you: 10000000 to 99999999 ~= 30s

#define _CRT_SECURE_NO_WARNINGS 1

#include <cstdio>
#include <cstdlib>

#include <ctime>

int main(int argc, char** argv)
{
    if(argc < 4)
        return 1;

    unsigned long a = atoi(argv[1]), b = atoi(argv[2]);
    if(b < a) {
        unsigned long n = a;
        a = b;
        b = n;
    }

    time_t timeStart, timeEnd;
    time(&timeStart);

    FILE *hf = fopen(argv[3], "w");
    if(hf) {
        for(unsigned long i=a; i<=b;++i) {
            fprintf(hf, "%d\n",i);
        }
    }
    fclose(hf);

    time(&timeEnd);
    printf("Elapsed time: %us\n", timeEnd-timeStart);

    return 0;
}
Edited by danielkza

Share this post


Link to post
Share on other sites

#10 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

I wonder what can be a "bizarre reason" for doing that, even more for placing every number in an individual file. Really.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Have you guys looked at this topic?

AutoIT Speed Tester

http://www.autoitscript.com/forum/index.php?showtopic=51604

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

That is outdated "benchmark". It may have some value, but operations may see their internal implementation vary widely accross releases so what's true today may be wrong one beta later.

I'm not against an ad hoc timing for fine tuning a critical part from time to time, but more important is to select the right data structure, code organization and algorithms. From then, doing $var /= 1 or $var = $var / 1 or nothing at all should be a reflex.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0