Sign in to follow this  
Followers 0
KiwiCoder

Faster file processing

10 posts in this topic

#1 ·  Posted (edited)

Hello again helpful autoit community :mellow:

I would appreciate it if someone can help me optimize this and make it run faster.

this turns my non-comma separated text files into comma separated ones and adds a comma after each two chars.

this code must be ugly because it takes 40 mins to process 500k file! and im looking forward to process 100mb files!!

also what would be the best way to deal with large files? put them in array's?

thank you

#include <File.au3>
#include<string.au3>
HotKeySet("{ESC}", "Terminate")

_LoadMainInput("Numerical-4.txt")
Func _LoadMainInput($File)
    $MList = FileRead($File)
    $len = StringLen($MList)
    For $j = 0 To $len * 2
        $j += 2
        ConsoleWrite($j & "," & $len * 2 & @CRLF)
        $MList = _StringInsert($MList, ",", $j)
    Next
    FileWrite("Num2Comma.txt", $MList)
    Return $MList
EndFunc   ;==>_LoadMainInput

Func Terminate()
    Exit 0
EndFunc   ;==>Terminate

the consolewrite() is important to make sure its not going to take double the age of the universe to process the file.

Edited by KiwiCoder

Share this post


Link to post
Share on other sites



Func _LoadMainInput($File)
    $fh_in = FileOpen($File)
    $fh_out = FileOpen("Output.txt", 2)
    While 1
        $twochars = FileRead($fh_in, 2)
        If @error = -1 Then ExitLoop
        FileWrite($fh_out, $twochars & ",")
    WEnd
    FileClose($fh_in)
    FileClose($fh_out)
EndFunc   ;==>_LoadMainInput

... Takes about 3 seconds on my PC for a 500kb file. :mellow:


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Func _LoadMainInput($File)
    $fh_in = FileOpen($File)
    $fh_out = FileOpen("Output.txt", 2)
    While 1
        $twochars = FileRead($fh_in, 2)
        If @error = -1 Then ExitLoop
        FileWrite($fh_out, $twochars & ",")
    WEnd
    FileClose($fh_in)
    FileClose($fh_out)
EndFunc   ;==>_LoadMainInput

... Takes about 3 seconds on my PC for a 500kb file. :)

haha awesome m8

this is magic! you saved my time!!

EDIT: nvm dont care how it works as long as it works... :mellow:

Edited by KiwiCoder

Share this post


Link to post
Share on other sites

I'm sure there's even a faster way than that.


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

Here it is:

Func _LoadMainInput($File)
    $s_t = ""
    Local $a_content
    _FileReadToArray($File, $a_content)
    For $i = 1 To $a_content[0]
        For $j = 1 To StringLen($a_content[$i]) Step 2
            $s_t &= StringMid($a_content[$i], $j, 2) & ","
        Next
        $s_t &= @CRLF
    Next
    $fh_out = FileOpen("Output.txt", 2)
    FileWrite($fh_out, $s_t)
    FileClose($fh_out)
EndFunc   ;==>_LoadMainInput

Same file 590ms vs 3000ms of prior version.


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

Hannes, are you a wizard?

Share this post


Link to post
Share on other sites

Nah, just bored.


Lofting the cyberwinds on teknoleather wings, I am...The Blue Drache

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Curious, how does this compare?

Global $gs_FileToConvert = "PutFileHere"
Global $gs_FileToConvertTo = "PutFileHere"

_AddDelimToFile($gs_FileToConvert, $gs_FileToConvertTo)

Func _AddDelimToFile($s_infile, $s_outfile, $s_delim = ",", $n_everxchars = 2)
    Local $s_fread = FileRead($s_infile)
    Local $h_outfile = FileOpen($s_outfile, 2)
    FileWrite($h_outfile, StringRegExpReplace($s_fread, "(.{" & $n_everxchars & "}(?!(?:\z|\v+)))", "$1,"))
    FileClose($h_outfile)
    Return
EndFunc
Edited by SmOke_N

Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites

Nah, just bored.

:mellow::)


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

Curious, how does this compare?

Global $gs_FileToConvert = "PutFileHere"
Global $gs_FileToConvertTo = "PutFileHere"

_AddDelimToFile($gs_FileToConvert, $gs_FileToConvertTo)

Func _AddDelimToFile($s_infile, $s_outfile, $s_delim = ",", $n_everxchars = 2)
    Local $s_fread = FileRead($s_infile)
    Local $h_outfile = FileOpen($s_outfile, 2)
    FileWrite($h_outfile, StringRegExpReplace($s_fread, "(.{" & $n_everxchars & "}(?!(?:\z|\v+)))", "$1,"))
    FileClose($h_outfile)
    Return
EndFunc

400ms :mellow:

RegEx is still a big mystery for me! :)


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0