Jump to content
Sign in to follow this  
KiwiCoder

Faster file processing

Recommended Posts

KiwiCoder

Hello again helpful autoit community :mellow:

I would appreciate it if someone can help me optimize this and make it run faster.

this turns my non-comma separated text files into comma separated ones and adds a comma after each two chars.

this code must be ugly because it takes 40 mins to process 500k file! and im looking forward to process 100mb files!!

also what would be the best way to deal with large files? put them in array's?

thank you

#include <File.au3>
#include<string.au3>
HotKeySet("{ESC}", "Terminate")

_LoadMainInput("Numerical-4.txt")
Func _LoadMainInput($File)
    $MList = FileRead($File)
    $len = StringLen($MList)
    For $j = 0 To $len * 2
        $j += 2
        ConsoleWrite($j & "," & $len * 2 & @CRLF)
        $MList = _StringInsert($MList, ",", $j)
    Next
    FileWrite("Num2Comma.txt", $MList)
    Return $MList
EndFunc   ;==>_LoadMainInput

Func Terminate()
    Exit 0
EndFunc   ;==>Terminate

the consolewrite() is important to make sure its not going to take double the age of the universe to process the file.

Edited by KiwiCoder

Share this post


Link to post
Share on other sites
hannes08

Func _LoadMainInput($File)
    $fh_in = FileOpen($File)
    $fh_out = FileOpen("Output.txt", 2)
    While 1
        $twochars = FileRead($fh_in, 2)
        If @error = -1 Then ExitLoop
        FileWrite($fh_out, $twochars & ",")
    WEnd
    FileClose($fh_in)
    FileClose($fh_out)
EndFunc   ;==>_LoadMainInput

... Takes about 3 seconds on my PC for a 500kb file. :mellow:


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites
KiwiCoder

Func _LoadMainInput($File)
    $fh_in = FileOpen($File)
    $fh_out = FileOpen("Output.txt", 2)
    While 1
        $twochars = FileRead($fh_in, 2)
        If @error = -1 Then ExitLoop
        FileWrite($fh_out, $twochars & ",")
    WEnd
    FileClose($fh_in)
    FileClose($fh_out)
EndFunc   ;==>_LoadMainInput

... Takes about 3 seconds on my PC for a 500kb file. :)

haha awesome m8

this is magic! you saved my time!!

EDIT: nvm dont care how it works as long as it works... :mellow:

Edited by KiwiCoder

Share this post


Link to post
Share on other sites
hannes08

I'm sure there's even a faster way than that.


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites
hannes08

Here it is:

Func _LoadMainInput($File)
    $s_t = ""
    Local $a_content
    _FileReadToArray($File, $a_content)
    For $i = 1 To $a_content[0]
        For $j = 1 To StringLen($a_content[$i]) Step 2
            $s_t &= StringMid($a_content[$i], $j, 2) & ","
        Next
        $s_t &= @CRLF
    Next
    $fh_out = FileOpen("Output.txt", 2)
    FileWrite($fh_out, $s_t)
    FileClose($fh_out)
EndFunc   ;==>_LoadMainInput

Same file 590ms vs 3000ms of prior version.


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites
Blue_Drache

Nah, just bored.


Lofting the cyberwinds on teknoleather wings, I am...The Blue Drache

Share this post


Link to post
Share on other sites
SmOke_N

Curious, how does this compare?

Global $gs_FileToConvert = "PutFileHere"
Global $gs_FileToConvertTo = "PutFileHere"

_AddDelimToFile($gs_FileToConvert, $gs_FileToConvertTo)

Func _AddDelimToFile($s_infile, $s_outfile, $s_delim = ",", $n_everxchars = 2)
    Local $s_fread = FileRead($s_infile)
    Local $h_outfile = FileOpen($s_outfile, 2)
    FileWrite($h_outfile, StringRegExpReplace($s_fread, "(.{" & $n_everxchars & "}(?!(?:\z|\v+)))", "$1,"))
    FileClose($h_outfile)
    Return
EndFunc
Edited by SmOke_N

Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites
hannes08

Nah, just bored.

:mellow::)


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites
hannes08

Curious, how does this compare?

Global $gs_FileToConvert = "PutFileHere"
Global $gs_FileToConvertTo = "PutFileHere"

_AddDelimToFile($gs_FileToConvert, $gs_FileToConvertTo)

Func _AddDelimToFile($s_infile, $s_outfile, $s_delim = ",", $n_everxchars = 2)
    Local $s_fread = FileRead($s_infile)
    Local $h_outfile = FileOpen($s_outfile, 2)
    FileWrite($h_outfile, StringRegExpReplace($s_fread, "(.{" & $n_everxchars & "}(?!(?:\z|\v+)))", "$1,"))
    FileClose($h_outfile)
    Return
EndFunc

400ms :mellow:

RegEx is still a big mystery for me! :)


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.