rossnixon

My script runs too slow - any alternative?

14 posts in this topic

My script works. It strips off the first 9 lines of each ASCII file (*.mif) in a folder.
But it is incredibly slow on large files. I previously used http://www.ecobyte.com/replacetext/ - but it won't work on Win 7 64-bit
Does anyone know of a way to speed up my code, or should I look for another program?

 

strip9.au3

Share this post


Link to post
Share on other sites



#Include <File.au3>
Dim $text

$FileList = _FileListToArray(@ScriptDir, "*.MIF", 1)
If @error = 4 Then
    MsgBox(0, "", "No MIF Files Found.")
    Exit
EndIf

For $i = 1 To $FileList[0]
    ProcessFile($FileList[$i])
Next
MsgBox(0, "", "Updated " & $i-1 & " MIF files.")

Func ProcessFile($filefromlist)

    $MaxCharsPerLine = 256
    $sFile = FileRead($filefromlist , $MaxCharsPerLine * 9)
    $aFile = stringsplit($sFile , @LF , 2)
    $hFile = FileOpen($filefromlist , 2)

      For $j = 0 To 8
         FileWriteLine($hFile , $aFile[$j] & @LF)
      Next

EndFunc   ;==>ProcessFile

 


,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Without posting your code you will get no help ...

EDIT: try this

Func ProcessFile($filefromlist)
    $text = FileRead($filefromlist)
    $i = StringInStr($text, @CRLF, 0, 9) ; position of 9th CRLF
    $text = StringMid($text, $i+2) ; skip first 9 lines, +2 to skip found 9th CRLF
      ;Write the new MIF file.
      $file = FileOpen($filefromlist,2)
      FileWrite($file, $text)
      FileClose($file)
EndFunc   ;==>ProcessFile

 

Edited by Zedna
1 person likes this

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

zed, its attached...

ross, you can just use code tags and paste it in, that is how its usually done

Edited by boththose

,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

Maybe load the file to an array (_FileReadToArray), edit the array, and then rewrite it to disk using _FileWriteFromArray?

Share this post


Link to post
Share on other sites

Without posting your code you will get no help ...

EDIT: try this

Func ProcessFile($filefromlist)
    $text = FileRead($filefromlist)
    $i = StringInStr($text, @CRLF, 0, 9) ; position of 9th CRLF
    $text = StringMid($text, $i+2) ; skip first 9 lines, +2 to skip found 9th CRLF
      ;Write the new MIF file.
      $file = FileOpen($filefromlist,2)
      FileWrite($file, $text)
      FileClose($file)
EndFunc   ;==>ProcessFile

 

​Great! And runs fast!
I only had to change $i to $j, as I was already using $i
Thanks very much Zedna! :-)

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

ahh i was backward, since you are already including file this would also do it

 

 

#Include <File.au3>
Dim $text

$FileList = _FileListToArray(@ScriptDir, "*.MIF", 1)
If @error = 4 Then
    MsgBox(0, "", "No MIF Files Found.")
    Exit
EndIf

For $i = 1 To $FileList[0]
    ProcessFile($FileList[$i])
Next
MsgBox(0, "", "Updated " & $i-1 & " MIF files.")

Func ProcessFile($filefromlist)

      for $i = 1 to 9
      _FileWriteToLine($filefromlist , $i , "" , 1)
      next

EndFunc   ;==>ProcessFile

 

 

Edited by boththose
back to functional

,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

Here is a regexpreplace way, which could be faster :

Func ProcessFile($sFilename)
    Local $sContent = FileRead($sFilename)
    If @error Then Return SetError(1, 0, 0)
    
    $sContent = StringRegExpReplace($sContent, "^(?:\N*\R){1,9}", "")
    
    Local $hFileW = FileOpen($sFilename, 2)
    If $hFileW = -1 Then Return SetError(1, 0, 0)
    
    FileWrite($hFileW, $sContent)
    FileClose($hFileW)
    Return 1
EndFunc

 

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

@jquinch

StringRegExpReplace() in this case on such big files/texts will be definitely much slower than one simple StringInStr() + StringMid()

EDIT: Anyway thanks for nice example for RegExp pattern ...

Edited by Zedna

Share this post


Link to post
Share on other sites

Try this:

Func FileStripFirstNLines($sFile, $iLines = 9)
    Local $hFile = FileOpen($sFile)
    If @error Then Return SetError(1, 0, 0)
    Local $sResult, $i
    For $i = 1 To $iLines
        $sResult &= FileReadLine($hFile, $i) & @CRLF
    Next
    FileClose($hFile)
    Return $sResult
EndFunc

For a 166 MB file it takes around 0.81 milli seconds to read first 9 lines.


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

Then I misunderstood op.

 

Sorry.


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

#14 ·  Posted (edited)

Ok, here another version:

Func FileCutFirstNLines($sFile, $iLines = 9)
    Local $hFile = FileOpen($sFile)
    If @error Then Return SetError(1, 0, 0)
    Local $sResult, $i
    For $i = 1 To $iLines
        $sResult &= FileReadLine($hFile, $i) & @CRLF
    Next
    FileClose($hFile)
    Local $iLength = StringLen($sResult), $iFilesize = FileGetSize($sFile), $nBytes, $iDelta = $iFilesize - $iLength
    Local $tBuffer = DllStructCreate("byte[" & $iDelta & "]")
    $hFile = _WinAPI_CreateFile($sFile, 2, 2)
    _WinAPI_SetFilePointer($hFile, $iLength)
    _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer), $iDelta, $nBytes)
    _WinAPI_CloseHandle($hFile)
    $hFile = FileOpen(StringTrimRight($sFile, 4) & "_new." & StringRight($sFile, 3) , 2)
    FileWrite($hFile, DllStructGetData($tBuffer, 1))
    FileClose($hFile)
    $tBuffer = 0
    Return 1
EndFunc

Takes around 1.7 seconds to create the new file (~170 MB).

 

The higher the $iLines value the slower the runtime.

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now