corz

Code Gotcha! + Bonus Fast, Low Memory StringInFile()

13 posts in this topic

#1 ·  Posted (edited)

Running this function here (in this simple test script) takes 16 seconds. The exact same function running inside my program takes over 22 seconds! WT*?!? This bamboozled me for a bit.

After some trial and error I discovered what was making the difference and thought I might share this nugget which I will keep in mind for future intensive operations and if possible, set/reset at each side of said operation!

In short, one, or rather two (each has an effect) common AutoIt script settings will cause some intensive code operations to run more slowly. The example searches in millions of lines of text from a file.

I wasn't quite sure where to post this, so I thought I'd leave in the actual function and put it in example scripts as a bonus! It's a fast, low-memory way to find a string inside any size of file.
 

; First, nice and fast..

;AutoItSetOption("TrayOnEventMode", 0)
;AutoItSetOption("GUIOnEventMode", 0)

;;---------- RUN x64 ----------
;;Searching for : ->one million a characters.txt
;;Time =>16.23 seconds
;;.\StringInFileSimpleTest.au3 (46) : ==> $found_array:
;;[0]   [0] = 100084        [1] = This is a line which contains the string "one million a characters.txt"
;;[1]   [0] = 100085        [1] = This is another line which contains the string "one million a characters.txt"




; uncomment the following two lines for almost 40% slower times!

;AutoItSetOption("TrayOnEventMode", 1)
;AutoItSetOption("GUIOnEventMode", 1)

;;---------- RUN x64 ----------
;;Searching for : ->one million a characters.txt
;;Time =>22.63 seconds
;;.\StringInFileSimpleTest.au3 (46) : ==> $found_array:
;;[0]   [0] = 100084        [1] = This is a line which contains the string "one million a characters.txt"
;;[1]   [0] = 100085        [1] = This is another line which contains the string "one million a characters.txt"



; init..
$string = "one million a characters.txt"        ; a string to search for (this happens to be a file name)
$test_file = "B:\Test\2,040,000 lines.data"     ; 2,040,000 longish (ave' 70 chr) lines of text in this file
$begin_time = TimerInit()



;; this would fix it, if your program UI design allows..
;local $old_tray_mode = AutoItSetOption("TrayOnEventMode", 0)
;local $old_menu_mode = AutoItSetOption("GUIOnEventMode", 0)

$found_array = StringInFileSimple($test_file, $string, default, true) ; (must search to end of file)
ConsoleWrite("Time =>" & Round(TimerDiff($begin_time)/1000, 2) & " seconds")

;; revert back to previous settings..
;AutoItSetOption("TrayOnEventMode", $old_tray_mode)
;AutoItSetOption("GUIOnEventMode", $old_menu_mode)


;if $do_debug = $ON then PrintArray($found_array,  "$found_array", 0, @ScriptLineNumber);debug





;
; StringInFileSimple()
;
; A low memory way to find a string inside any size of file.
; This is also fairly fast and handles UTFx strings & files.
;
; In its simplest form, you send a file path and a string and it returns the
; line number where the string was first found, or false if not found in the file.
;
; You can force this function to return an array by passing the third param as
; true. It is a 2D array with one main element which looks like this:
;
;        [0][0] = {int} line number of matching line, [0][1] = {string} entire line containing found string
;
; This is handy when you need to return the entire found line.
;
; It will also return an array if you set the fourth parameter, return_all=true,
; which returns ALL the matches found in the file, in the same two-dimensional
; array format as before, elements starting at 0, e.g.:
;
;
;        [0][0] = line number of matching line,        [0][1] = entire line containing found string
;        [1][0] = line number of 2nd matching line,    [1][1] = entire line containing found string
;        [2][0] = line number of 3rd matching line,    [2][1] = entire line containing found string
;        [3][0] = etc.
;
; This was designed to search within a file for relative or absolute file paths
; using a single search operation, but works great for any string.
;
func StringInFileSimple($file_path, $string_to_find, $return_array=false, $return_all=false)

    local $line_number = false
    local $file_handle = FileOpen($file_path, 0) ; or make a param out of the "0", or omit.

    if $return_all then $return_array = true
    if $return_array then
        local $array_count = 1
        local $ret_array[$array_count][2]
    endif

    local $this_line

    while true

        local $found = false

        $this_line = FileReadLine($file_handle)
        if @error then
            $line_number = false
            exitloop
        endif

        $line_number += 1

        if StringInStr($this_line, $string_to_find) then

            $found = true

            if $return_array then
                $ret_array[$array_count-1][0] = $line_number
                $ret_array[$array_count-1][1] = $this_line
            endif

            if $return_all then
                $array_count += 1
                redim $ret_array[$array_count][2]
            else
                exitloop
            endif

        endif
    wend

    FileClose($file_handle)

    if $return_array then
        if $return_all then redim $ret_array[$array_count-1][2] ; chop off last (empty) element
        return $ret_array
    endif

    if $found then
        return $line_number
    endif

    return false

endfunc

; I also have a StringInFile() which allows you to optionally pass an array of
; search terms. But even asking if $x = true on each iteration of a 1,000,000
; lines can cost you a second, so this is the "simple" version. All speed-up
; tips for this code warmly welcomed!

So there you have it!

;o) Cor

Edited by corz
3 people like this

nothing is foolproof to the sufficiently talented fool..

Share this post


Link to post
Share on other sites



Hey, I didn't expect a Nobel for this, but BEJEEEZUS!FuxnFuk! Not only one but two pretty startling AutoIt discoveries (and workarounds) but a free super-useful, super-fast function to boot! Six months, nada.

You are a dry bunch!

;o) Cor

1 person likes this

nothing is foolproof to the sufficiently talented fool..

Share this post


Link to post
Share on other sites

What can I say?  I have a limited intelligence and I've used it in a particular direction.

I try not to do exhaustive searches for strings.  There are reasons to do them, but don't normally write that type of Application.  We're all busy.  I'm busy, you're hopefully busy.  Actually I'll try to remember this if I find a use for it.  Thank you for sharing.

Share this post


Link to post
Share on other sites

There is a far better approach than this and before you ask, no I won't.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

JohnOne, the last time I looked, the functions on the forum used gobs of memory to do the exact same thing. If I missed another, better method, feel free to point me in that direction.

And I've not come across any mention of these gotchas before, ergo, they are new.

guinness, why not?

 

;o) Cor

Edited by corz

nothing is foolproof to the sufficiently talented fool..

Share this post


Link to post
Share on other sites

It's a function for searching through a file line by line, there are many example of that in the forum.

No idea what you mean by a gotcha, unless you are again implying that your code is super special.

It's not, it's bog standard code.


AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

There is a faster option.  rutine in assembly.

 

Saludos

Edited by Danyfirex

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

I guess nobody really saw what you were referring to... interesting discovery ;)...

Local $test, $timer
For $z = 1 To 10
    For $y = 0 To 1
        For $x = 0 To 1
            AutoItSetOption("TrayOnEventMode", $y)
            AutoItSetOption("GUIOnEventMode", $x)
            $timer = TimerInit()
            For $i = 0 To 100000
                $test = 100 ^ 2
            Next
            ConsoleWrite($y & "x" & $x & @TAB & TimerDiff($timer) & @CRLF)
        Next
    Next
    ConsoleWrite(@CRLF)
Next

 

Edited by KaFu

Share this post


Link to post
Share on other sites

Wow, very interesting indeed.  I need to go check a few of my scripts...

Share this post


Link to post
Share on other sites

corz,

Perhaps if you had bothered to explain your "discovery" in the OP, you might have got more responses. Now KaFu has pointed it out I must say that I find it very interesting and makes me glad I use MessageLoop mode almost exclusively.

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

I think that the difference in speed of the loops is minimal, but measurable. I've also noticed that how you set the OnEvent mode alters the loop differences, if you change the two inner loops ($x and $y) to 1 to 0 Step -1 sometimes OnEvent mode goes faster than with it turned off.

AutoIt has a problem with making accurate timing measurements of script execution below a certain point, the order in which you do things can show differences where there aren't any or they're a lot smaller than if you run the exact same code but move things around in it.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now