Jump to content

For Next Loops Speed


Go to solution Solved by Jos,

Recommended Posts

So I have array of data and an index of lines that need to some follow up data, some of the lines maybe duplicated so I am doing stringregexp 3 and then returning data and then doing another FOR NEXT with the result of that, Some lines are not duplicated so the Loop is 0 to 0, so it runs through once, but I noticed the less loops the more time needed. Which I do not understand. So why the increase time, and any ideas for work arounds3w3fes5245w vcre45tfgdeedefdfdffdfr343w3w5edvggegsdacZ

Here is some times from script 

first number is the second number in for statement 

Ex:For $i = 0 to 0

0 - 2954.91724688962

2 - 6.81581250237796
2 - 6.30711866986265
0 - 2905.05535897729
0 - 2900.23817676825
I made example code and here the results
3 - 34.9009790513521
2 - 70.7139274168353
0 - 233.322919074548
0 - 231.709153067621
0 - 231.155909447909
1 - 122.447368578827
 
Here is my example code 
#include <array.au3>
Global $main[1600][8],$index = '<>', $num
For $i = 1 to 1000
    $main[$i][0] = 'somedata'
    $main[$i][1] = 'somedata'
    _randomdata($i)
    $main[$i][3] ='somedata'
    $main[$i][4] ='somedata'
    $main[$i][5] ='somedata'
    $main[$i][6] ='somedata'
Next
_ArrayDisplay($main)
$num = StringSplit($num, '|')
MsgBox(0,'',$index)
For $i = 1 to $num[0]
    $st = TimerInit()
    $arr = StringRegExp($index,'.+?'&$num[$i]&'\<(\d+?)\>',3)
    If Not IsArray($arr) Then ContinueLoop
    $r = Ubound($arr)-1
    For $f = 0 to $r
        $main[$arr[$f]][7] = $main[$arr[$f]][2]
    Next
ConsoleWrite($r & ' - ' & TimerDiff($st) & @CRLF)
Next
_ArrayDisplay($main)
Func _Randomdata($n)
    $x = Random(1,2,1)
    If $x = 2 Then
         $r = Random(100000,100500,1)
         $main[$n][2] = $r
        $index &= $r & '<'&$n&'>'
        If Not StringInStr($num,$r) Then $num &= $r & '|'
    EndIf
EndFunc

 

Link to comment
Share on other sites

  • Moderators

step887,

As clear as mud - and the code is no help. Try explaining again and adding some comments to the script so we can get an idea of what is supposed to be happening. ;)

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to comment
Share on other sites

  • Developers

It is a bit of a complex way of showing that  StringRegExp() takes longer when nothing is found than when it returns something.

Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

  • Developers

easier example:

$num = "<>100231<4>100159<7>100480<8>100407<15>100423<16>100372<17>100090<18>100202<21>100330<23>100091<24>100352<25>100430<26>100238<28>100221"
For $x = 1 to 6
        $num &= $num
Next
$st = TimerInit()
$arr = StringRegExp($num,'.+?100423\<(\d+?)\>',3)
ConsoleWrite(Ubound($arr) & ' - ' & TimerDiff($st) & @CRLF)

$st = TimerInit()
$arr = StringRegExp($num,'.+?999999\<(\d+?)\>',3)
ConsoleWrite(Ubound($arr) & ' - ' & TimerDiff($st) & @CRLF)

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

  • Developers
  • Solution

I am not a RegEx person, but does this give the same result but faster?

$arr = StringRegExp($index,'.*?'&$num[$i]&'\<(\d*?)\>',3)

Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

step887,

As clear as mud - and the code is no help. Try explaining again and adding some comments to the script so we can get an idea of what is supposed to be happening. ;)

M23

Sorry I try to clear, basically I have array of files and column 2 is MD5 and if that is populated I use to look up data on that file and put in column 7 (8)

but sometimes there is same md5 in the array of files, so I will populate the same data to all lines with the same md5.  I have commented on the code below.

 

Should this line $num = StringSplit($num, '|') be  $num = StringSplit($main, '|')  ??

No $num is a lists of numbers I used in column 2 and need go through the array and add data to column 7

It is a bit of a complex way of showing that  StringRegExp() takes longer when nothing is found than when it returns something.

Jos

Actually StringregExp flag 3 is returning at least one result, and since you mentioned this I updated the code and it is Stringregexp that is taken the time.  Is there a workaround for this?

first number is num of returns - 1 because I am using ubound, SRE is time difference after the reg exp and total is after the For  

1 SRE: 202.508481847547 total: 202.540124318271
0 SRE: 222.228402681416 total: 222.256714365748
0 SRE: 221.022324928871 total: 221.050303534564
1 SRE: 22.6979769469612 total: 22.7189609012309
0 SRE: 218.70642915051 total: 218.735073913481
 
Updated code with explanation 
#include <array.au3>
Global $main[1600][8],$index = '<>', $num
For $i = 1 to 1000
    $main[$i][0] = 'somedata'
    $main[$i][1] = 'somedata'
    _randomdata($i)
    $main[$i][3] ='somedata'
    $main[$i][4] ='somedata'
    $main[$i][5] ='somedata'
    $main[$i][6] ='somedata'
Next
;populating data in my array
;if something is column 2, I need to follow up and add more data into column 7
_ArrayDisplay($main)
$num = StringSplit($num, '|');this is a index of the numbers used on column 2
MsgBox(0,'',$index); here is the index
For $i = 1 to $num[0]; so this goes through all the numbers used in column 2
    $st = TimerInit()
    $arr = StringRegExp($index,'.+?'&$num[$i]&'\<(\d+?)\>',3); checks the index and returns everytime a number is used in main

    If Not IsArray($arr) Then ContinueLoop; no matches
    $r = Ubound($arr)-1; number of returns -1
        ConsoleWrite($r & ' SRE: '  & TimerDiff($st))
    For $f = 0 to $r
        $main[$arr[$f]][7] = $main[$arr[$f]][2];writes data the to all numbers used.
    Next
ConsoleWrite(' total: ' & TimerDiff($st) & @CRLF); so this writes how many times
Next
_ArrayDisplay($main)
Func _Randomdata($n)
    $x = Random(1,2,1);use this to writ 50% of the clls
    If $x = 2 Then
         $r = Random(100000,100500,1); assign one of 500 numbers
         $main[$n][2] = $r
        $index &= $r & '<'&$n&'>'
        If Not StringInStr($num,$r) Then $num &= $r & '|'; keep a list of numbers used
    EndIf
EndFunc
2 SRE: 16.0430657357341 total: 16.0647158472821
Link to comment
Share on other sites

  • Developers

 

.  Is there a workaround for this?

 

Did you see the post above your last one?

Edited by Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

I am not a RegEx person, but does this give the same result but faster?

$arr = StringRegExp($index,'.*?'&$num[$i]&'\<(\d*?)\>',3)

Jos

that works

0 SRE: 0.315758549962129 total: 0.380042727327837
3 SRE: 0.272791405505257 total: 0.307098505342914
1 SRE: 0.263798282246842 total: 0.287446865630082
2 SRE: 0.253472844431625 total: 0.279119899650068
3 SRE: 0.273124484144458 total: 0.303767718950909

 

Thanks a lot Jos, so I guess the question is what is the difference between + and * in regex?

Link to comment
Share on other sites

  • Moderators

Regex generally makes my eyes bleed, but I thought + requires at least one character to match, where * does not.

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites

  • Developers

Thanks a lot Jos, so I guess the question is what is the difference between + and * in regex?

The difference is explained in the Helpfile: +? = 1 or more Lazy; *? = 0 or more Lazy, but I have no idea why the difference is so big.

I am sure somebody will stop by that understands :)

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

Your sample code is overly complex to show your point.

Notice that neither .+? nor .*? are necessary: these sub-pattern just waste cycles for no benefit.Worse, .+? may cause duplicates to be missed.

Now to the point: PCRE (the regexp engine used by AutoIt) performs complex initial optimizations by default. It is possible to obtain the bytecode generated by the internal pattern compiler but it isn't worse the pain in this case.

As a general rule of thumb, avoid .+ and .* as much as possible for efficiency.

Try this version:

$arr = StringRegExp($index, $num[$i] & '\<(\d+)\>', 3); checks the index and returns everytime a number is used in main

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...