Jump to content
Sign in to follow this  
step887

For Next Loops Speed

Recommended Posts

step887

So I have array of data and an index of lines that need to some follow up data, some of the lines maybe duplicated so I am doing stringregexp 3 and then returning data and then doing another FOR NEXT with the result of that, Some lines are not duplicated so the Loop is 0 to 0, so it runs through once, but I noticed the less loops the more time needed. Which I do not understand. So why the increase time, and any ideas for work arounds3w3fes5245w vcre45tfgdeedefdfdffdfr343w3w5edvggegsdacZ

Here is some times from script 

first number is the second number in for statement 

Ex:For $i = 0 to 0

0 - 2954.91724688962

2 - 6.81581250237796
2 - 6.30711866986265
0 - 2905.05535897729
0 - 2900.23817676825
I made example code and here the results
3 - 34.9009790513521
2 - 70.7139274168353
0 - 233.322919074548
0 - 231.709153067621
0 - 231.155909447909
1 - 122.447368578827
 
Here is my example code 
#include <array.au3>
Global $main[1600][8],$index = '<>', $num
For $i = 1 to 1000
    $main[$i][0] = 'somedata'
    $main[$i][1] = 'somedata'
    _randomdata($i)
    $main[$i][3] ='somedata'
    $main[$i][4] ='somedata'
    $main[$i][5] ='somedata'
    $main[$i][6] ='somedata'
Next
_ArrayDisplay($main)
$num = StringSplit($num, '|')
MsgBox(0,'',$index)
For $i = 1 to $num[0]
    $st = TimerInit()
    $arr = StringRegExp($index,'.+?'&$num[$i]&'\<(\d+?)\>',3)
    If Not IsArray($arr) Then ContinueLoop
    $r = Ubound($arr)-1
    For $f = 0 to $r
        $main[$arr[$f]][7] = $main[$arr[$f]][2]
    Next
ConsoleWrite($r & ' - ' & TimerDiff($st) & @CRLF)
Next
_ArrayDisplay($main)
Func _Randomdata($n)
    $x = Random(1,2,1)
    If $x = 2 Then
         $r = Random(100000,100500,1)
         $main[$n][2] = $r
        $index &= $r & '<'&$n&'>'
        If Not StringInStr($num,$r) Then $num &= $r & '|'
    EndIf
EndFunc

 

Share this post


Link to post
Share on other sites
Melba23

step887,

As clear as mud - and the code is no help. Try explaining again and adding some comments to the script so we can get an idea of what is supposed to be happening. ;)

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites
kylomas

Should this line $num = StringSplit($num, '|') be  $num = StringSplit($main, '|')  ??

edit: Ignore this, I see it, set in _Randomdata

Edited by kylomas

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites
Jos

easier example:

$num = "<>100231<4>100159<7>100480<8>100407<15>100423<16>100372<17>100090<18>100202<21>100330<23>100091<24>100352<25>100430<26>100238<28>100221"
For $x = 1 to 6
        $num &= $num
Next
$st = TimerInit()
$arr = StringRegExp($num,'.+?100423\<(\d+?)\>',3)
ConsoleWrite(Ubound($arr) & ' - ' & TimerDiff($st) & @CRLF)

$st = TimerInit()
$arr = StringRegExp($num,'.+?999999\<(\d+?)\>',3)
ConsoleWrite(Ubound($arr) & ' - ' & TimerDiff($st) & @CRLF)

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
step887

step887,

As clear as mud - and the code is no help. Try explaining again and adding some comments to the script so we can get an idea of what is supposed to be happening. ;)

M23

Sorry I try to clear, basically I have array of files and column 2 is MD5 and if that is populated I use to look up data on that file and put in column 7 (8)

but sometimes there is same md5 in the array of files, so I will populate the same data to all lines with the same md5.  I have commented on the code below.

 

Should this line $num = StringSplit($num, '|') be  $num = StringSplit($main, '|')  ??

No $num is a lists of numbers I used in column 2 and need go through the array and add data to column 7

It is a bit of a complex way of showing that  StringRegExp() takes longer when nothing is found than when it returns something.

Jos

Actually StringregExp flag 3 is returning at least one result, and since you mentioned this I updated the code and it is Stringregexp that is taken the time.  Is there a workaround for this?

first number is num of returns - 1 because I am using ubound, SRE is time difference after the reg exp and total is after the For  

1 SRE: 202.508481847547 total: 202.540124318271
0 SRE: 222.228402681416 total: 222.256714365748
0 SRE: 221.022324928871 total: 221.050303534564
1 SRE: 22.6979769469612 total: 22.7189609012309
0 SRE: 218.70642915051 total: 218.735073913481
 
Updated code with explanation 
#include <array.au3>
Global $main[1600][8],$index = '<>', $num
For $i = 1 to 1000
    $main[$i][0] = 'somedata'
    $main[$i][1] = 'somedata'
    _randomdata($i)
    $main[$i][3] ='somedata'
    $main[$i][4] ='somedata'
    $main[$i][5] ='somedata'
    $main[$i][6] ='somedata'
Next
;populating data in my array
;if something is column 2, I need to follow up and add more data into column 7
_ArrayDisplay($main)
$num = StringSplit($num, '|');this is a index of the numbers used on column 2
MsgBox(0,'',$index); here is the index
For $i = 1 to $num[0]; so this goes through all the numbers used in column 2
    $st = TimerInit()
    $arr = StringRegExp($index,'.+?'&$num[$i]&'\<(\d+?)\>',3); checks the index and returns everytime a number is used in main

    If Not IsArray($arr) Then ContinueLoop; no matches
    $r = Ubound($arr)-1; number of returns -1
        ConsoleWrite($r & ' SRE: '  & TimerDiff($st))
    For $f = 0 to $r
        $main[$arr[$f]][7] = $main[$arr[$f]][2];writes data the to all numbers used.
    Next
ConsoleWrite(' total: ' & TimerDiff($st) & @CRLF); so this writes how many times
Next
_ArrayDisplay($main)
Func _Randomdata($n)
    $x = Random(1,2,1);use this to writ 50% of the clls
    If $x = 2 Then
         $r = Random(100000,100500,1); assign one of 500 numbers
         $main[$n][2] = $r
        $index &= $r & '<'&$n&'>'
        If Not StringInStr($num,$r) Then $num &= $r & '|'; keep a list of numbers used
    EndIf
EndFunc
2 SRE: 16.0430657357341 total: 16.0647158472821

Share this post


Link to post
Share on other sites
step887

I am not a RegEx person, but does this give the same result but faster?

$arr = StringRegExp($index,'.*?'&$num[$i]&'\<(\d*?)\>',3)

Jos

that works

0 SRE: 0.315758549962129 total: 0.380042727327837
3 SRE: 0.272791405505257 total: 0.307098505342914
1 SRE: 0.263798282246842 total: 0.287446865630082
2 SRE: 0.253472844431625 total: 0.279119899650068
3 SRE: 0.273124484144458 total: 0.303767718950909

 

Thanks a lot Jos, so I guess the question is what is the difference between + and * in regex?

Share this post


Link to post
Share on other sites
step887

Did you see the post above your last one?

 

You posted while I typing up my previous response 

Share this post


Link to post
Share on other sites
JLogan3o13

Regex generally makes my eyes bleed, but I thought + requires at least one character to match, where * does not.


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites
Jos

Thanks a lot Jos, so I guess the question is what is the difference between + and * in regex?

The difference is explained in the Helpfile: +? = 1 or more Lazy; *? = 0 or more Lazy, but I have no idea why the difference is so big.

I am sure somebody will stop by that understands :)


SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
step887

I do not understand it either, but I went from 480 secs to less then one sec making that one change

Thanks for the help. 

Share this post


Link to post
Share on other sites
jchd

Your sample code is overly complex to show your point.

Notice that neither .+? nor .*? are necessary: these sub-pattern just waste cycles for no benefit.Worse, .+? may cause duplicates to be missed.

Now to the point: PCRE (the regexp engine used by AutoIt) performs complex initial optimizations by default. It is possible to obtain the bytecode generated by the internal pattern compiler but it isn't worse the pain in this case.

As a general rule of thumb, avoid .+ and .* as much as possible for efficiency.

Try this version:

$arr = StringRegExp($index, $num[$i] & '\<(\d+)\>', 3); checks the index and returns everytime a number is used in main

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×