langthang084

how to make this script run faster??

8 posts in this topic

Here's my code:

Func _Classified($SourceArray)
   ProgressOn('Classified', '')
   Local $ExportArray[][] = [['', '']]
   $EndLoop = UBound($SourceArray) - 1
   for $a = 1 to UBound($SourceArray) - 1
      ProgressSet( int(($a/$EndLoop)*100), int(($a/$EndLoop)*100) & '%' )
      $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, 0, 0, 1)
      if @error then
         _ArrayAdd($ExportArray, '')
         $ExportArray[UBound($ExportArray) - 1][0] = $SourceArray[$a]
         $ExportArray[UBound($ExportArray) - 1][1] = 1
      Else
         $ExportArray[$Line][1] += 1
      EndIf
   Next
   ProgressOff()
   _ArrayDelete($ExportArray, 0)
   ;_ArrayDisplay($ExportArray, 'After Distribute')
   Return $ExportArray
EndFunc

But if $SourceArray has a huge elements, It take too long. So how to make it run faster???

:sweating:

Share this post


Link to post
Share on other sites



Huge ???  Too long???? 

Would help when you have a reproducer script and at least more facts like what it exactly needs to do! ;)

Jos


Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
#include <array.au3>

local $SourceText[] = ['']
$TotalWord = 1000
for $a = 1 to $TotalWord
   $Element = 'ABC' & Random(1, 9, 1)
   _ArrayAdd($SourceText, $Element)
Next
_ArrayDisplay($SourceText, 'Source')
 $Export = _Classified($SourceText)
_ArrayDisplay($Export, 'Export')

Func _Classified($SourceArray)
   ProgressOn('Classified', '')
   Local $ExportArray[][] = [['', '']]
   $EndLoop = UBound($SourceArray) - 1
   for $a = 1 to UBound($SourceArray) - 1
      ProgressSet( int(($a/$EndLoop)*100), int(($a/$EndLoop)*100) & '%' )
      $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, 0, 0, 1)
      if @error then
         _ArrayAdd($ExportArray, '')
         $ExportArray[UBound($ExportArray) - 1][0] = $SourceArray[$a]
         $ExportArray[UBound($ExportArray) - 1][1] = 1
      Else
         $ExportArray[$Line][1] += 1
      EndIf
   Next
   ProgressOff()
   _ArrayDelete($ExportArray, 0)
   ;_ArrayDisplay($ExportArray, 'After Distribute')
   Return $ExportArray
EndFunc

I want to split a paragraph into word and classify by the total times  it appears. If it has 1000 word (like above), it takes a little time. But if a paragraph contain to many word (ex: $TotalWord = 100000000000000000000), it takes too long. 

So I want to make it runs faster

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Why not sort the array and then iterate through it counting each unique word and put that into a second Array with totals found?
Should be (a lot) faster!?

Jos 

Edited by Jos

Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

Did you mean _ArrayUnique and _ArrayFindAll

Func _Classified($SourceArray)
   ProgressOn('Classified', '')
   $ExportArray = _ArrayUnique($SourceArray)
   _ArrayDelete($ExportArray, 0)
   _ArrayColInsert($ExportArray, 1)
  ; _ArrayDisplay($ExportArray, 'Export')
   $EndLoop = UBound($ExportArray) - 1
   for $a = 0 to $EndLoop
      ProgressSet( int(($a/$EndLoop)*100), int(($a/$EndLoop)*100) & '%' )
      $Line = _ArrayFindAll($SourceArray, $ExportArray[$a][0], 0, 0, 0, 1)
      $ExportArray[$a][1] = UBound($Line)
   Next
   ProgressOff()
   _ArrayDelete($ExportArray, 0)
   ;_ArrayDisplay($ExportArray, 'After Distribute')
   Return $ExportArray
EndFunc

 

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

This is a test version where I've optimised your version and added a version that uses the principal I was trying to explain. :)

Just try and see what results it gives you. these were my results:

Time:56422.5298089697  (Your original func)
Time:24182.4416734366  (Your Func with changes)
Time:4662.7327334754  (Sort and count)

#include <array.au3>

$TotalWord = 100000
Local $SourceText[$TotalWord+1] = ['']
For $a = 1 To $TotalWord
    $Element = 'ABC' & Random(1, 9, 1)
    $SourceText[$a]=$Element
Next
;~ _ArrayDisplay($SourceText, 'Source')
$t=TimerInit()
$Export = _oClassified($SourceText)
Consolewrite("Time:" & TimerDiff($T) &@CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')
;
$t=TimerInit()
$Export = _Classified2($SourceText)
Consolewrite("Time:" & TimerDiff($T) &@CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')
;
$t=TimerInit()
$Export = _Classified3($SourceText)
Consolewrite("Time:" & TimerDiff($T) &@CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')

Func _oClassified($SourceArray)
    ProgressOn('Classified', '')
    Local $ExportArray[][] = [['', '']]
    $EndLoop = UBound($SourceArray) - 1
    For $a = 1 To UBound($SourceArray) - 1
        ProgressSet(Int(($a / $EndLoop) * 100), Int(($a / $EndLoop) * 100) & '%')
        $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, 0, 0, 1)
        If @error Then
            _ArrayAdd($ExportArray, '')
            $ExportArray[UBound($ExportArray) - 1][0] = $SourceArray[$a]
            $ExportArray[UBound($ExportArray) - 1][1] = 1
        Else
            $ExportArray[$Line][1] += 1
        EndIf
    Next
    ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc    ;==>_oClassified

Func _Classified2($SourceArray)
    ProgressOn('Classified 2', '')
    Local $ExportArray[UBound($SourceArray)][2]
    $WordCount=0
    $EndLoop = UBound($SourceArray) - 1
    For $a = 1 To UBound($SourceArray) - 1
        If mod($a,int($EndLoop/100)) = 0 Then
            ProgressSet(Int(($a / $EndLoop) * 100), Int(($a / $EndLoop) * 100) & '%')
        EndIf
        $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, $WordCount, 0, 1)
        If @error Then
            $WordCount += 1
            $ExportArray[$WordCount][0] = $SourceArray[$a]
            $ExportArray[$WordCount][1] = 1

        Else
            $ExportArray[$Line][1] += 1
        EndIf
    Next
    ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ReDim $ExportArray[$WordCount][2]
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc    ;==>_oClassified

Func _Classified3($SourceArray)
    ProgressOn('Classified2', 'Start Sort')
    _ArraySort($SourceArray,0,0,0,0,1)
    Local $ExportArray[UBound($SourceArray)][2]
    Local $LastWord
    Local $WordOccCount = 0
    Local $WordCount = 0
    Local $EndLoop = UBound($SourceArray) - 1
    ProgressSet(0, '0%', 'Start counting')
    For $a = 1 To UBound($SourceArray) - 1
        If mod($a,int($EndLoop/100)) = 0 Then
            ProgressSet(Int(($a / $EndLoop) * 100), Int(($a / $EndLoop) * 100) & '%')
        EndIf
        If $a = 1 Then
            $LastWord = $SourceArray[$a]
            $WordCount = 1
            $WordOccCount = 1
            ContinueLoop
        EndIf
        If $LastWord = $SourceArray[$a] Then
            $WordOccCount += 1
        Else
            $ExportArray[$WordCount][0] = $LastWord
            $ExportArray[$WordCount][1] = $WordOccCount
            $LastWord = $SourceArray[$a]
            $WordCount += 1
            $WordOccCount = 1
        EndIf
    Next
    $ExportArray[$WordCount][0] = $LastWord
    $ExportArray[$WordCount][1] = $WordOccCount
    ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ReDim $ExportArray[$WordCount][2]
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc    ;==>_Classified

Jos

Edited by Jos
Modified _Classified2
1 person likes this

Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

Using maps seems to be very fast:

#AutoIt3Wrapper_Version=b

#include <array.au3>

$TotalWord = 100000
Local $SourceText[$TotalWord] = ['']
$fTimer = TimerInit()
For $a = 0 To $TotalWord - 1
    $SourceText[$a] = 'ABC' & Random(1, 9, 1)
Next
ConsoleWrite("Array created in " & TimerDiff($fTimer) & " ms" & @CRLF)

$fTimer = TimerInit()
$Export = _Classified($SourceText)
ConsoleWrite(TimerDiff($fTimer) & @CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')

$fTimer = TimerInit()
$Export = _Classified3($SourceText)
ConsoleWrite(TimerDiff($fTimer) & @CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')

$fTimer = TimerInit()
$Export2 = _Classified2_UEZ($SourceText)
ConsoleWrite(TimerDiff($fTimer) & @CRLF)
Local $aResult[UBound($Export2)][2]
$y = 0
For $key In $Export2.keys()
    $aResult[$y][0] = $key
    $aResult[$y][1] = $Export2[$key]
    $y += 1
Next
_ArraySort($aResult)
_ArrayDisplay($aResult)

Func _Classified2_UEZ($SourceArray)
    Local $aMaps[], $i
    For $i = 0 To UBound($SourceArray) - 1
        If $aMaps.Exists($SourceArray[$i]) Then
            $aMaps[$SourceArray[$i]] += 1
        Else
            $aMaps[$SourceArray[$i]] = 1
        EndIf
    Next
    Return $aMaps
EndFunc   ;==>_Classified2_UEZ

Func _Classified3($SourceArray)
;~     ProgressOn('Classified2', 'Start Sort')
    _ArraySort($SourceArray, 0, 0, 0, 0, 1)
    Local $ExportArray[UBound($SourceArray)][2]
    Local $LastWord
    Local $WordOccCount = 0
    Local $WordCount = 0
    Local $EndLoop = UBound($SourceArray) - 1
;~     ProgressSet(0, '0%', 'Start counting')
    For $a = 1 To UBound($SourceArray) - 1
;~         If mod($a,int($EndLoop/100)) = 0 Then
;~             ProgressSet(Int(($a / $EndLoop) * 100), Int(($a / $EndLoop) * 100) & '%')
;~         EndIf
        If $a = 1 Then
            $LastWord = $SourceArray[$a]
            $WordCount = 1
            $WordOccCount = 1
            ContinueLoop
        EndIf
        If $LastWord = $SourceArray[$a] Then
            $WordOccCount += 1
        Else
            $ExportArray[$WordCount][0] = $LastWord
            $ExportArray[$WordCount][1] = $WordOccCount
            $LastWord = $SourceArray[$a]
            $WordCount += 1
            $WordOccCount = 1
        EndIf
    Next
    $ExportArray[$WordCount][0] = $LastWord
    $ExportArray[$WordCount][1] = $WordOccCount
;~     ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ReDim $ExportArray[$WordCount][2]
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc   ;==>_Classified3

Func _Classified($SourceArray)
;~    ProgressOn('Classified', '')
    Local $ExportArray[][] = [['', '']]
    $EndLoop = UBound($SourceArray) - 1
    For $a = 1 To UBound($SourceArray) - 1
;~       ProgressSet( int(($a/$EndLoop)*100), int(($a/$EndLoop)*100) & '%' )
        $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, 0, 0, 1)
        If @error Then
            _ArrayAdd($ExportArray, '')
            $ExportArray[UBound($ExportArray) - 1][0] = $SourceArray[$a]
            $ExportArray[UBound($ExportArray) - 1][1] = 1
        Else
            $ExportArray[$Line][1] += 1
        EndIf
    Next
;~    ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc   ;==>_Classified

To test this function you need the latest beta!

Edited by UEZ
1 person likes this

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

Thanks Jos and UEZ very much.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now