how to make this script run faster??

[langthang084]

Here's my code:

Func _Classified($SourceArray)
   ProgressOn('Classified', '')
   Local $ExportArray[][] = [['', '']]
   $EndLoop = UBound($SourceArray) - 1
   for $a = 1 to UBound($SourceArray) - 1
      ProgressSet( int(($a/$EndLoop)*100), int(($a/$EndLoop)*100) & '%' )
      $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, 0, 0, 1)
      if @error then
         _ArrayAdd($ExportArray, '')
         $ExportArray[UBound($ExportArray) - 1][0] = $SourceArray[$a]
         $ExportArray[UBound($ExportArray) - 1][1] = 1
      Else
         $ExportArray[$Line][1] += 1
      EndIf
   Next
   ProgressOff()
   _ArrayDelete($ExportArray, 0)
   ;_ArrayDisplay($ExportArray, 'After Distribute')
   Return $ExportArray
EndFunc

But if $SourceArray has a huge elements, It take too long. So how to make it run faster???

[Jos]

Huge ???  Too long???? 

Would help when you have a reproducer script and at least more facts like what it exactly needs to do!

Jos

[langthang084]

#include <array.au3>

local $SourceText[] = ['']
$TotalWord = 1000
for $a = 1 to $TotalWord
   $Element = 'ABC' & Random(1, 9, 1)
   _ArrayAdd($SourceText, $Element)
Next
_ArrayDisplay($SourceText, 'Source')
 $Export = _Classified($SourceText)
_ArrayDisplay($Export, 'Export')

Func _Classified($SourceArray)
   ProgressOn('Classified', '')
   Local $ExportArray[][] = [['', '']]
   $EndLoop = UBound($SourceArray) - 1
   for $a = 1 to UBound($SourceArray) - 1
      ProgressSet( int(($a/$EndLoop)*100), int(($a/$EndLoop)*100) & '%' )
      $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, 0, 0, 1)
      if @error then
         _ArrayAdd($ExportArray, '')
         $ExportArray[UBound($ExportArray) - 1][0] = $SourceArray[$a]
         $ExportArray[UBound($ExportArray) - 1][1] = 1
      Else
         $ExportArray[$Line][1] += 1
      EndIf
   Next
   ProgressOff()
   _ArrayDelete($ExportArray, 0)
   ;_ArrayDisplay($ExportArray, 'After Distribute')
   Return $ExportArray
EndFunc

I want to split a paragraph into word and classify by the total times  it appears. If it has 1000 word (like above), it takes a little time. But if a paragraph contain to many word (ex: $TotalWord = 100000000000000000000), it takes too long. 

So I want to make it runs faster

[Jos]

Why not sort the array and then iterate through it counting each unique word and put that into a second Array with totals found?
Should be (a lot) faster!?

Jos 

Edited February 29, 2016 by Jos

[langthang084]

Did you mean _ArrayUnique and _ArrayFindAll

Func _Classified($SourceArray)
   ProgressOn('Classified', '')
   $ExportArray = _ArrayUnique($SourceArray)
   _ArrayDelete($ExportArray, 0)
   _ArrayColInsert($ExportArray, 1)
  ; _ArrayDisplay($ExportArray, 'Export')
   $EndLoop = UBound($ExportArray) - 1
   for $a = 0 to $EndLoop
      ProgressSet( int(($a/$EndLoop)*100), int(($a/$EndLoop)*100) & '%' )
      $Line = _ArrayFindAll($SourceArray, $ExportArray[$a][0], 0, 0, 0, 1)
      $ExportArray[$a][1] = UBound($Line)
   Next
   ProgressOff()
   _ArrayDelete($ExportArray, 0)
   ;_ArrayDisplay($ExportArray, 'After Distribute')
   Return $ExportArray
EndFunc

 

[Jos]

This is a test version where I've optimised your version and added a version that uses the principal I was trying to explain.

Just try and see what results it gives you. these were my results:

Time:56422.5298089697  (Your original func)
Time:24182.4416734366  (Your Func with changes)
Time:4662.7327334754  (Sort and count)

expandcollapse popup

#include <array.au3>

$TotalWord = 100000
Local $SourceText[$TotalWord+1] = ['']
For $a = 1 To $TotalWord
    $Element = 'ABC' & Random(1, 9, 1)
    $SourceText[$a]=$Element
Next
;~ _ArrayDisplay($SourceText, 'Source')
$t=TimerInit()
$Export = _oClassified($SourceText)
Consolewrite("Time:" & TimerDiff($T) &@CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')
;
$t=TimerInit()
$Export = _Classified2($SourceText)
Consolewrite("Time:" & TimerDiff($T) &@CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')
;
$t=TimerInit()
$Export = _Classified3($SourceText)
Consolewrite("Time:" & TimerDiff($T) &@CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')

Func _oClassified($SourceArray)
    ProgressOn('Classified', '')
    Local $ExportArray[][] = [['', '']]
    $EndLoop = UBound($SourceArray) - 1
    For $a = 1 To UBound($SourceArray) - 1
        ProgressSet(Int(($a / $EndLoop) * 100), Int(($a / $EndLoop) * 100) & '%')
        $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, 0, 0, 1)
        If @error Then
            _ArrayAdd($ExportArray, '')
            $ExportArray[UBound($ExportArray) - 1][0] = $SourceArray[$a]
            $ExportArray[UBound($ExportArray) - 1][1] = 1
        Else
            $ExportArray[$Line][1] += 1
        EndIf
    Next
    ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc    ;==>_oClassified

Func _Classified2($SourceArray)
    ProgressOn('Classified 2', '')
    Local $ExportArray[UBound($SourceArray)][2]
    $WordCount=0
    $EndLoop = UBound($SourceArray) - 1
    For $a = 1 To UBound($SourceArray) - 1
        If mod($a,int($EndLoop/100)) = 0 Then
            ProgressSet(Int(($a / $EndLoop) * 100), Int(($a / $EndLoop) * 100) & '%')
        EndIf
        $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, $WordCount, 0, 1)
        If @error Then
            $WordCount += 1
            $ExportArray[$WordCount][0] = $SourceArray[$a]
            $ExportArray[$WordCount][1] = 1

        Else
            $ExportArray[$Line][1] += 1
        EndIf
    Next
    ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ReDim $ExportArray[$WordCount][2]
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc    ;==>_oClassified

Func _Classified3($SourceArray)
    ProgressOn('Classified2', 'Start Sort')
    _ArraySort($SourceArray,0,0,0,0,1)
    Local $ExportArray[UBound($SourceArray)][2]
    Local $LastWord
    Local $WordOccCount = 0
    Local $WordCount = 0
    Local $EndLoop = UBound($SourceArray) - 1
    ProgressSet(0, '0%', 'Start counting')
    For $a = 1 To UBound($SourceArray) - 1
        If mod($a,int($EndLoop/100)) = 0 Then
            ProgressSet(Int(($a / $EndLoop) * 100), Int(($a / $EndLoop) * 100) & '%')
        EndIf
        If $a = 1 Then
            $LastWord = $SourceArray[$a]
            $WordCount = 1
            $WordOccCount = 1
            ContinueLoop
        EndIf
        If $LastWord = $SourceArray[$a] Then
            $WordOccCount += 1
        Else
            $ExportArray[$WordCount][0] = $LastWord
            $ExportArray[$WordCount][1] = $WordOccCount
            $LastWord = $SourceArray[$a]
            $WordCount += 1
            $WordOccCount = 1
        EndIf
    Next
    $ExportArray[$WordCount][0] = $LastWord
    $ExportArray[$WordCount][1] = $WordOccCount
    ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ReDim $ExportArray[$WordCount][2]
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc    ;==>_Classified

Jos

Edited February 29, 2016 by Jos
Modified _Classified2

[UEZ]

Using maps seems to be very fast:

expandcollapse popup

#AutoIt3Wrapper_Version=b

#include <array.au3>

$TotalWord = 100000
Local $SourceText[$TotalWord] = ['']
$fTimer = TimerInit()
For $a = 0 To $TotalWord - 1
    $SourceText[$a] = 'ABC' & Random(1, 9, 1)
Next
ConsoleWrite("Array created in " & TimerDiff($fTimer) & " ms" & @CRLF)

$fTimer = TimerInit()
$Export = _Classified($SourceText)
ConsoleWrite(TimerDiff($fTimer) & @CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')

$fTimer = TimerInit()
$Export = _Classified3($SourceText)
ConsoleWrite(TimerDiff($fTimer) & @CRLF)
_ArraySort($Export)
_ArrayDisplay($Export, 'Export')

$fTimer = TimerInit()
$Export2 = _Classified2_UEZ($SourceText)
ConsoleWrite(TimerDiff($fTimer) & @CRLF)
Local $aResult[UBound($Export2)][2]
$y = 0
For $key In $Export2.keys()
    $aResult[$y][0] = $key
    $aResult[$y][1] = $Export2[$key]
    $y += 1
Next
_ArraySort($aResult)
_ArrayDisplay($aResult)

Func _Classified2_UEZ($SourceArray)
    Local $aMaps[], $i
    For $i = 0 To UBound($SourceArray) - 1
        If $aMaps.Exists($SourceArray[$i]) Then
            $aMaps[$SourceArray[$i]] += 1
        Else
            $aMaps[$SourceArray[$i]] = 1
        EndIf
    Next
    Return $aMaps
EndFunc   ;==>_Classified2_UEZ

Func _Classified3($SourceArray)
;~     ProgressOn('Classified2', 'Start Sort')
    _ArraySort($SourceArray, 0, 0, 0, 0, 1)
    Local $ExportArray[UBound($SourceArray)][2]
    Local $LastWord
    Local $WordOccCount = 0
    Local $WordCount = 0
    Local $EndLoop = UBound($SourceArray) - 1
;~     ProgressSet(0, '0%', 'Start counting')
    For $a = 1 To UBound($SourceArray) - 1
;~         If mod($a,int($EndLoop/100)) = 0 Then
;~             ProgressSet(Int(($a / $EndLoop) * 100), Int(($a / $EndLoop) * 100) & '%')
;~         EndIf
        If $a = 1 Then
            $LastWord = $SourceArray[$a]
            $WordCount = 1
            $WordOccCount = 1
            ContinueLoop
        EndIf
        If $LastWord = $SourceArray[$a] Then
            $WordOccCount += 1
        Else
            $ExportArray[$WordCount][0] = $LastWord
            $ExportArray[$WordCount][1] = $WordOccCount
            $LastWord = $SourceArray[$a]
            $WordCount += 1
            $WordOccCount = 1
        EndIf
    Next
    $ExportArray[$WordCount][0] = $LastWord
    $ExportArray[$WordCount][1] = $WordOccCount
;~     ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ReDim $ExportArray[$WordCount][2]
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc   ;==>_Classified3

Func _Classified($SourceArray)
;~    ProgressOn('Classified', '')
    Local $ExportArray[][] = [['', '']]
    $EndLoop = UBound($SourceArray) - 1
    For $a = 1 To UBound($SourceArray) - 1
;~       ProgressSet( int(($a/$EndLoop)*100), int(($a/$EndLoop)*100) & '%' )
        $Line = _ArraySearch($ExportArray, $SourceArray[$a], 0, 0, 0, 1)
        If @error Then
            _ArrayAdd($ExportArray, '')
            $ExportArray[UBound($ExportArray) - 1][0] = $SourceArray[$a]
            $ExportArray[UBound($ExportArray) - 1][1] = 1
        Else
            $ExportArray[$Line][1] += 1
        EndIf
    Next
;~    ProgressOff()
    _ArrayDelete($ExportArray, 0)
    ;_ArrayDisplay($ExportArray, 'After Distribute')
    Return $ExportArray
EndFunc   ;==>_Classified

To test this function you need the latest beta!

Edited February 29, 2016 by UEZ

[langthang084]

Thanks Jos and UEZ very much.