Jump to content

[SOLVED] How to add a progress bar to _IETableWriteToArray


Recommended Posts

Hello,

 

I am currently writing a program that parses a massive table from a website, and need a way to add a progress bar while parsing.

I am currently using the function _IETableWriteToArray($oObj, True) to parse the array. I need the progress bar to update as the table is parsed, not just at the end of the parsing.

Any help at all would be very much appreciated!

 

*EDIT --> The array I am left with after parsing is $array[0-50000][16]

Edited by rm4453
[SOLVED]
Link to post
Share on other sites

Hi @rm4453

a bad idea might be to change the IE library.... ( I suggest not to do it :thumbsdown:) , a better idea is to only clone the _IETableWriteToArray function and modify it by adding the progressbar and then use the modified function instead of the original one (in the modified function I marked with "; <-------- mod #" the lines I added):

#include <Array.au3>
#include <IE.au3>

Local $oIE = _IECreate("https://en.wikipedia.org/wiki/List_of_North_American_settlements_by_year_of_foundation")
Local $oTable = _IETableGetCollection($oIE, 1)
Local $aTableData = _IETableWriteToArray_mod($oTable, True)
_ArrayDisplay($aTableData)

_IEQuit($oIE)

; #FUNCTION# ====================================================================================================================
; Author ........: Dale Hohm
; modified ......: Chimp
; ===============================================================================================================================
Func _IETableWriteToArray_mod(ByRef $oObject, $bTranspose = False)
    If Not IsObj($oObject) Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidDataType")
        Return SetError($_IESTATUS_InvalidDataType, 1, 0)
    EndIf
    ;
    If Not __IEIsObjType($oObject, "table") Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidObjectType")
        Return SetError($_IESTATUS_InvalidObjectType, 1, 0)
    EndIf
    ;
    Local $iCols = 0, $oTds, $iCol
    Local $oTrs = $oObject.rows
    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $iCol = $iCol + $oTd.colSpan
        Next
        If $iCol > $iCols Then $iCols = $iCol
    Next
    Local $iRows = $oTrs.length
    Local $aTableCells[$iCols][$iRows]
    Local $iRow = 0
    Local $iPercent ; <-------- mod 1

    ProgressOn("Progress Meter", "processing rows ", 0 & "/" & $iRows, -1, -1, 16) ; <-------- mod 2

    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $aTableCells[$iCol][$iRow] = String($oTd.innerText)
            If @error Then ; Trap COM error, report and return
                __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_COMError", @error)
                Return SetError($_IESTATUS_ComError, @error, 0)
            EndIf
            $iCol = $iCol + $oTd.colSpan
        Next
        $iRow = $iRow + 1
        $iPercent = Int($iRow / $iRows * 100) ; <-------- mod 3
        ProgressSet($iPercent, $iRow & "/" & $iRows & @TAB & "(" & $iPercent & "%)") ; <-------- mod 4
    Next
    If $bTranspose Then
        Local $iD1 = UBound($aTableCells, $UBOUND_ROWS), $iD2 = UBound($aTableCells, $UBOUND_COLUMNS), $aTmp[$iD2][$iD1]
        For $i = 0 To $iD2 - 1
            For $j = 0 To $iD1 - 1
                $aTmp[$i][$j] = $aTableCells[$j][$i]
            Next
        Next
        $aTableCells = $aTmp
    EndIf
    ProgressOff() ; <-------- mod 5
    Return SetError($_IESTATUS_Success, 0, $aTableCells)
EndFunc   ;==>_IETableWriteToArray_mod

 

Edited by Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Link to post
Share on other sites
3 minutes ago, Chimp said:

Hi @rm4453

a bad idea might be to change the IE library.... ( I suggest not to do it :thumbsdown:) , a better idea is to only clone the _IETableWriteToArray function and modify it by adding the progressbar and then use the modified function function instead of the original one (in the modified function I marked with "; <-------- mod #" the lines I added):

 

Hello, @Chimp

 

I added your modified function to the IE library, and renamed it to _IETableWriteToArray_ProgressBar. I am testing it right now to see how well it works out, and appreciate all the help so far!

Link to post
Share on other sites

@Chimp

I have added one more modification, and it is working perfectly thank you!

*The modification allows you to append a message to be on the progress bar at the end of the function inputs.*

; #FUNCTION# ====================================================================================================================
; Author ........: Dale Hohm
; modified ......: Chimp
; ===============================================================================================================================
Func _IETableWriteToArray_ProgressBar(ByRef $oObject, $bTranspose = False, $message = "Processing Data!") ; <------- If copied into IE.au3 can call normally with _ProgressBar appended to have progress bar pop up, you can also include a message as the last arg if wanted!
    If Not IsObj($oObject) Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidDataType")
        Return SetError($_IESTATUS_InvalidDataType, 1, 0)
    EndIf
    ;
    If Not __IEIsObjType($oObject, "table") Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidObjectType")
        Return SetError($_IESTATUS_InvalidObjectType, 1, 0)
    EndIf
    ;
    Local $iCols = 0, $oTds, $iCol
    Local $oTrs = $oObject.rows
    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $iCol = $iCol + $oTd.colSpan
        Next
        If $iCol > $iCols Then $iCols = $iCol
    Next
    Local $iRows = $oTrs.length
    Local $aTableCells[$iCols][$iRows]
    Local $iRow = 0
    Local $iPercent ; <-------- mod 1

    ProgressOn("Progress Meter", $message, 0 & "/" & $iRows, -1, -1, 16) ; <-------- mod 2

    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $aTableCells[$iCol][$iRow] = String($oTd.innerText)
            If @error Then ; Trap COM error, report and return
                __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_COMError", @error)
                Return SetError($_IESTATUS_ComError, @error, 0)
            EndIf
            $iCol = $iCol + $oTd.colSpan
        Next
        $iRow = $iRow + 1
        $iPercent = Int($iRow / $iRows * 100) ; <-------- mod 3
        ProgressSet($iPercent, $iRow & "/" & $iRows & @TAB & "(" & $iPercent & "%)") ; <-------- mod 4
    Next
    If $bTranspose Then
        Local $iD1 = UBound($aTableCells, $UBOUND_ROWS), $iD2 = UBound($aTableCells, $UBOUND_COLUMNS), $aTmp[$iD2][$iD1]
        For $i = 0 To $iD2 - 1
            For $j = 0 To $iD1 - 1
                $aTmp[$i][$j] = $aTableCells[$j][$i]
            Next
        Next
        $aTableCells = $aTmp
    EndIf
    ProgressOff() ; <-------- mod 5
    Return SetError($_IESTATUS_Success, 0, $aTableCells)
EndFunc   ;==>_IETableWriteToArray_ProgressBar

 

Edited by rm4453
Fixed Spelling Mishap
Link to post
Share on other sites

You are welcome :)
p.s. better to also add an ProgressOff() statement right after the line "If @error Then ; Trap COM error, report and return" so that the progress bar will be closed also in case of errors.

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By vinnyMS
      #Include <Array.au3> #include <Constants.au3> $s = FileRead("2.txt") Local $w = StringRegExp($s, '(?is)(\b\w+\b)(?!.*\b\1\b)', 3) _ArrayColInsert($w, 1) For $i = 0 to UBound($w)-1 StringRegExpReplace($s, '(?i)\b' & $w[$i][0] & '\b', $w[$i][0]) $w[$i][1] = @extended Next _ArraySort($w, 1, 0, 0, 1) _ArrayDisplay($w) i have this script that returns 3 columns  
       
      i need to copy the  Col 0 and Col 1 as text to paste on notepad or excel
      you will have to create a "copy" button if possible
      array.au3 2.txt
    • By DannyJ
      I have a dataset like this, (a strubg)
      Username: User1 Type: Admin RegDate: 1999 Username: User2 Type: User RegDate: 2000 How to make a 2 dimensional array that I can display with _ArrayDisplay?
      This would be a perfect 2D array to represent my data:
      Username           Tpye RegDate User1              Admin 1999 User2              User 2000   If you run this Powershell this powershell command, you can get this dataset that I am talking about:
      Get-LocalUser | Select * With this code you can try it to read into a string:
      #include <GuiConstantsEx.au3> #include <WindowsConstants.au3> #include "GUIListViewEx.au3" #include <Array.au3> ; Just for display in example #RequireAdmin #Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseX64=y #EndRegion $sCommand = "powershell.exe Get-LocalUser | Select *" Local $iPid = Run($sCommand, @WorkingDir , @SW_SHOW , $STDOUT_CHILD) ProcessWaitClose($iPid) Local $sOutput = StdoutRead($iPID) ConsoleWrite($sOutput) How can I correctly split $sOutput into a 2D array (with the above mentioned layout) that I can display and I work with?
    • By kovlad
      My solution is to write nested arrays without copying.
      The problem was described hier.
       
      Function:
      #include <Array.au3> ; #FUNCTION# ==================================================================================================================== ; Name ..........: _ArrayNestedSet ; Description ...: Assigns a value to an element of a nested 1D array. ; Syntax ........: _ArrayNestedSet(ByRef $aArray, $vIndex, $vValue) ; Parameters ....: $aArray - an array of arrays. ; $vIndex - an index or 1d-array of indexes; ; a size if $vValue not defined (zero to delete). ; $vValue - a value (create, resize or delete if not defined). ; ; Return values .: on success - 1 ; @extended - nesting level of operation ; on failure - 0 ; @extended - nesting level of error ; @error = 1 - invalid array ; @error = 2 - invalid index ; Author ........: ; Modified ......: kovlad ; Remarks .......: ; Related .......: ; Link ..........: https://www.autoitscript.com/forum/topic/185638-assign-a-value-to-an-array-in-array-element/ ; https://www.autoitscript.com/trac/autoit/ticket/3515?replyto=description ; Example .......: Yes ; =============================================================================================================================== Func _ArrayNestedSet(ByRef $aArray, $vIndex, $vValue = Default) Local $extended = @extended + 1 If IsArray($vIndex) Then If UBound($vIndex, 0) <> 1 Then _ Return SetError(2, $extended) If UBound($vIndex) > 1 Then If UBound($aArray, 0) <> 1 Then _ Return SetError(1, $extended) ; keep index for this array Local $i = $vIndex[0] If $i < 0 Or UBound($aArray) <= $i Then _ Return SetError(2, $extended) ; delete index of this array _ArrayDelete($vIndex, 0) ; recursive function call Local $return = _ArrayNestedSet($aArray[$i], $vIndex, $vValue) If @error Then Return SetError(@error, @extended + 1, 0) Else Return SetExtended(@extended + 1, 1) EndIf Else $vIndex = $vIndex[0] EndIf EndIf If $vValue = Default Then If $vIndex < 0 Then _ Return SetError(2, $extended) If $vIndex = 0 Then ; delete array and free memory $aArray = 0 Return SetExtended($extended, 1) EndIf If UBound($aArray, 0) = 1 Then ; resize array keeping data ReDim $aArray[$vIndex] Return SetExtended($extended, 1) Else ; create new nested array Local $aTmp[$vIndex] $aArray = $aTmp Return SetExtended($extended, 1) EndIf Else If UBound($aArray) <= $vIndex Then _ Return SetError(2, $extended + 1) ; set value of array entry $aArray[$vIndex] = $vValue Return SetExtended($extended, 1) EndIf EndFunc  
      Examples:
      ; write value to 1st nested array ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : write value to 1st nested array" & @CRLF) Local $aTmp1[4] = [1,2,3,4] _ArrayDisplay($aTmp1, "$aTmp1") Local $aArray[2] = [$aTmp1] ConsoleWrite( _ "_ArrayNestedSet($aArray[0], 3, 14) = " & _ArrayNestedSet($aArray[0], 3, 14) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF & @CRLF) _ArrayDisplay($aArray[0], "$aArray[0]") ; resize 1st nested array ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : resize 1st nested array" & @CRLF) ConsoleWrite( _ "_ArrayNestedSet($aArray[0], 8) = " & _ArrayNestedSet($aArray[0], 8) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF & @CRLF) _ArrayDisplay($aArray[0], "$aArray[0]") ; write array to 1st nested array ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : write array to 1st nested array" & @CRLF) Local $aTmp11[4] = [11,12,13,14] _ArrayDisplay($aTmp11, "$aTmp11") ConsoleWrite( _ "_ArrayNestedSet($aArray[0], 2, $aTmp11) = " & _ArrayNestedSet($aArray[0], 2, $aTmp11) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF & @CRLF) _ArrayDisplay(($aArray[0])[2], "($aArray[0])[2]") ; write value to 2nd nested array using index array ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : write value to 2nd nested array using index array" & @CRLF) Local $aIndex1[2] = [2,3] _ArrayDisplay($aIndex1, "$aIndex1") ConsoleWrite( _ "_ArrayNestedSet($aArray[0], $aIndex1, 140) = " & _ArrayNestedSet($aArray[0], $aIndex1, 140) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF & @CRLF) _ArrayDisplay(($aArray[0])[2], "($aArray[0])[2]") ; resize 2nd nested array ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : resize 2nd nested array" & @CRLF) Local $aIndex1[2] = [2,8] _ArrayDisplay($aIndex1, "$aIndex1") ConsoleWrite( _ "_ArrayNestedSet($aArray[0], $aIndex1) = " & _ArrayNestedSet($aArray[0], $aIndex1) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF & @CRLF) _ArrayDisplay(($aArray[0])[2], "($aArray[0])[2]") ; create new 3rd nested array ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : create new 3rd nested array" & @CRLF) Local $aIndex2[3] = [2,7,6] _ArrayDisplay($aIndex2, "$aIndex2") ConsoleWrite( _ "_ArrayNestedSet($aArray[0], $aIndex2) = " & _ArrayNestedSet($aArray[0], $aIndex2) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF & @CRLF) _ArrayDisplay((($aArray[0])[2])[7], ")($aArray[0])[2])[7]") ; delete 3rd nested array ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : delete 3rd nested array" & @CRLF) Local $aIndex3[3] = [2,7,0] _ArrayDisplay($aIndex3, "$aIndex2") ConsoleWrite( _ "_ArrayNestedSet($aArray[0], $aIndex3) = " & _ArrayNestedSet($aArray[0], $aIndex3) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF) ConsoleWrite("IsArray((($aArray[0])[2])[7]) = " & IsArray((($aArray[0])[2])[7]) & @CRLF & @CRLF) ; write 0 in 1st nested array to delete the 2nd nested array ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : write 0 in 1st nested array to delete the 2nd nested array" & @CRLF) Local $aIndex4[1] = [2] _ArrayDisplay($aIndex4, "$aIndex4") ConsoleWrite( _ "_ArrayNestedSet($aArray[0], $aIndex4, 0) = " & _ArrayNestedSet($aArray[0], $aIndex4, 0) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF) ConsoleWrite("IsArray(($aArray[0])[2]) = " & IsArray(($aArray[0])[2]) & @CRLF & @CRLF) ; delete 1st nested array (same as '$aArray[0] = 0') ConsoleWrite("@@ Debug(" & @ScriptLineNumber & ") : delete 1st nested array (same as '$aArray[0] = 0')" & @CRLF) Local $aIndex5[1] = [0] _ArrayDisplay($aIndex5, "$aIndex5") ConsoleWrite( _ "_ArrayNestedSet($aArray[0], $aIndex5) = " & _ArrayNestedSet($aArray[0], $aIndex5) & @CRLF & _ " @error = " & @error & @CRLF & _ " @extended = " & @extended & @CRLF) ConsoleWrite("IsArray($aArray[0]) = " & IsArray($aArray[0]) & @CRLF & @CRLF)  
    • By DannyJ
      $sCommands1 = 'powershell.exe Get-ChildItem' $iPid = run($sCommands1   , @WorkingDir , @SW_SHOW , 0x2) $sOutput = ""  While 1     $sOutput &= StdoutRead($iPID)         If @error Then             ExitLoop         EndIf  WEnd ;~ msgbox(0, '' , $sOutput) ConsoleWrite("$sOutput") ConsoleWrite($sOutput) ConsoleWrite(@CRLF) $aOutput = stringsplit($sOutput ,@LF , 2) For $i=0 To  UBound($aOutput) - 1 Step 1     ConsoleWrite($aOutput[$i]) Next The script above reads the whole directory into a one dimensional array, but I need to work with the array, so I need to split the array into multiple dimensions.
      I have already read some forum answers here, and I have already tried these commands:
       
      Are there any way to use the $aOutput variable like in PowerShell:
      PowerShell:
      $a = Get-ChildItem $a.Mode I imagine this in AutoIt  $aOutput
      ConsoleWrite($aOutput[i].Mode) Or if I split this command into 2 dimension like:
      For $i To UBound($aOutput)-1 Step 1 ConsoleWrite($aOutput[$i][1]) ConsoleWrite($aOutput[$i][2]) Next  
    • By Mannyfresh31
      Please help with this 3D Array the first example works the secound doesn't.
      Need help to understand how Arrays work.
      Many thanks in advance
      ;First Example Dim $aArray[2][2][2] $aArray[0][0][0] = 1 $aArray[0][0][1] = 2 $aArray[0][1][0] = 3 $aArray[0][1][1] = 4 $aArray[1][0][0] = 5 $aArray[1][0][1] = 6 $aArray[1][1][0] = 7 $aArray[1][1][1] = 8 For $a = 0 to 1 for $b = 0 to 1 for $c = 0 to 1 ConsoleWrite($aArray[$a][$b][$c] & @CRLF) Next Next Next ;Secound Example Local $aArraym [2][2][2]=[[[1,2,],[3,4],[5,6],[7,8]]] For $a = 0 to 1 for $b = 0 to 1 for $c = 0 to 1 ConsoleWrite($aArraym[$a][$b][$c] & @CRLF) Next Next Next  
×
×
  • Create New...