Jump to content
rm4453

[SOLVED] How to add a progress bar to _IETableWriteToArray

Recommended Posts

Hello,

 

I am currently writing a program that parses a massive table from a website, and need a way to add a progress bar while parsing.

I am currently using the function _IETableWriteToArray($oObj, True) to parse the array. I need the progress bar to update as the table is parsed, not just at the end of the parsing.

Any help at all would be very much appreciated!

 

*EDIT --> The array I am left with after parsing is $array[0-50000][16]

Edited by rm4453
[SOLVED]

Share this post


Link to post
Share on other sites

Hi @rm4453

a bad idea might be to change the IE library.... ( I suggest not to do it :thumbsdown:) , a better idea is to only clone the _IETableWriteToArray function and modify it by adding the progressbar and then use the modified function instead of the original one (in the modified function I marked with "; <-------- mod #" the lines I added):

#include <Array.au3>
#include <IE.au3>

Local $oIE = _IECreate("https://en.wikipedia.org/wiki/List_of_North_American_settlements_by_year_of_foundation")
Local $oTable = _IETableGetCollection($oIE, 1)
Local $aTableData = _IETableWriteToArray_mod($oTable, True)
_ArrayDisplay($aTableData)

_IEQuit($oIE)

; #FUNCTION# ====================================================================================================================
; Author ........: Dale Hohm
; modified ......: Chimp
; ===============================================================================================================================
Func _IETableWriteToArray_mod(ByRef $oObject, $bTranspose = False)
    If Not IsObj($oObject) Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidDataType")
        Return SetError($_IESTATUS_InvalidDataType, 1, 0)
    EndIf
    ;
    If Not __IEIsObjType($oObject, "table") Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidObjectType")
        Return SetError($_IESTATUS_InvalidObjectType, 1, 0)
    EndIf
    ;
    Local $iCols = 0, $oTds, $iCol
    Local $oTrs = $oObject.rows
    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $iCol = $iCol + $oTd.colSpan
        Next
        If $iCol > $iCols Then $iCols = $iCol
    Next
    Local $iRows = $oTrs.length
    Local $aTableCells[$iCols][$iRows]
    Local $iRow = 0
    Local $iPercent ; <-------- mod 1

    ProgressOn("Progress Meter", "processing rows ", 0 & "/" & $iRows, -1, -1, 16) ; <-------- mod 2

    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $aTableCells[$iCol][$iRow] = String($oTd.innerText)
            If @error Then ; Trap COM error, report and return
                __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_COMError", @error)
                Return SetError($_IESTATUS_ComError, @error, 0)
            EndIf
            $iCol = $iCol + $oTd.colSpan
        Next
        $iRow = $iRow + 1
        $iPercent = Int($iRow / $iRows * 100) ; <-------- mod 3
        ProgressSet($iPercent, $iRow & "/" & $iRows & @TAB & "(" & $iPercent & "%)") ; <-------- mod 4
    Next
    If $bTranspose Then
        Local $iD1 = UBound($aTableCells, $UBOUND_ROWS), $iD2 = UBound($aTableCells, $UBOUND_COLUMNS), $aTmp[$iD2][$iD1]
        For $i = 0 To $iD2 - 1
            For $j = 0 To $iD1 - 1
                $aTmp[$i][$j] = $aTableCells[$j][$i]
            Next
        Next
        $aTableCells = $aTmp
    EndIf
    ProgressOff() ; <-------- mod 5
    Return SetError($_IESTATUS_Success, 0, $aTableCells)
EndFunc   ;==>_IETableWriteToArray_mod

 

Edited by Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites
3 minutes ago, Chimp said:

Hi @rm4453

a bad idea might be to change the IE library.... ( I suggest not to do it :thumbsdown:) , a better idea is to only clone the _IETableWriteToArray function and modify it by adding the progressbar and then use the modified function function instead of the original one (in the modified function I marked with "; <-------- mod #" the lines I added):

 

Hello, @Chimp

 

I added your modified function to the IE library, and renamed it to _IETableWriteToArray_ProgressBar. I am testing it right now to see how well it works out, and appreciate all the help so far!

Share this post


Link to post
Share on other sites

@Chimp

I have added one more modification, and it is working perfectly thank you!

*The modification allows you to append a message to be on the progress bar at the end of the function inputs.*

; #FUNCTION# ====================================================================================================================
; Author ........: Dale Hohm
; modified ......: Chimp
; ===============================================================================================================================
Func _IETableWriteToArray_ProgressBar(ByRef $oObject, $bTranspose = False, $message = "Processing Data!") ; <------- If copied into IE.au3 can call normally with _ProgressBar appended to have progress bar pop up, you can also include a message as the last arg if wanted!
    If Not IsObj($oObject) Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidDataType")
        Return SetError($_IESTATUS_InvalidDataType, 1, 0)
    EndIf
    ;
    If Not __IEIsObjType($oObject, "table") Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidObjectType")
        Return SetError($_IESTATUS_InvalidObjectType, 1, 0)
    EndIf
    ;
    Local $iCols = 0, $oTds, $iCol
    Local $oTrs = $oObject.rows
    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $iCol = $iCol + $oTd.colSpan
        Next
        If $iCol > $iCols Then $iCols = $iCol
    Next
    Local $iRows = $oTrs.length
    Local $aTableCells[$iCols][$iRows]
    Local $iRow = 0
    Local $iPercent ; <-------- mod 1

    ProgressOn("Progress Meter", $message, 0 & "/" & $iRows, -1, -1, 16) ; <-------- mod 2

    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $aTableCells[$iCol][$iRow] = String($oTd.innerText)
            If @error Then ; Trap COM error, report and return
                __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_COMError", @error)
                Return SetError($_IESTATUS_ComError, @error, 0)
            EndIf
            $iCol = $iCol + $oTd.colSpan
        Next
        $iRow = $iRow + 1
        $iPercent = Int($iRow / $iRows * 100) ; <-------- mod 3
        ProgressSet($iPercent, $iRow & "/" & $iRows & @TAB & "(" & $iPercent & "%)") ; <-------- mod 4
    Next
    If $bTranspose Then
        Local $iD1 = UBound($aTableCells, $UBOUND_ROWS), $iD2 = UBound($aTableCells, $UBOUND_COLUMNS), $aTmp[$iD2][$iD1]
        For $i = 0 To $iD2 - 1
            For $j = 0 To $iD1 - 1
                $aTmp[$i][$j] = $aTableCells[$j][$i]
            Next
        Next
        $aTableCells = $aTmp
    EndIf
    ProgressOff() ; <-------- mod 5
    Return SetError($_IESTATUS_Success, 0, $aTableCells)
EndFunc   ;==>_IETableWriteToArray_ProgressBar

 

Edited by rm4453
Fixed Spelling Mishap

Share this post


Link to post
Share on other sites

You are welcome :)
p.s. better to also add an ProgressOff() statement right after the line "If @error Then ; Trap COM error, report and return" so that the progress bar will be closed also in case of errors.


small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By LxP
      I'm trying to pass a nested array to a function, such that the function alters the inner array.
      I was surprised to find that this minimal reproducible example, despite its use of ByRef, seems to pass a copy of the inner array to the function:
      #include <Array.au3> ; a boring old array Local $aInnerArray[5] = [1, 2, 3, 4, 5] ; a one-element array containing a reference to the other array Local $aOuterArray[1] = [$aInnerArray] ; intention: take a nested array and alter its inner array ; reality: the inner array seems to be getting copied Func ChangeIt(ByRef $aOuter) Local $aInner = $aOuter[0] $aInner[2] = 0 EndFunc ; Expected: [1, 2, 3, 4, 5] ; Actual: [1, 2, 3, 4, 5] ✔ _ArrayDisplay($aInnerArray, 'Before') ; $aOuterArray passed by-ref, should receive reference to $aInnerArray ; Therefore should change $aInnerArray to [1, 2, 0, 4, 5] ChangeIt($aOuterArray) ; Expected: [1, 2, 0, 4, 5] ; Actual: [1, 2, 3, 4, 5] ✘ _ArrayDisplay($aInnerArray, 'After') I suspect that either:
      the copy is taking place in the first line of the function (I couldn't find a way to access the inner array without first assigning it to a variable though); or ByRef doesn't propagate into inner levels of the data structure being passed, which seems less likely to me. Could someone please point me in the right direction to get this working as intended?
      Update
      I misunderstood what was happening on this line:
      ; WRONG: ; a one-element array containing a reference to the other array Local $aOuterArray[1] = [$aInnerArray] In other languages, would give $aOuterArray a reference to $aInnerArray here. In AutoIt, a copy of $aInnerArray is placed in $aOuterArray. This was the primary source of my misunderstanding. I'm looking for a way, if possible, to store an array reference in another array.
    • By VinMe
      HELLO All,
      is there any function in Auto it to Remove the repeating  strings separated by ","  which is present in the array.
      ex.
      Available state
      a[0]= D97,D96,,D85,D86,D85,D86,D85,D86,D85,,D86,D85,D86,D85,D86
      a[1]=D85,D24,,,,,D85
      Required state
      a[0]= D97,D96,D85,D86,
      a[1]=D85,D24
      thank  you in advance!
      vin!
    • By ahha
      Okay I know this will be one of those - how stupid can I be when I see the answer but I'm baffled at the current time.
      #include <Debug.au3> Local $aArray[11][2] = [ [10,10],[1,5],[2,0],[3,0],[4,"M"],[5,0],[6,0],[7,0],[8,"M"],[9,0],[10,2] ] _DebugArrayDisplay($aArray, "$aArray") Local $iCount = 0 ;init Local $i For $i = 1 to $aArray[0][0] If $aArray[$i][1] = "M" Then $iCount = $iCount + 1 ;debug ;MsgBox($MB_OK + $MB_TOPMOST, "Debug", "$aArray[$i][0] = " & $aArray[$i][0] & @CRLF & "$aArray[$i][1] = " & $aArray[$i][1]) EndIf Next MsgBox($MB_OK + $MB_TOPMOST, "Info", "M's found = " & $iCount) When I run this it states there are 8 M's in the array.
    • By adjist
      Hello all! 
       
      Getting this error :
      (22) : ==> Variable used without being declared.: if $vNumber = 0 Then if ^ ERROR  
      But I'm sure I have defined the variable, as in the top of my script has 
      Global $vNumber = 0  
      How would I go about fixing this?
       
    • By D2thunder
      Hi all, I wasnt active in this Forum forum for Years. 😅
      Since I need something with Json in AutoIt i write a little UDF to save a variable to file to read it with another language - and back. After a while until my other script grows i thought - lets integrate all variables wich is possible, because i want have arrays to.
      This is my first public UDF in Autoit - normaly my AutoIt scripts are quick an dirty.... - like the examples for this udf. - someone want to write nice examples?
      I tried to write this UDF properly, cared for processing speed, low mem consumption as possible, a short code, and a high usability without beeing very familiar with arrays.
      What is it for? - convert nearly all types of variables to a json string for saving it in a file or to an ini. Read it back in to a same formated variable and preserv variable types. All array types are supported with nested arrays, the only restriction is ram and array/stringsize - and time if the array is very big. You can use Filters for variable types and some other options.
      If anyone found a bug or any suggestions, please post it.
       
      get Json.au3 from here Json.au3
       
       
      JsonVar 2019.01.25.1.zip
×
×
  • Create New...