Jump to content
rm4453

[SOLVED] How to add a progress bar to _IETableWriteToArray

Recommended Posts

Hello,

 

I am currently writing a program that parses a massive table from a website, and need a way to add a progress bar while parsing.

I am currently using the function _IETableWriteToArray($oObj, True) to parse the array. I need the progress bar to update as the table is parsed, not just at the end of the parsing.

Any help at all would be very much appreciated!

 

*EDIT --> The array I am left with after parsing is $array[0-50000][16]

Edited by rm4453
[SOLVED]

Share this post


Link to post
Share on other sites

Hi @rm4453

a bad idea might be to change the IE library.... ( I suggest not to do it :thumbsdown:) , a better idea is to only clone the _IETableWriteToArray function and modify it by adding the progressbar and then use the modified function instead of the original one (in the modified function I marked with "; <-------- mod #" the lines I added):

#include <Array.au3>
#include <IE.au3>

Local $oIE = _IECreate("https://en.wikipedia.org/wiki/List_of_North_American_settlements_by_year_of_foundation")
Local $oTable = _IETableGetCollection($oIE, 1)
Local $aTableData = _IETableWriteToArray_mod($oTable, True)
_ArrayDisplay($aTableData)

_IEQuit($oIE)

; #FUNCTION# ====================================================================================================================
; Author ........: Dale Hohm
; modified ......: Chimp
; ===============================================================================================================================
Func _IETableWriteToArray_mod(ByRef $oObject, $bTranspose = False)
    If Not IsObj($oObject) Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidDataType")
        Return SetError($_IESTATUS_InvalidDataType, 1, 0)
    EndIf
    ;
    If Not __IEIsObjType($oObject, "table") Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidObjectType")
        Return SetError($_IESTATUS_InvalidObjectType, 1, 0)
    EndIf
    ;
    Local $iCols = 0, $oTds, $iCol
    Local $oTrs = $oObject.rows
    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $iCol = $iCol + $oTd.colSpan
        Next
        If $iCol > $iCols Then $iCols = $iCol
    Next
    Local $iRows = $oTrs.length
    Local $aTableCells[$iCols][$iRows]
    Local $iRow = 0
    Local $iPercent ; <-------- mod 1

    ProgressOn("Progress Meter", "processing rows ", 0 & "/" & $iRows, -1, -1, 16) ; <-------- mod 2

    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $aTableCells[$iCol][$iRow] = String($oTd.innerText)
            If @error Then ; Trap COM error, report and return
                __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_COMError", @error)
                Return SetError($_IESTATUS_ComError, @error, 0)
            EndIf
            $iCol = $iCol + $oTd.colSpan
        Next
        $iRow = $iRow + 1
        $iPercent = Int($iRow / $iRows * 100) ; <-------- mod 3
        ProgressSet($iPercent, $iRow & "/" & $iRows & @TAB & "(" & $iPercent & "%)") ; <-------- mod 4
    Next
    If $bTranspose Then
        Local $iD1 = UBound($aTableCells, $UBOUND_ROWS), $iD2 = UBound($aTableCells, $UBOUND_COLUMNS), $aTmp[$iD2][$iD1]
        For $i = 0 To $iD2 - 1
            For $j = 0 To $iD1 - 1
                $aTmp[$i][$j] = $aTableCells[$j][$i]
            Next
        Next
        $aTableCells = $aTmp
    EndIf
    ProgressOff() ; <-------- mod 5
    Return SetError($_IESTATUS_Success, 0, $aTableCells)
EndFunc   ;==>_IETableWriteToArray_mod

 

Edited by Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites
3 minutes ago, Chimp said:

Hi @rm4453

a bad idea might be to change the IE library.... ( I suggest not to do it :thumbsdown:) , a better idea is to only clone the _IETableWriteToArray function and modify it by adding the progressbar and then use the modified function function instead of the original one (in the modified function I marked with "; <-------- mod #" the lines I added):

 

Hello, @Chimp

 

I added your modified function to the IE library, and renamed it to _IETableWriteToArray_ProgressBar. I am testing it right now to see how well it works out, and appreciate all the help so far!

Share this post


Link to post
Share on other sites

@Chimp

I have added one more modification, and it is working perfectly thank you!

*The modification allows you to append a message to be on the progress bar at the end of the function inputs.*

; #FUNCTION# ====================================================================================================================
; Author ........: Dale Hohm
; modified ......: Chimp
; ===============================================================================================================================
Func _IETableWriteToArray_ProgressBar(ByRef $oObject, $bTranspose = False, $message = "Processing Data!") ; <------- If copied into IE.au3 can call normally with _ProgressBar appended to have progress bar pop up, you can also include a message as the last arg if wanted!
    If Not IsObj($oObject) Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidDataType")
        Return SetError($_IESTATUS_InvalidDataType, 1, 0)
    EndIf
    ;
    If Not __IEIsObjType($oObject, "table") Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidObjectType")
        Return SetError($_IESTATUS_InvalidObjectType, 1, 0)
    EndIf
    ;
    Local $iCols = 0, $oTds, $iCol
    Local $oTrs = $oObject.rows
    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $iCol = $iCol + $oTd.colSpan
        Next
        If $iCol > $iCols Then $iCols = $iCol
    Next
    Local $iRows = $oTrs.length
    Local $aTableCells[$iCols][$iRows]
    Local $iRow = 0
    Local $iPercent ; <-------- mod 1

    ProgressOn("Progress Meter", $message, 0 & "/" & $iRows, -1, -1, 16) ; <-------- mod 2

    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $aTableCells[$iCol][$iRow] = String($oTd.innerText)
            If @error Then ; Trap COM error, report and return
                __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_COMError", @error)
                Return SetError($_IESTATUS_ComError, @error, 0)
            EndIf
            $iCol = $iCol + $oTd.colSpan
        Next
        $iRow = $iRow + 1
        $iPercent = Int($iRow / $iRows * 100) ; <-------- mod 3
        ProgressSet($iPercent, $iRow & "/" & $iRows & @TAB & "(" & $iPercent & "%)") ; <-------- mod 4
    Next
    If $bTranspose Then
        Local $iD1 = UBound($aTableCells, $UBOUND_ROWS), $iD2 = UBound($aTableCells, $UBOUND_COLUMNS), $aTmp[$iD2][$iD1]
        For $i = 0 To $iD2 - 1
            For $j = 0 To $iD1 - 1
                $aTmp[$i][$j] = $aTableCells[$j][$i]
            Next
        Next
        $aTableCells = $aTmp
    EndIf
    ProgressOff() ; <-------- mod 5
    Return SetError($_IESTATUS_Success, 0, $aTableCells)
EndFunc   ;==>_IETableWriteToArray_ProgressBar

 

Edited by rm4453
Fixed Spelling Mishap

Share this post


Link to post
Share on other sites

You are welcome :)
p.s. better to also add an ProgressOff() statement right after the line "If @error Then ; Trap COM error, report and return" so that the progress bar will be closed also in case of errors.


small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By MarkIT
      Hi AutoIT masters,
      Good day! Sorry to have bothered this forum but we really need help. We are working on an automation project that is running on VDI server. The BOTS are in .exe are running fine until AV detected them and deleted the files. The files were re-compiled and AV kept on deleting them. The copy of the .exe BOT deleted were sent to Symantec for whitelisting. After whitelisting, it is no longer deleted but no longer working as designed (showing Line script error). We checked the scripts and there were no issues since we run it using SciTE editor and it performed the desired task. Good thing we found on this thread the solution using .a3x and the BOTS worked fine and no longer deleted. Now, the problem is they are asking why the BOTS won't run in .EXE and what is the reason behind Symantec AV deleting them. We raised a case with Symantec but they cannot provide further information as they are always seeing the file as "False Positive". We even tested with Symantec turned off and those .EXE files are working fine, however, after re-enabling, it got deleted.
      Just seeking help on how to better convince them that it is really Symantec causing the issue and the .a3x file.
    • By moimon
      Hi all. 
      Sorry if my words are confusing because I am not good at English.
      I am writing code to embed website with Recaptcha in GUI. And then, the code will automatically press the "I'm not a robot" button. The problem here is:
         - The code still works fine when entering iFrame to interact with Recaptcha with the URL is "https://www.google.com/recaptcha/api2/demo"
      <snip>
       
         - But for other URLs (such as "https://recaptcha-demo.appspot.com/recaptcha-v2-checkbox.php"), the code cannot be entered into the iframe.
       
      Code:
      <snip>  
      I did a lot of research but didn't know why.
       I sincerely thanks for the help. 
    • By SkysLastChance
      I am pretty sure the site that I am using was updated and now I am not seeing the same tags/elements that I used to. I tried using a UIAspy to see if I could grab them that way with no luck. 
      What do I need to do to be able to automate this again. Are the elements hidden somewhere?
      Instead of highlighting established account is highlights all of the web page almost. How can I get to the detail again? 
      I used to grab establish account by the name. Now I have nothing to even try to grab. 
      There are not elements in the event viewer either. 
       


       
      This is what happens when I try to inspect the element and click established account. 
      I get the same type of results in chrome
    • By AutoitMike
      Scite 3.4.4
      Win 10
      I click "Help" or press F1, there is no response
      If I use the file explorer and double click Autoit.chm or Autoit3.chm help opens.
      There is no dialog to check or uncheck "Always ask before opening this file" when clicking on these files.
       
      If you are curious as to why I dont have the latest version, I am creating a back up laptop that has a VERY extensive automation application that I have written over the past 15 years.
       
      An extremely potent, powerful, needed function has been deleted in the upgrade of Autoit in recent years that I can not do without. If my main laptop dies, which it almost did, I am in a very bad position. So I bought the exact same laptop and I am "cofiguring" it to work exactly the same as my main laptop. However, this one has been "Upgraded" to Win 10 which I hope is not the problem. 
      Thanks for any help
    • By sksbir
      Hi
      Trying this from autoit v3.3.14.5 and SCITE 3.19.102.1901.0 :
      help file , page "Language Reference - Variables" , sample autoit script about maps:

      Maps must be declared before use by defining their scope using the 'Global/Local/Static' keywords. Local $mControls[]
      So is my test script : only with this local declation.
      -check syntax is OK
      - running script : 

      test.au3" (13) : ==> Variable subscript badly formatted.: Local $mControls[] Local $mControls[^ ERROR ->14:51:49 AutoIt3.exe ended.rc:1
      any clue ?
×
×
  • Create New...