2 dimensional array, _INet, _IE help please

[ssubirias3]

Since my last request for additional help went unanswered, Let me try to explain this more better for the programmers / brains / gurus out there. big_daddy and PsaltyDS have given me some great code and I'm so close to having this finished, but I do need some help with the finishing touch.

The following is my version of big_daddy's and PsaltyDS' combined code. I'm using SciTE and will be referring to line numbers in an attempt to more clearly communicate my thoughts. Hopefully this will make it easier for people to follow along.

big_daddy & PsaltyDS combined code (begins on line 1 and ends on line 147)

expandcollapse popup

#include <IE.au3>
#include <File.au3>
#include <Array.au3>
#include <Date.au3>

; File to save data in:
$sFile = @ScriptDir & "\MarketWatchData.txt"

$sURL = "http://www.marketwatch.com/tools/marketsummary/screener.asp?"
_IEErrorHandlerRegister()
_IEToggleImages(False)
$oIE = _IECreate("about:blank", 0, 0, 1)
_IEToggleImages(True)

$i = 1
$iExch = 13
While $iExch < 16
    Switch $iExch
        Case 13
            $sExch = "NYSE"
        Case 14
            $sExch = "Amex"
        Case 15
            $sExch = "Nasdaq"
    EndSwitch
    Switch $i
        Case 1
            $iView = 0
        Case 2
            $iView = 1
        Case 3
            $iView = 6
        Case 4
            $iView = 7
    EndSwitch
    ConsoleWrite($sURL & "exchange=" & $iExch & "&view=" & $iView & @CR)
    TrayTip("MarketWatchData", "Getting data from: " & $sExch & @LF & "View " & $iView + 1 & " of 4", 10)
    _EnumTable($sURL & "exchange=" & $iExch & "&view=" & $iView)
    If $i = 4 Then
        $iExch += 1
        $i = 1
    Else
        $i += 1
    EndIf
    Sleep(5000)
WEnd

While 1
    Sleep(10)
WEnd

Func _EnumTable($s_URL)
    Local $oTable, $aResults, $aResults2[56][9], $sDate = _NowDate()
    ; Remove any previous data from the listview
    ;;  _GUICtrlListViewDeleteAllItems($ListView1)
    _IENavigate($oIE, $s_URL, 0)
    _IEErrorNotify(0)
    While Not IsObj($oTable)
        $oTable = _IETableGetCollection($oIE, 1)
        Sleep(100)
    WEnd
    _IEErrorNotify(1)
    While $oTable.readyState <> "complete"
        Sleep(100)
    WEnd
    _IEAction($oIE, "stop")
    $aResults = _IETableWriteToArray($oTable)
    _ArrayDisplay($aResults, "_ArrayDisplay() Test 0")

    For $i = 4 To UBound($aResults, 2) - 2
        ; There are seperator lines between each row of data we need, this bypasses those
        If $aResults[0][$i] = "0" Then ContinueLoop
        
        ; Create 2nd array to display results in desired format
        $aResults2[$i][0] = $aResults[0][$i]
        $aResults2[$i][1] = $aResults[1][$i]
        $aResults2[$i][2] = $aResults[2][$i]
        $aResults2[$i][3] = $aResults[3][$i]
        $aResults2[$i][4] = $aResults[4][$i]
        $aResults2[$i][5] = $aResults[5][$i]
        $aResults2[$i][6] = $aResults[6][$i]
        $aResults2[$i][7] = $sExch
        $aResults2[$i][8] = $sDate
        ; Create our file entry
        ; $sLine = $aResults[0][$i] & "|" & $aResults[1][$i] & "|" & $aResults[2][$i] & "|" & $aResults[3][$i] & "|" & _
        ; $aResults[4][$i] & "|" & $aResults[5][$i] & "|" & $aResults[6][$i] & "|" & $sExch
        ; _FileWriteLog($sFile, $sLine)

    Next
    _ArrayDisplay($aResults2, "_ArrayDisplay() Test 1")

    _IENavigate($oIE, "about:blank")
EndFunc   ;==>_EnumTable

Func _IEToggleImages($f_State = True)
    If $f_State Then
        $s_Status = "yes"
    Else
        $s_Status = "no"
    EndIf
    RegWrite("HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main", _
            "Display Inline Images", "REG_SZ", $s_Status)
EndFunc   ;==>_IEToggleImages

Func _IETable2Array_RC(ByRef $o_object)
    If Not IsObj($o_object) Then
        __IEErrorNotify("Error", "_IETable2Array", "$_IEStatus_InvalidDataType")
        SetError($_IEStatus_InvalidDataType, 1)
        Return 0
    EndIf
    ;
    If Not __IEIsObjType($o_object, "table") Then
        __IEErrorNotify("Error", "_IETable2Array", "$_IEStatus_InvalidObjectType")
        SetError($_IEStatus_InvalidObjectType, 1)
        Return 0
    EndIf
    ;
    Local $i_cols = 0, $trs, $tr, $tds, $i_col, $i_rows, $col, $row
    $trs = $o_object.rows
    For $tr In $trs
        $tds = $tr.cells
        $i_col = 0
        For $td In $tds
            $i_col = $i_col + $td.colSpan
        Next
        If $i_col > $i_cols Then $i_cols = $i_col
    Next
    $i_rows = $trs.length
    Local $a_TableCells[$i_rows][$i_cols]
    $row = 0
    For $tr In $trs
        $tds = $tr.cells
        $col = 0
        For $td In $tds
            $a_TableCells[$row][$col] = $td.innerText
            $col = $col + $td.colSpan
        Next
        $row = $row + 1
    Next
    SetError($_IEStatus_Success)
    Return $a_TableCells
EndFunc   ;==>_IETable2Array_RC

Func _Exit()
    _IEQuit($oIE)
    Exit
EndFunc   ;==>_ExitoÝ÷ ÚæÜ¢Z+ QtÓM,¥©ì·)^oÝ÷ Øì)z»(hv¬mr©ë^+§uê쵩ÝjëhÖ§uë)yÈ©®+ji¢0j{^vÚ,yì!jÜ(ºWaj^'Èr§çtߦi¹^ ée²Ú&j×!¶«£(é§×"Ø^jøvØ^ DÚnW®+^N+­¬¡×­¡·¢g6í®¶²D+zâ)ÚX§zÍtç^6jwkzZqçmê+§jYbî»jëh×6    $aResults = _IETableWriteToArray($oTable)oÝ÷ ÚÚì%wºí«­¢+ØÀÌØíIÍÕ±ÑÌô}%Q±ÉÉÉå}I ÀÌØí½Q±¤oÝ÷ Ù+«¦¸¬x%Èbv}ýµºÞjF§ÊØb§vØ^Ó~Eë.Û®¶²$yÙ¨­ébëazf«ëpj×!µ¦åx²¥)àu«Z~º&jwb³Z´(ºWaj÷Ƶçmæ«­¬­¢Z(X¤z)íçâ®Ëi§±ç²)à DÚnW¶ºÚÉmèÚ*¹ë-â|¯zyÞyÛazÇ¢wZ®¶²iû^­©eEè­Ù^²ËaÆ©¦ºr-Yey«Þ±I27[§­ë(÷J+zZr)ඨ®("©ew¬ë¾¼Â+ajëh×6$aResults = _IETable2Array_RC($oTable)
_ArrayDisplay($aResults, "_ArrayDisplay() - Before Delete")
_ArrayDelete($aResults, 1)
_ArrayDisplay($aResults, "_ArrayDisplay() - After Delete")

This killed the script and resulted in the following error to the console:

C:\Program Files\AutoIt3\Include\Array.au3 (139) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:

$avNewArray[$iCntr] = $avArray[$iCntr]

$avNewArray[$iCntr] = ^ ERROR

->23:17:33 AutoIT3.exe ended.rc:1

+>23:17:34 AutoIt3Wrapper Finished

>Exit code: 1 Time: 36.494

Q-2a) .. Why does _ArrayDelete($aResults,1) produce the above error?

Q-2b) .. If "Exit code: 1" means that $aResults isn't an array (per help file) then why does _ArrayDisplay work?

Q-2c) .. Any idea why _IETableWriteToArray doesn't use the regular [row][column] format when creating the array?

fig.4 (ArrayDisplay of $aResults - using _IETable2Array_RC)

QUESTION #3

Often I've read the forum and seen the seasoned people asking for more information or the code presenting the problem. I think information and code have been provided in this topic, but I'm still confused why folks are silent on this topic.

Q-3) .. Why aren't people responding to this thread when this seems to be an easy problem?

Thanks in advance for any and all help on this. ESPECIALLY big_daddy and PsaltyDS!!

--ss3

...

[Paulie]

Q-2a) .. Why does _ArrayDelete($aResults,1) produce the above error?

Q-2b) .. If "Exit code: 1" means that $aResults isn't an array (per help file) then why does _ArrayDisplay work?

The answer to this would be simply that _ArrayDelete is not capable of handling 2D arrays.

Example:

#include <Array.au3>

Dim $testarray[11][2]

For $i = 0 to 10
    For $x= 0 to 1
        $testarray[$i][$x] = Random(1,100,1)
    Next
Next
_ArrayDisplay($testarray,"Before")
_ArrayDelete($testarray, 4)
_ArrayDisplay($testarray,"After")

Not how the error code here is exactly the same as the one returned by your code, this would be because it is trying to ArrayDelete from a 2D array, that is why the "Incorrect Number Of Subscripts" error is returned.

Will see if i can find a work-around. Will get back to you.

EDIT: No dice on the workaround sorry

I now know why there isn't a 2D _ArrayDelete holy crap... confusing

Edited August 4, 2007 by Paulie

[randallc]

Hi,

Try the Array2D.au3 UDF from my sig;

$aResults = _IETable2Array_RC($oTable)
_ArrayDisplay($aResults, "_ArrayDisplay() - Before Delete")
_ArrayDelete($aResults, 1)
_ArrayDisplay($aResults, "_ArrayDisplay() - After Delete")oÝ÷ Ù«­¢+ØÀÌØíIÍÕ±ÑÌô}%Q±]É¥ÑQ½ÉÉä ÀÌØí½Q±¤(%}ÉÉäÉQɹÍÁ½ÍTÌ ÀÌØíIÍÕ±Ñ̤(}ÉÉå¥ÍÁ±ä ÀÌØíIÍÕ±ÑÌ°ÅÕ½Ðí}ÉÉå¥ÍÁ±ä ¤QÍÐÀÅÕ½Ðì¤(%}ÉÉå±ÑÉ ÀÌØíIÍÕ±Ñ̰Ĥ(}ÉÉå¥ÍÁ±ä ÀÌØíIÍÕ±ÑÌ°ÅÕ½Ðí}ÉÉå¥ÍÁ±ä ¤QÍÐÀÅÕ½Ðì¤

Best, Randall

[PS OR - look in the include files "array.au3" for a better transpose routine, perhaps, or just use the transpose parameter in ArrayDisplay() anyway?]

[MHz]

_ArrayDelete2D()

http://www.autoitscript.com/forum/index.ph...hl=_arraydelete

_2DArrayDelete()

http://www.autoitscript.com/forum/index.ph...hl=_arraydelete

Try either of the 2 above to delete from from a 2D array. If you delete from an array then the index count needs to also change.

Answer3.

We all have personal lives to live so do not get high expectations for priceless support

[PsaltyDS]

Stop trying to work on six question at once!

Starting from the version I gave you in post #18, the next step seemed to be generating a formatted array of data.

This appears to do it:

expandcollapse popup

#include <IE.au3>
#include <array.au3>
#include <File.au3>
#include <Date.au3>

; --------------------------------------------
; Global variables
; --------------------------------------------
; File to save data in:
Global $sFile = @ScriptDir & "\MarketWatchData.txt"
; Array of exchanges to loop through ([13] thru [15])
Global $avExch[16] = ["", "", "", "", "", "", "", "", "", "", "", "", "", "NYSE", "Amex", "Nasdaq"]
; Array of view selections to loop through ([0] thru [3])
Global $avViews[4] = [0, 1, 6, 7]
; --------------------------------------------

; Register IE.au3 default error handler
_IEErrorHandlerRegister()

; Create an invisible browser object with images off
_IEToggleImages(False)
$oIE = _IECreate("about:blank", 0, 0, 1)
_IEToggleImages(True)

; For each exchange...
For $iExch = 13 To 15
    ; For each view type...
    For $iView = 0 To 3
        ; Call _EnumTable() to get data
        _EnumTable($iExch, $iView)
    Next
Next

; Done, exit
_IEQuit($oIE)
Exit


;===============================================================================
; Function Name:    _EnumTable($E, $V)
; Description::     Enumerates MarketWatch data from web site
; Parameter(s):     $E - Specifies one of the exchanges from the $avExch array
;                   $V - Specifies one of the view indexes from the $avViews array
; Author(s):    Bob Anthony (big_daddy)
;               Modified by PsaltyDS to remove GUI functions
;===============================================================================
Func _EnumTable($E, $V)
    Local $oTable, $avResults, $avFormatted[1][9]
    Local $sURL = "http://www.marketwatch.com/tools/marketsummary/screener.asp?"
    Local $sCurrURL = $sURL & "exchange=" & $E & "&view=" & $avViews[$V]

    ; Keep user apprised on progress
    ConsoleWrite("Debug: $sCurrURL = " & $sCurrURL & @LF)
    TrayTip("MarketWatchData", "Getting data from: " & $avExch[$E] & @LF & "View " & $V + 1 & " of 4", 10)

    ; Navigate to the URL without waiting for the page to load
    _IENavigate($oIE, $sCurrURL, 0)

    ; Turn error notification while waiting from table to be available
    _IEErrorNotify(0)
    While Not IsObj($oTable)
        $oTable = _IETableGetCollection($oIE, 1)
        Sleep(100)
    WEnd
    _IEErrorNotify(1)

    ; Wait for our table to finish loading, otherwise we would get partial data
    While $oTable.readyState <> "complete"
        Sleep(100)
    WEnd

    ; Stop the page from loading more than the target table
    _IEAction($oIE, "stop")

    ; Write the table to an array (note: [column][row] format here)
    $avResults = _IETableWriteToArray($oTable)
    
    ; Start at $i = 4 because data starts in the fifth row (<tr>)
    ; Stop at total rows minus 2 to trim "Add symbols to My Portfolio" at the bottom
    For $i = 4 To UBound($avResults, 2) - 2
        ; Bypass seperator lines between each row
        If $avResults[0][$i] = "0" Then ContinueLoop

        ; Add data to formatted array
        ReDim $avFormatted[UBound($avFormatted) + 1][9]
        $avFormatted[0][0] = UBound($avFormatted) - 1
        For $c = 0 To 6
            $avFormatted[$avFormatted[0][0]][$c] = $avResults[$c][$i]
        Next
        $avFormatted[$avFormatted[0][0]][7] = $avExch[$E]
        $avFormatted[$avFormatted[0][0]][8] = _Now()
        
        ; Create file entry
        $sLine = $avResults[0][$i] & "|" & $avResults[1][$i] & "|" & $avResults[2][$i] & "|" & $avResults[3][$i] & "|" & _
                $avResults[4][$i] & "|" & $avResults[5][$i] & "|" & $avResults[6][$i] & "|" & $avExch[$E]
        _FileWriteLog($sFile, $sLine)
    Next

    ; Debug display
    _ArrayDisplay($avFormatted, "Debug: $avFormatted")
    
    ; Browse to blank page to clear data
    _IENavigate($oIE, "about:blank")
EndFunc   ;==>_EnumTable

;===============================================================================
; Function Name:   _IEToggleImages
; Description::    Toggles images on/off for IE.
; Parameter(s):    $f_State - Specifies whether images are turned on (Ture) or off (False).
; Author(s):       Bob Anthony (big_daddy)
;===============================================================================
Func _IEToggleImages($f_State = True)
    If $f_State Then
        $s_Status = "yes"
    Else
        $s_Status = "no"
    EndIf
    RegWrite("HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main", _
            "Display Inline Images", "REG_SZ", $s_Status)
EndFunc   ;==>_IEToggleImages

Whatever you want to do next DO NOT COPY/PASTE CODE INTO IT AND EXPECT IT TO WORK! Pick ONE thing you want to change AND MAKE THAT WORK BEFORE DOING ANYTHING ELSE!

[DaleHohm]

Transposing a 2D array is a pretty simple exercise. _IETableWriteToArray in the forthcoming version of IE.au3 will do this as an option.

Pass any 2D array (like the output from _IETableWriteToArray) to thsi function and the return value will be the transposed array.

Func Transpose($a2Din)
    Local $i_d1 = UBound($a2Din, 1), $i_d2 = UBound($a2Din, 2), $a2Dout[$i_d2][$i_d1]
    For $i = 0 To $i_d2 - 1
        For $j = 0 To $i_d1 - 1
            $a2Dout[$i][$j] = $a2Din[$j][$i]
        Next
    Next
    Return $a2Dout
EndFunc   ;==>Transpose

Dale

[ssubirias3]

Thank you Paulie, randallc, MHz, PsaltyDS, and Dale for the feedback! I can see that I have some more goodies to investigate. Should keep me busy for a little bit. You guys are really the greatest at helping idiots like me.

To be continued...

[ssubirias3]

............ Starting from the version I gave you in post #18, the next step seemed to be generating a formatted array of data.

This appears to do it: .......

Its amazing how easy you guys make this stuff look. Completely amazing! To think it was as easy as changing

For $i = 4 To UBound($aResults, 2) - 2
        ; There are seperator lines between each row of data we need, this bypasses those
        If $aResults[0][$i] = "0" Then ContinueLoop
        
        ; Create our file entry
        $sLine = $aResults[0][$i] & "|" & $aResults[1][$i] & "|" & $aResults[2][$i] & "|" & $aResults[3][$i] & "|" & _
                $aResults[4][$i] & "|" & $aResults[5][$i] & "|" & $aResults[6][$i] & "|" & $avExch[$E]
        _FileWriteLog($sFile, $sLine)
    Next

WOW -- Completely amazing what 7 lines of good code will do for a script! I am greatly thankful for everyone providing their feedback. One thing is for sure, I've learned a lot from this simple exercise. Again thank you all! big_daddy & PsaltyDS, I couldn't have done this without your code and charity. Life is good!