Screen scraping IE but too tired to find easy commands

[buckley15]

I have a rusty background in programming so after spending hours trying code and re-searching the forums and the Help, I've decided I better take a nap and ask for help.

I am opening an IE insurance claim screen, then switching over to an excel spreadsheet to get some data.  Pop back over to the IE and populate the fields and submit. (In the grand scheme of things, I'll somehow capture the output, (or open a dialogue (Gui) to have the user get what I need); then loop back to the excel and do it again.

I can handle the excel part.  What has been torturing me is how to populate the IE using tags or whatever with ControlSend.  I can make it work with clicks and tabs, but I thought the ControlSend would be more precise.  I'm missing the syntax or this IE form doesn't have what I need.

The code is below with my questions:

• Main question - how to use the ControlSend better
• 2nd question - I've conquered how to tell if the IE is already open (and open if not) but the diagnostics I put in the bottom don't work when it is open. (So I'm missing something there but its secondary and more a curiosity thing)

Under the code is the output from the diags statements, which I thought would show me the Input fields.  I've tried a dozen different variants of the ControlSend with the "Input tags" (or whatever they are) - but I'm exhausted.  Something isn't clicking in MY head.

Code:

expandcollapse popup

#include <IE.au3>
#include <Excel.au3>

; Check for an IE Claim window already signed on and open - if not there, open one
If Not (WinExists("[CLASS:IEFrame]", "Claim Inquiry - Windows Internet Explorer")) Then
   MsgBox(0, "IE", "Window does not exist - open it")
   Local $oInternet = ObjCreate("InternetExplorer.Application")
   $oInternet.Navigate("https://www.insurance.com/logon.xhtml") ; Opening a web page that contains a form

   $oInternet.Visible = 1 ; Let the guy show himself
   WinWaitActive("Log on - Windows Internet Explorer")
   Local $oDoc = $oInternet.document ; Example object to test
   Local $oForm = $oDoc.forms(0) ; Example object to test

   ;Put this in for my own clarification - which hasn't helped
   ;MsgBox(0, "", "Interface name of $oInternet is: " & ObjName($oInternet) & @CRLF & _
   ;        "Object name of $oInternet is:    " & ObjName($oInternet, 2) & @CRLF & _
   ;        "Interface name of $oDoc is:      " & ObjName($oDoc) & @CRLF & _
   ;        "Object name of $oDoc is:         " & ObjName($oDoc, 2) & @CRLF & _
   ;        "Interface name of $oForm is:     " & ObjName($oForm) & @CRLF & _
   ;        "Object name of $oForm is:        " & ObjName($oForm, 2))
        
   Local $handle = WinGetHandle("[CLASS:IEFrame]") ; thought if I started understanding handles, I could grab an already open IE window
   
   WinActivate ("Log on - Windows Internet Explorer")
   ;error check
   WinWaitActive("Log on - Windows Internet Explorer")
   ;error check
   send("signon{TAB}password")
   sleep(2)
   send("{Enter}")
   sleep(5); allow time for signon
   ; go to claim page
   $oInternet.Navigate("https://www.insurance.com/inquiry.xhtml") ; Opening a web page that contains a form
   ;error check

Else; Claim Window already open
   MsgBox(0,"IE","Claim window already open")
   local $oInternet = ObjGet("","Claim Inquiry - Windows Internet Explorer") ; Get an existing window
   Local $handle = WinGetHandle("[CLASS:IEFrame]", "Claim Inquiry - Windows Internet Explorer")
   MsgBox(0, "Handle", $handle)
EndIf

#cs --->  Run through info in an excel sheet
#ce<--------------
; back to IE
WinActivate ($handle)
WinWaitActive ($handle)
Opt("MouseCoordMode", 2) ;1=absolute, 0=relative, 2=client
ControlClick($handle, "", "", "Left", 1,187, 214); click into open field

;**************  Am using Sends or ControlSends, but would like to use ControlSend with a Tag
;Send($Excel_ID)
;Send("{TAB 6}")
;Send($Excel_SD & "{TAB}" & $Excel_SD)
;Send("{Enter}")
; OR
;ControlSend( $handle, "", "[CLASS:Internet Explorer_Server; INSTANCE:1]", $Excel_ID)
;sleep(3)
;ControlSend( $handle, "", "[CLASS:Internet Explorer_Server; INSTANCE:1]", "{TAB 6}")
;************* How do I specify the edit area to send to rather than tabbing?

; On a side note
;**********  These work if I create the IE object but fail if the window is already open
;**********  What am I doing wrong?
ConsoleWrite("_IETagNameGetCollection Start --------------------------------------------------------"&@CRLF)
local $oInputs = _IETagNameGetCollection($oInternet, "Input")
    For $oInput in $oInputs
        consolewrite(string($oInput.id) & @CRLF)
Next
ConsoleWrite("_IETagNameGetCollection Stop --------------------------------------------------------"&@CRLF&@CRLF)

ConsoleWrite("_IEFormGetCollection Start --------------------------------------------------------"&@CRLF)
$oforms = _IEFormGetCollection($oInternet,-1)
For $oform In $oforms     
   ConsoleWrite($oform.id & @CRLF);     
   $oeles = _IEFormElementGetCollection($oform)     
   For $oele In $oeles
      ConsoleWrite(@TAB & 'Name: ' & $oele.id & ' Tag Type: ' & $oele.tagName &@CRLF)     
   Next
Next
ConsoleWrite("_IEFormGetCollection Stop --------------------------------------------------------"&@CRLF&@CRLF)

ConsoleWrite("_IETagNameAllGetCollection Start --------------------------------------------------------"&@CRLF)
Local $oElements = _IETagNameAllGetCollection($oInternet)
For $oElement In $oElements
    ;MsgBox(0, "Element Info", "Tagname: " & $oElement.tagname & @CR & "innerText: " & $oElement.innerText)
consolewrite("Tagname: " & $oElement.tagname & @CR & "innerText: " & $oElement.innerText & @CRLF)
Next
ConsoleWrite("_IETagNameAllGetCollection Stop --------------------------------------------------------"&@CRLF)

------------------------------------------------------------------- Ouput from Diags - the fields I want to populate should be the RID, FromDOS and ThruDOS

_IETagNameGetCollection Start --------------------------------------------------------
sakClaim
RID
CLMICN
0
0
CLMICN
FromWarrant
FromDOS
ThruWarrant
ThruDOS
_IETagNameGetCollection Stop --------------------------------------------------------

_IEFormGetCollection Start --------------------------------------------------------
clmInq
    Name: sakClaim Tag Type: INPUT
_IEFormGetCollection Stop --------------------------------------------------------

_IETagNameAllGetCollection Start -------------------------------------------------------- *** This is an abbreviation, but the blue part is what I will be trying to automate the capture of down the road.  If anyone has the time to point me to an example, I'd appreciate it also ***
Tagname: !
innerText: 0
Tagname: HTML
innerText: Claim Inquiry

Main Claims Eligibility Pricing Prior Auth Budgets Trade Files   Account Mailbox Help Log Off .
 Tuesday 19 November 2013 10:46 am

Claim Inquiry: XXXXXX500D
CriteriaMember ID Claim Status Any Status Denied Paid Suspended Resubmit Claim Type Any Inpatient Outpatient Home Health Long Term Care Crossover - Inpatient Crossover - Outpatient Professional Crossover - Professional Dental Pharmacy Compound Drug
Patient Acct. # Date Type: Date of Service Warrant Date
ICN From Date  Thru Date  
 
Search

**** Here is what I will want to scrape - eventually ****
ICNMember IDPatient Acct. #Claim TypeFrom DateThru DateWarrant DateBilled AmountWarrant Amount Status
X999998X
4814976
M############ Professional 20130708 20130708 20130717 31.12 0 Denied
X999999X
4971686
M############ Professional 20130708 20130708 0 31.12 31.12 Paid

PreviousNext
----------------------------------------------------------

Many thanks for any help!

Rusty, tired old SQL programmer

[Danp2]

Main question - how to use the ControlSend better

 

Don't. Use the _IE* functions instead.

2nd question - I've conquered how to tell if the IE is already open (and open if not) but the diagnostics I put in the bottom don't work when it is open. (So I'm missing something there but its secondary and more a curiosity thing)

 

Again, I would use the _IE* functions instead of ObjCreate / ObjGet. Have a look in the help file at _IECreate and _IEAttach.

[somdcomputerguy]

I agree that you should study all the _IE functions available, in particular the function _IEFormElementSetValue should prove to be useful.

[buckley15]

Thanks.  I went back and converted everything I could find to the _IE elements (my understanding of IE OBJ is growing - but not fast enough).

Now it places the first value 100% correctly every time.  However there are 2 more fields plus the button to submit.  The other 2 fields only work the first time.  (I'm still missing something on the submit button too, but was only up until 3 AM). So the first run through works fine (push submit manually) but then every one in the loop after the first set ONLY fills in the first field.  I read a little about _IEGetObjByName losing its pointer, but I'm not sure what to do with it.  Any more advice would be greatly appreciated.

expandcollapse popup

#include <IE.au3>
#include <Excel.au3>
local $ExcelName = "Microsoft Excel - Book2", $ExcelRow = 2, $Excel_ID_Col = 1, $Excel_SD_Col  = 2, $Excel_BL_Col=3 ; will need to adjust to find data needed

Local $oIE = _IEAttach("https://www.insurance.com/Claims/inquiry.xhtml", "url")
If @error Then
   MsgBox(0, "Internet Explorer", "Need Insurance Window open to Claims. Error code: " & Hex(@error, 8))
   Exit
EndIf

Local $handle = WinGetHandle($oIE)
Local $objRID = _IEGetObjByName($oIE, "RID")
Local $objFromDOS = _IEGetObjByName($oIE, "FromDOS")
Local $objThruDOS = _IEGetObjByName($oIE, "ThruDOS")
Local $objSubmit = _IEGetObjByName($oIE, "submit_button")

; Excel should be open and rows and columns preset for now (Will add form to ask later on)
WinActivate ($ExcelName)
Local $oExcel = ObjGet("","Excel.Application") ; Get an existing Excel Object
If @error Then
    MsgBox(0, "ExcelTest", "Error Getting an active Excel Object. Looking for " & $ExcelName & "Error code: " & Hex(@error, 8))
    Exit
EndIf

$oExcel.Visible = 1 ; Let the guy show himself
Local $Excelhandle = WinGetHandle($ExcelName)
local $Excel_ID = "", $Excel_SD = '07/01/2013', $Excel_BL = 0, $Excel_Date = "00/00/0000";
; Read from Excel
$Excel_ID = _ExcelReadCell($oExcel, $ExcelRow, $Excel_ID_Col)
$Excel_SD = _ExcelReadCell($oExcel, $ExcelRow, $Excel_SD_Col)
$Excel_Date = StringMid($Excel_SD,5,2) & "/" & StringMid($Excel_SD,7,2) & "/" & StringLeft($Excel_SD,4)
$Excel_BL = _ExcelReadCell($oExcel, $ExcelRow, $Excel_BL_Col)

While $Excel_ID <> "" and $count <100
   $count = $count + 1; in case testing gets away from us

   WinActivate ("Claim Inquiry - Windows Internet Explorer"); IE
   WinWaitActive ("Claim Inquiry - Windows Internet Explorer"); IE

   ;*********  I added these to try and get the obj back but no success
   $oIE = _IEAttach("https://www.insurance.com/Claims/inquiry.xhtml", "url")
   $objRID = _IEGetObjByName($oIE, "RID")
   $objFromDOS = _IEGetObjByName($oIE, "FromDOS")
   $objThruDOS = _IEGetObjByName($oIE, "ThruDOS")
   $objSubmit = _IEGetObjByName($oIE, "submit_button")

   _IEFormElementSetValue($objRID, $Excel_ID) ; Works every Time
   _IEFormElementSetValue($objFromDOS, $Excel_Date); Only works once **********
   _IEFormElementSetValue($objThruDOS, $Excel_Date); Only works once **********
   _IEFormSubmit($objSubmit); Doesn't work at all ***************

   MSGBOX(0, "Results", $excel_ID & " - " & $Excel_SD & " - " & string($Excel_BL) )
   ; get next row in excel
   $ExcelRow = $ExcelRow + 1
   $Excel_ID = _ExcelReadCell($oExcel, $ExcelRow, $Excel_ID_Col)
   $Excel_SD = _ExcelReadCell($oExcel, $ExcelRow, $Excel_SD_Col)
   $Excel_BL = _ExcelReadCell($oExcel, $ExcelRow, $Excel_BL_Col)
WEnd
; I also tried switching to an Until ;until $Excel_ID ="" or $count > 5
; I also tried just doing 2 records with no loop - still RID works, others dont

msgbox(0,"Done", "Finished")

Great language with lots of possibiities for me.  I'm afraid I don't speak enough HTML to fully grasp the IE_Example though, but I'm working on it.  (Fortran was easy; SQL is easy... brain too old!)

Thanks

[Danp2]

A couple of observations...

1) Try removing the WinActivate and WinWaitActive commands for interacting with the IE window.

2) You don't need the _IEAttach inside your While loop.

3) You don't need the _IEGetObjByName commands outside your While loop

4) You have the Excel code in two places, before the While loop and at the bottom of the While loop. The first instance calculates $Excel_Date; the latter one does not. You should eliminate the 2nd section of this code and move the first to the beginning of your While loop.

[buckley15]

Thanks for the feedback.  I got that form to work and will save me hours.

I thought I was beginning to understand IE and names & ids but my next form has me baffled (again).

The form has a number of input fields, which I can reference and manipulate:

   Local $oQuery_fdos = _IEGetObjByName($oIE, "Query_fdos")

  _IEFormElementSetValue($oQuery_fdos, "3/13/2013")

However, the "Button" to Search has no id.  I've used AutoIt v3 Window Tool as well as ModiV2 and looped the _IETagNameGetCollection.  All the other fields show up as does a second button, but the primary search button does not.

ModiV2 only says:

attributes
onclick : Button_Click()
style : null
type : button

parent structure
html
body
form id="eligForm"
table
tbody
tr

 

The HTML line: <INPUT onclick=Button_Click(); type=button value=Search>

I've tried dozens of variations on:

Local $oSubmit = _IEGetObjByName($oIE, "onclick=Button_Click()") <-- with different second parameters (Followed by    _IEAction($oSubmit, "click") )

and

ControlClick("XXX - Windows Internet Explorer", "", "[CLASS:Button; TEXT=Search;]") <--- with different third parameters like: "[CLASS:Internet Explorer_Server; INSTANCE:1]"

and

ControlSend($handle, "", "[CLASS:Button; TEXT=Search]", "{Enter}") <---- again with different parameters  (My $handle works)

If I click it manually, I have no problem getting to the data the form returns.

(I have no Winactive or WinWaitactive commands).

Local $oIE = _IEAttach("https://www.insurance.com/Eligibility/recipients.xhtml", "url")

   Local $oQuery_Id = _IEGetObjByName($oIE, "Query_Id")
   Local $oQuery_fdos = _IEGetObjByName($oIE, "Query_fdos")
   Local $oQuery_tdos = _IEGetObjByName($oIE, "Query_tdos")
   Local $oSubmit = _IEGetObjByName($oIE, "onclick=Button_Click()"); *** with variations - usually Returns warnings
      
   _IEFormElementSetValue($oQuery_Id, "12345")
   _IEFormElementSetValue($oQuery_fdos, "3/13/2013")
   _IEFormElementSetValue($oQuery_tdos, "3/13/2013")
   _IEAction($oSubmit, "click")
   ;_IELoadWait($oIE)

OR
ControlClick("XXX - Windows Internet Explorer", "", "[CLASS:Button; TEXT=Search;]") ; **** With variations

OR
ControlSend($handle, "", "[CLASS:Internet Explorer_Server; INSTANCE:1]", "{Enter}")  ; **** With variations

OR
Local $oForm = _IEFormGetObjByName($oIE, "eligForm")
Local $oText = _IEFormElementGetObjByName($oForm, "onclick=Button_Click()") ; **** With variations
_IEAction($oForm, "click")

;************ answer ************
Local $oDiv = _IEGetObjById($oIE, "response")
local $message = _IEPropertyGet($oDiv, "innertext")
msgbox(0,"Answer", $message)

How can I access a button with no id? What am I missing?

(I have searched and searched - and tried hundred times....  PLEASE help)

great tool - I need to get better at it - quickly!

[Danp2]

In general, you don't want to use ControlClick, ControlSend, etc. when interacting with a web browser. Use the _IE functions instead.

Have you tried using _IEFormSubmit? You will need a reference to the desired form for this to work.

If that doesn't work, then you could review the code found >here and modify for your specific needs.

[buckley15]

Oh my goodness - thanks

I put in the code:

$oLinks = _IETagNameGetCollection($oIE, "input") 
   For $oLink In $oLinks     
      If String($oLink.type) = "button" And String($oLink.value) = "Search" Then           
         ConsoleWrite(String($oLink.type) & @CRLF & string($olink.value) & @CRLF)
      _IEAction($oLink, "click")           
      ExitLoop     
   EndIf 
Next 
_IELoadWait($oIE, 1000)

as suggested. It still wasn't "working" - complaining about dates that were put in correctly (and a dozen other ways).  I finally found if I set the "fire" manually, it worked:    _IEFormElementSetValue($oQuery_fdos, "040513", 1) but    _IEFormElementSetValue($oQuery_fdos, "040513", 0) also works!

Thanks for the tidbits.  I'll leave it up to someone else to understand WHY this form works with either FIRE.  I'm just ecstatic that it works.