oemript

Powershell - how to import data from aspx websites into text file?

15 posts in this topic

Referring to following link, I would like to know on how to save data from aspx websites into text file using Powershell.


When I input any stock code, such as 5, web page display related information, and would like to know on how to save those data into Text file.


Does anyone have any suggestions?
Thanks in advance for any suggestions


http://www.hkexnews.hk/sdw/search/searchsdw.aspx

Share this post


Link to post
Share on other sites



oemript,

As all your threads seem to deal with Powershell, would you not be better off posting them on a more focused forum? Here we focus on Autoit and any questions solely relating to other apps/languages take very much the back seat.

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

@oemript 

Actually this is now possible in 2 ways ...

1. Using DotNet CLR 

 

See :  .NET CLR Danyfirex Examples.au3 included in the ZIP file Example 6.

Func _Example6()
    Local $oAssembly = _CLR_LoadLibrary("System.Net")
    ConsoleWrite("!$oAssembly: " & IsObj($oAssembly) & @CRLF)
    Local $oWebClient = _CLR_CreateObject($oAssembly, "System.Net.WebClient")
    ConsoleWrite("!$oWebClient: " & IsObj($oWebClient) & @CRLF)
    Local $sString = $oWebClient.DownloadString("http://checkip.dyndns.org")
    ConsoleWrite("!" & $sString & @CRLF)
    $sString = StringRegExp($sString, '((?:\d{1,3}\.){3}\d{1,3})', 3)
    MsgBox(0, "WebClient.DownloadString", "Current IP Address: " & (IsArray($sString) ? $sString[0] : "Ups something was wrong... :("))
EndFunc   ;==>_Example6

2. Using Using DotNet CLR / Powershell Automation

#include "CLR.Au3"

    _Run_PSHost_Script('$client = new-object System.Net.WebClient' & @CRLF & '$Client.DownloadFile("https://msdn.microsoft.com/en-us/library/system.management.automation.psobject(v=vs.85).aspx",[Environment]::GetFolderPath("Desktop")+"\ASPX.txt")')

Func _Run_PSHost_Script($PSScript, $iOutput = 0)
    Local $oAssembly = _CLR_LoadLibrary("System.Management.Automation")
    ConsoleWrite("!$oAssembly: " & IsObj($oAssembly) & @CRLF)

    ; Create Object
    Local $pAssemblyType = 0
    $oAssembly.GetType_2("System.Management.Automation.PowerShell", $pAssemblyType)
    ConsoleWrite("$pAssemblyType = " & Ptr($pAssemblyType) & @CRLF)

    Local $oActivatorType = ObjCreateInterface($pAssemblyType, $sIID_IType, $sTag_IType)
    ConsoleWrite("IsObj( $oAssemblyType ) = " & IsObj($oActivatorType) & @TAB & @CRLF)

    ; Create Object
    Local $pObjectPS = 0
    $oActivatorType.InvokeMember_3("Create", 0x158, 0, 0, 0, $pObjectPS)
    ConsoleWrite("IsObject: " & IsObj($pObjectPS) & @TAB & "$pObject: " & ObjName($pObjectPS) & @CRLF)

; <<<<<<<<<<<<<<<<<<< PS COMMAND HERE >>>>>>>>>>>>>>>>>>>>

    $pObjectPS.AddScript($PSScript) 

    $objAsync = $pObjectPS.BeginInvoke()
;~      ConsoleWrite("$objAsync " & IsObj($objAsync & @TAB & "$pObject: " & ObjName($objAsync) ) & @CRLF)

    While $objAsync.IsCompleted = False
;~      ConsoleWrite($objAsync.IsCompleted & @CRLF)
        ContinueLoop
    WEnd
        ConsoleWrite("Completed : " & $objAsync.IsCompleted & @CRLF)

    $objPsCollection = $pObjectPS.EndInvoke($objAsync)

EndFunc

Enjoy :)

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

I would like to know on whether above coding can be run under PowerShell or not, which seems to be .Net coding.

Do you have any suggestions?
Thanks, to everyone very much for any suggestions (^v^)

Edited by Melba23
Huge quote removed

Share this post


Link to post
Share on other sites

oemript,

When you reply, please use the "Reply to this topic" button at the top of the thread or the "Reply to this topic" editor at the bottom rather than the "Quote" button - responders know what they wrote and it just pads the thread unnecessarily.

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

@oemript 

1. PowerShell :

The script used and executed is plain PowerShell ?

You just copy and paste it in your PowerShell Console or PowerShell ISE.

$client = new-object System.Net.WebClient; $Client.DownloadFile("https://msdn.microsoft.com/en-us/library/system.management.automation.psobject(v=vs.85).aspx",[Environment]::GetFolderPath("Desktop")+"\ASPX.txt

Give it a try and you will see that this is working out of the box.

1.  DotNet or PowerShell :

As you might know PowerShell runs on top of .NET... , so we can access all .NET classed using PoswerShell :)

Because Powershell has no Cmdlet to download files from the internet. We therefore need to use a DotNET class to overcome this.

So basically we use the .NET "System.Management.Automation" Assembly in AutoIT.

Local $oAssembly = _CLR_LoadLibrary("System.Management.Automation")

To run and execute the PowerShell Script we use "System.Management.Automation.PowerShell" Class.

If you want to know more about all of this ... please check here  :

At the bottom of the first post there is a nice PDF that explains the whole concept behind it : .NET CLR Framework for AutoIT.pdf

 

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

Iin order to retrieving text by submitting 5 on following link using PowerShell and save into a text file as final result.
http://www.hkexnews.hk/sdw/search/searchsdw.aspx

$client = new-object System.Net.WebClient; $Client.DownloadFile("https://msdn.microsoft.com/en-us/library/system.management.automation.psobject(v=vs.85).aspx",[Environment]::GetFolderPath("Desktop")+"\ASPX.txt

$client = new-object System.Net.WebClient; $Client.DownloadFile("http://www.hkexnews.hk/sdw/search/searchsdw.aspx",[Environment]::GetFolderPath("Desktop")+"\ASPX.txt

I would like to know on where to insert 5 on above parameter.

Do you have any suggestions?
Thanks, to everyone very much for any suggestions (^v^)

 

Edited by oemript

Share this post


Link to post
Share on other sites

Hi oemript,

I am not sure why you want to use PowerShell persé, in order to get the data from a website? 

When I enter 5 I get the result listed for HSBC bank correct ?

If you want to do Webbrowser automation AutoIT has much better tools to do so... 

Maybe this can give you a good headstart : 

 

Regards (in Kowloon :-)

ptrex

 

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

I prefer to use PowerShell for this task, based on following coding, do you have any suggestions on how to insert 5 on above parameter, then download it and save as a text file.

$client = new-object System.Net.WebClient; $Client.DownloadFile("http://www.hkexnews.hk/sdw/search/searchsdw.aspx",[Environment]::GetFolderPath("Desktop")+"\ASPX.txt

Do you have any suggestions?
Thanks, to everyone very much for any suggestions (^v^)

Edited by oemript

Share this post


Link to post
Share on other sites

Sorry I don't have examples on this myself ... but there's plenty them around on the internet...

This is a good start already :

http://www.westerndevs.com/simple-powershell-automation-browser-based-tasks/ 

I don't have the time right now to do a fuil coding on your requirements, but maybe next week, can't make any promises though...

 

Share this post


Link to post
Share on other sites
2 minutes ago, ptrex said:

Sorry I don't have examples on this myself ... but there's plenty them around on the internet...

This is a good start already :

http://www.westerndevs.com/simple-powershell-automation-browser-based-tasks/ 

I don't have the time right now to do a fuil coding on your requirements, but maybe next week, can't make any promises though...

 

No problem, I can wait and very appreciated your time and efforts.

Thanks, to everyone very much for any suggestions (^v^)

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

Here you go ...

$ieObject = New-Object -ComObject 'InternetExplorer.Application'

$ieObject

$ieObject.Visible = $true

$ieObject.Navigate("http://www.hkexnews.hk/sdw/search/searchsdw.aspx")

while($ieObject.Busy) { Start-Sleep -Milliseconds 50 }

$Doc = $ieObject.Document

# $Doc.getElementsByName("txtStockCode") | % {$_.Name,$_.Type,$_.Value}

$Doc.getElementById("txtStockCode").value = "5"

$Doc.getElementsByName("btnSearch") | % {$_.Name,$_.Id} 

$Button = $Doc.getElementById("btnSearch")

$Button.click()

# $Doc.body.innerHTML

$Doc.body.innerHTML | Out-File "C:\Temp\HTML.txt" 


$ieObject.Quit()

Success !

 

Edited by ptrex

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

I have tried those coding, it seems that there is a problem on saving content into text file and ie cannot be closed automatically.

# $Doc.body.innerHTML

$Doc.body.innerHTML | Out-File "C:\Temp\HTML.txt"

Furthermore, I have tried following simple html page, the coding still get problem to save into a text file too. The text file is blank without any text.

$ieObject = New-Object -ComObject 'InternetExplorer.Application'

$ieObject

$ieObject.Visible = $true

$ieObject.Navigate("http://www.hkexnews.hk/index_c.htm")
# $Doc.body.innerHTML

$Doc.body.innerHTML | Out-File "C:\Temp\HTML.txt"
$ieObject.Quit()

Do you have any suggestions on how to solve this issue?

Thanks, to everyone very much for any suggestions (^v^)

Edited by oemript

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

1 hour ago, ptrex said:

All works fine on my side ?

So best is that you take this up in a PowerShell Forum now, since these questions are really PS specific....

rgds

P.

 

I am working on this issues on PowerShell Forum.

Thanks, to everyone very much for any suggestions (^v^)

Edited by oemript

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now