oemript Posted May 20, 2017 Share Posted May 20, 2017 (edited) I have used the following url to fetch the historical data from yahoo finance. From last 16th May, 2017 the url is not working. http://ichart.finance.yahoo.com/table.csv?s=AAL&a=8&b=7&c=2003&d=4&e=16&f=2017&g=d&ignore=.csv Seems like they have changed the url and the new url is: https://query1.finance.yahoo.com/v7/finance/download/AAL?period1=1494873000&period2=1494959400&interval=1d&events=history&crumb=l0aEtuOKocj In the above changed URL has a session cookie which is crumb. Is there any idea how to get this cookie programmatically(in PowerShell)? Does anyone have any suggestions? Thanks in advance for any suggestions :> Edited May 21, 2017 by oemript Link to comment Share on other sites More sharing options...
oemript Posted May 21, 2017 Author Share Posted May 21, 2017 Nobody? Link to comment Share on other sites More sharing options...
Developers Jos Posted May 21, 2017 Developers Share Posted May 21, 2017 This isn't a powershell community, but rather more an Autoit3 scripting one, so maybe not the most logical place to ask? Jos SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
Danyfirex Posted May 21, 2017 Share Posted May 21, 2017 Hello. You can use Powershell WebRequest. Basically You need to do a first request to yahoo finance page parse the html an get crumb then you add it to your query url and do the second web request. Saludos Danysys.com AutoIt... UDFs: VirusTotal API 2.0 UDF - libZPlay UDF - Apps: Guitar Tab Tester - VirusTotal Hash Checker Examples: Text-to-Speech ISpVoice Interface - Get installed applications - Enable/Disable Network connection PrintHookProc - WINTRUST - Mute Microphone Level - Get Connected NetWorks - Create NetWork Connection ShortCut Link to comment Share on other sites More sharing options...
MattHiggs Posted May 21, 2017 Share Posted May 21, 2017 1 hour ago, Danyfirex said: Hello. You can use Powershell WebRequest. Basically You need to do a first request to yahoo finance page parse the html an get crumb then you add it to your query url and do the second web request. Saludos I would try using either CURL or wget (depending on exactly what you are trying to do) to obtain the information. These tools have an insane number of features and might let you accomplish what you want to accomplish. Link to comment Share on other sites More sharing options...
oemript Posted May 22, 2017 Author Share Posted May 22, 2017 (edited) On 5/21/2017 at 9:36 PM, Danyfirex said: Hello. You can use Powershell WebRequest. Basically You need to do a first request to yahoo finance page parse the html an get crumb then you add it to your query url and do the second web request. Once yahoo finance page parse the html into a text file D:\Yahoo.txt. Do you have any suggestions on what kind of coding can retrieve "CrumbStore" from text file? Thanks, to everyone very much for any suggestions Edited May 22, 2017 by oemript Link to comment Share on other sites More sharing options...
Danyfirex Posted May 22, 2017 Share Posted May 22, 2017 hello. Something like this should work. $Data = Get-Content 'c:\yahoo.txt' $result = [regex]::match($Data, '"CrumbStore":{"crumb":"(.*?)"').Groups[1].Value Saludos Danysys.com AutoIt... UDFs: VirusTotal API 2.0 UDF - libZPlay UDF - Apps: Guitar Tab Tester - VirusTotal Hash Checker Examples: Text-to-Speech ISpVoice Interface - Get installed applications - Enable/Disable Network connection PrintHookProc - WINTRUST - Mute Microphone Level - Get Connected NetWorks - Create NetWork Connection ShortCut Link to comment Share on other sites More sharing options...
oemript Posted May 22, 2017 Author Share Posted May 22, 2017 (edited) 20 minutes ago, Danyfirex said: hello. Something like this should work. $Data = Get-Content 'c:\yahoo.txt' $result = [regex]::match($Data, '"CrumbStore":{"crumb":"(.*?)"').Groups[1].Value Saludos Once $result is retrieved, such as "5SBcSO78HGZ". I would like to know on how to add "5SBcSO78HGZ" into the end of following link. https://query1.finance.yahoo.com/v7/finance/download/^GSPC?period1=1492876180&period2=1495468180&interval=1d&events=history&crumb= Furthermore, if I want to download the csv file, would following code be correct? (Invoke-WebRequest -Uri "https://query1.finance.yahoo.com/v7/finance/download/^GSPC?period1=1492876180&period2=1495468180&interval=1d&events=history&crumb=5SBcSO78HGZ" -UseBasicParsing).Content | Out-File -FilePath D:\GSPC.csv Do you have any suggestions? Thanks, to everyone very much for any suggestions Edited May 22, 2017 by oemript Link to comment Share on other sites More sharing options...
Danyfirex Posted May 22, 2017 Share Posted May 22, 2017 (edited) Just append $result to the url and do the request and do your own tests. Saludos Edited May 22, 2017 by Danyfirex Danysys.com AutoIt... UDFs: VirusTotal API 2.0 UDF - libZPlay UDF - Apps: Guitar Tab Tester - VirusTotal Hash Checker Examples: Text-to-Speech ISpVoice Interface - Get installed applications - Enable/Disable Network connection PrintHookProc - WINTRUST - Mute Microphone Level - Get Connected NetWorks - Create NetWork Connection ShortCut Link to comment Share on other sites More sharing options...
oemript Posted May 22, 2017 Author Share Posted May 22, 2017 (edited) 48 minutes ago, Danyfirex said: Just append $result to the url and do the request. Saludos (Invoke-WebRequest -Uri "https://finance.yahoo.com/quote/%5EGSPC/history?p=%5EGSPC" -SessionVariable $Yahoo -UseBasicParsing).Content | Out-File -FilePath D:\Yahoo.txt $Data = Get-Content 'd:\yahoo.txt' $result = [regex]::match($Data, '"CrumbStore":{"crumb":"(.*?)"').Groups[1].Value $URLresult = "https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1492876180&period2=1495468180&interval=1d&events=history&crumb=" + $result (Invoke-WebRequest -Uri $URLresult -WebSession $Yahoo -UseBasicParsing).Content | Out-File -FilePath D:\Yahoo.csv I get the correct URLresult, but when I download the file using Invoke-WebRequest on next step, it fails because of "Invalid cookie". Is there anything wrong on how to handle session by using -SessionVariable and -WebSession? Do you have any suggestions? Thanks, to everyone very much for any suggestions Edited May 22, 2017 by oemript Link to comment Share on other sites More sharing options...
oemript Posted May 24, 2017 Author Share Posted May 24, 2017 Referring to following coding, when I manually download file from Yahoo by clicking the download link, the data shows on different cells properly, but when I download using following coding, data is shown everything on the same single cell 1 without splitting into different column as shown below: Coding: $URL = "https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1493072773&period2=1495664773&interval=1d&events=history&crumb=kK.4Q7f/hsM" $file = "C:\GSCP.csv" (Invoke-WebRequest -Uri $URL -WebSession $Y -UseBasicParsing).Content | Out-File -FilePath $file File: Date,Open,High,Low,Close,Adj Close,Volume 1990-01-08,3842.000000,3942.000000,3742.000000,3850.000000,3850.000000,0 Does Powershell have other function to download file instead of using (...).Content? which "(...).Content" should be the problem on downloading as a text. When I try following coding, the format is wrong neither. Invoke-WebRequest -Uri $URL -WebSession $Y | Out-File -FilePath $file Do you have any suggestions? Thanks, to everyone very much for any suggestions Link to comment Share on other sites More sharing options...
Danyfirex Posted May 25, 2017 Share Posted May 25, 2017 Hello. You can do something like this. (Invoke-WebRequest -Uri $URL -WebSession $mysession -UseBasicParsing).Content | ConvertFrom-Csv -Delim ',' | Export-Csv -NoTypeInformation -Path $file -UseCulture Saludos Danysys.com AutoIt... UDFs: VirusTotal API 2.0 UDF - libZPlay UDF - Apps: Guitar Tab Tester - VirusTotal Hash Checker Examples: Text-to-Speech ISpVoice Interface - Get installed applications - Enable/Disable Network connection PrintHookProc - WINTRUST - Mute Microphone Level - Get Connected NetWorks - Create NetWork Connection ShortCut Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now