Sign in to follow this  
Followers 0
safdar19

downloading date ranges

14 posts in this topic

Hello everyone

I want to download data from www.bseindia.com.

this is a sample ulr:

http://www.bseindia.com/Hisbhav/eq020911_csv.zip

here in eq020911_csv.zip date is 02 september 2011

I can download data by using this code

Inetget("http://www.bseindia.com/Hisbhav/eq020911_csv.zip","E:\eq020911_csv.zip")

Now what I am not able to do is to download data by user defined date ranges.

For example, I am not able to download data from 01 jan 2010 to 25 jan 2011 or any user defined range.

Only "eq020911_csv.zip" would change in the ulr twise to any other date.

Also it should not abort if data for any date is not available. it should simply move on to the next date

Thanks in advance

Safdar

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

This should download the whole year of 2011

For $iMonth = 1 To 12
For $iDay = 1 To 31
  $sDateFile = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip"
  Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDateFile, @DesktopDir & "\Test\" & $sDateFile)
Next
Next
Edited by rogue5099

Share this post


Link to post
Share on other sites

Thanks,

It helped.

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Edited post to fix couple of things, added progress bar.

#Include <Date.au3>
For $iMonth = 1 To 12
 ProgressOn("Download", _DateToMonth($iMonth))
 For $iDay = 1 To 31
  $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip"
  ProgressSet($iDay/31*100, "Downloading " & $sDate)
  Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate)
 Next
 ProgressOff()
Next
MsgBox(0, "Complete", "All Complete")
Edited by rogue5099

Share this post


Link to post
Share on other sites

You could probably save a lot of internet traffic searching for invalid dates (Feb 29th, April 31st, etc) by using a loop that goes from your startdate to your endate and uses _dateAdd() to add 1 day each loop. Since this appears to be stock market info, I'm guessing you could cut down the internet calls by another 2/7th's by using a test _DateToDayofWeek() to discard searches that fall on a Saturday or Sunday.

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

You could probably save a lot of internet traffic searching for invalid dates (Feb 29th, April 31st, etc) by using a loop that goes from your startdate to your endate and uses _dateAdd() to add 1 day each loop. Since this appears to be stock market info, I'm guessing you could cut down the internet calls by another 2/7th's by using a test _DateToDayofWeek() to discard searches that fall on a Saturday or Sunday.

#Include <Date.au3>
$Year = "2011"
For $iMonth = 1 To 12
 ProgressOn("Download", _DateToMonth($iMonth))
 For $iDay = 1 To _DateDaysInMonth($Year, $iMonth)
  If _DateToDayOfWeek($Year, StringFormat("%02d", $iMonth), StringFormat("%02d", $iDay)) > 1 Then
   If _DateToDayOfWeek($Year, StringFormat("%02d", $iMonth), StringFormat("%02d", $iDay)) < 7 Then
    $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip"
    ProgressSet($iDay/_DateDaysInMonth($Year, $iMonth)*100, "Downloading " & $sDate)
    Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate)
   EndIf
  EndIf
 Next
 ProgressOff()
Next
MsgBox(0, "Complete", "All Complete")
Edited by rogue5099

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

Negating Sunday and Saturday results = 46 Files

Including Sunday and Saturday results = 68 Files

Best to keep them:

#Include <Date.au3>
$Year = "2011"
For $iMonth = 1 To 12
ProgressOn("Download", _DateToMonth($iMonth) & " " & $Year)
For $iDay = 1 To _DateDaysInMonth($Year, $iMonth)
  $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip"
  ProgressSet($iDay/_DateDaysInMonth($Year, $iMonth)*100, "Downloading " & $sDate)
  Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate)
Next
ProgressOff()
Next
MsgBox(0, "Complete", "All Complete")
Edited by rogue5099

Share this post


Link to post
Share on other sites

#include <Date.au3>
Global $StartDate = "2010/01/01", $EndDate = "2011/01/25", $Search
Global $Days = _DateDiff("D", $StartDate, $EndDate)
For $x = 0 to $Days
$WorkDate = _DateAdd("D", $x, $StartDate)
$aWorkDate = StringSplit($WorkDate, "/")
Switch _DateToDayOfWeek($aWorkDate[1],$aWorkDate[2],$aWorkDate[3])
  Case 2 to 6
   $WorkDate = $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2)
   $FileName = "eq" & $WorkDate & "_csv.zip"
;   InetGet("http://www.bseindia.com/Hisbhav/" & $FileName, @DesktopDir & "\" & $FileName)
   $Search += 1; not required
EndSwitch
Next
MsgBox(1,"Search from " & $StartDate & " to " & $EndDate, $Days & " days reduced to " & $Search & " searches")

I see rogue has already made the refinements to reduce internet traffic while I was off making this...

It's basically the same thing...

Share this post


Link to post
Share on other sites

It worked as promised, only a folder named test on destop needed to be created. I still am not not able to work out a way to be able to download user defined. dates. that is it should ask for a date range.

Say from 25 feb 2011 to 11 march 2011

It has to said it worked great. downloaded very fast.

Share this post


Link to post
Share on other sites

#10 ·  Posted (edited)

Since there is no ability to use wildcards with INetGet(), you're stuck doing an individual search for each filename.

Here's a UDF'ed version:

#include <Date.au3>
Download_Files("2010/01/01", "2011/01/25", @DesktopDir & "\Test\")
 
Func Download_Files($StartDate, $EndDate, $DestFolder)
For $x = 0 to _DateDiff("D", $StartDate, $EndDate)
  $WorkDate = _DateAdd("D", $x, $StartDate)
  $aWorkDate = StringSplit($WorkDate, "/")
  Switch _DateToDayOfWeek($aWorkDate[1],$aWorkDate[2],$aWorkDate[3])
   Case 2 to 6 ; ignore Sunday and Saturday
    $WorkDate = $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2)
    $FileName = "eq" & $WorkDate & "_csv.zip"
;   InetGet("[url="http://www.bseindia.com/Hisbhav/"]http://www.bseindia.com/Hisbhav/[/url]" & $FileName, $DestFolder & $FileName)
  EndSwitch
Next
EndFunc

PS - I hate how the new forum software screws around with pasted code removing blank lines and changing the indentation :) I do know implementing the new version is a lot of work, and am sure eventually all the kinks will be worked out. It is appreciated, Devs/Mods :mellow:

Edit: It appears Saturday and Sunday data do appear on that site, so maybe just:

#include <Date.au3>
Download_Files("2011/01/01", "2011/01/25", @DesktopDir & "\Test\")
 
Func Download_Files($StartDate, $EndDate, $DestFolder)
  If Not FileExists($DestFolder) Then DirCreate($DestFolder)
  For $x = 0 to _DateDiff("D", $StartDate, $EndDate)
    $aWorkDate = StringSplit(_DateAdd("D", $x, $StartDate), "/")
    $FileName = "eq" & $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2) & "_csv.zip"
    InetGet("[url="http://www.bseindia.com/Hisbhav/"]http://www.bseindia.com/Hisbhav/[/url]" & $FileName, $DestFolder & $FileName)
  Next
EndFunc
Edited by Spiff59

Share this post


Link to post
Share on other sites

I got bored here ya go:

#include <GUIConstantsEx.au3>
#include <Date.au3>
GUICreate("Download Between", 200, 200, 800, 200)
GUICtrlCreateLabel("Start Date:", 10, 10)
$Date1 = GUICtrlCreateDate("", 10, 40, 185, 20)
GUICtrlCreateLabel("End Date:", 10, 70)
$Date2 = GUICtrlCreateDate("", 10, 100, 185, 20)
$Button = GUICtrlCreateButton("OK", 60, 140, 70)
$DTM_SETFORMAT_ = 0x1032
$style = "yyyy/MM/dd"
GUICtrlSendMsg($Date1, $DTM_SETFORMAT_, 0, $style)
GUICtrlSendMsg($Date2, $DTM_SETFORMAT_, 0, $style)
GUISetState()
While 1
 $msg = GUIGetMsg()
 Switch $msg
  Case $GUI_EVENT_CLOSE
   Exit
  Case $Button
   ExitLoop
 EndSwitch
WEnd
$Start = GUICtrlRead($Date1)
$End = GUICtrlRead($Date2)
GUIDelete()
If $Start = "" Or $End = "" Then Exit
Download_Between($Start, $End, @DesktopDir & "\Test\")
Func Download_Between($StartDate, $EndDate, $Destination)
 DirCreate($Destination)
 $sDays = _DateDiff("D", $StartDate, $EndDate)
 ProgressOn("Download", "", "", 0, 0)
 For $x = 0 to $sDays
  $dDate = _DateAdd("D", $x, $StartDate)
  $aDate = StringSplit($dDate, "/")
  $dDate = $aDate[3] & $aDate[2] & StringRight($aDate[1], 2)
  $sFile = "eq" & $dDate & "_csv.zip"
  ProgressSet($x/$sDays*100, "Downloading " & $sFile & @CRLF & Int($x/$sDays*100) & "%", _DateToMonth($aDate[2]) & " " & $aDate[1])
  InetGet("http://www.bseindia.com/Hisbhav/" & $sFile, $Destination & $sFile)
 Next
 ProgressOff()
 Return 0
EndFunc

Share this post


Link to post
Share on other sites

rogue5099, the last code works perfect !!

Great work !!

One last request. There is no way to extract the compressed file. So Run() function will have to be applied at the end, to run winrar.

Run("D:\Program Files\WinRAR\WinRAR.exe")

these extracted file need to be processed in a certain way. As these files are .csv, and not .xlsx , these files can not be processed( only a few colums and rows need to adjusted , this can be easily done) .auto it has functions for exec files and not .csv files.

any suggestions?

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

Check this out instead of using winrar:

Then depending on what data you want depends on the next steps you take.

Edited by rogue5099

Share this post


Link to post
Share on other sites

There was a CSV udf so it should be easy. Also you can use the Excel udf - CSV files can be opened by excel.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0