downloading date ranges

[safdar19]

Hello everyone

I want to download data from www.bseindia.com.

this is a sample ulr:

http://www.bseindia.com/Hisbhav/eq020911_csv.zip

here in eq020911_csv.zip date is 02 september 2011

I can download data by using this code

Inetget("http://www.bseindia.com/Hisbhav/eq020911_csv.zip","E:\eq020911_csv.zip")

Now what I am not able to do is to download data by user defined date ranges.

For example, I am not able to download data from 01 jan 2010 to 25 jan 2011 or any user defined range.

Only "eq020911_csv.zip" would change in the ulr twise to any other date.

Also it should not abort if data for any date is not available. it should simply move on to the next date

Thanks in advance

Safdar

[Rogue5099]

This should download the whole year of 2011

For $iMonth = 1 To 12
For $iDay = 1 To 31
  $sDateFile = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip"
  Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDateFile, @DesktopDir & "\Test\" & $sDateFile)
Next
Next

Edited September 4, 2011 by rogue5099

[safdar19]

Thanks,

It helped.

[Rogue5099]

Edited post to fix couple of things, added progress bar.

#Include <Date.au3>
For $iMonth = 1 To 12
 ProgressOn("Download", _DateToMonth($iMonth))
 For $iDay = 1 To 31
  $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip"
  ProgressSet($iDay/31*100, "Downloading " & $sDate)
  Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate)
 Next
 ProgressOff()
Next
MsgBox(0, "Complete", "All Complete")

Edited September 4, 2011 by rogue5099

[Spiff59]

You could probably save a lot of internet traffic searching for invalid dates (Feb 29th, April 31st, etc) by using a loop that goes from your startdate to your endate and uses _dateAdd() to add 1 day each loop. Since this appears to be stock market info, I'm guessing you could cut down the internet calls by another 2/7th's by using a test _DateToDayofWeek() to discard searches that fall on a Saturday or Sunday.

[Rogue5099]

You could probably save a lot of internet traffic searching for invalid dates (Feb 29th, April 31st, etc) by using a loop that goes from your startdate to your endate and uses _dateAdd() to add 1 day each loop. Since this appears to be stock market info, I'm guessing you could cut down the internet calls by another 2/7th's by using a test _DateToDayofWeek() to discard searches that fall on a Saturday or Sunday.

#Include <Date.au3>
$Year = "2011"
For $iMonth = 1 To 12
 ProgressOn("Download", _DateToMonth($iMonth))
 For $iDay = 1 To _DateDaysInMonth($Year, $iMonth)
  If _DateToDayOfWeek($Year, StringFormat("%02d", $iMonth), StringFormat("%02d", $iDay)) > 1 Then
   If _DateToDayOfWeek($Year, StringFormat("%02d", $iMonth), StringFormat("%02d", $iDay)) < 7 Then
    $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip"
    ProgressSet($iDay/_DateDaysInMonth($Year, $iMonth)*100, "Downloading " & $sDate)
    Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate)
   EndIf
  EndIf
 Next
 ProgressOff()
Next
MsgBox(0, "Complete", "All Complete")

Edited September 4, 2011 by rogue5099

[Rogue5099]

Negating Sunday and Saturday results = 46 Files

Including Sunday and Saturday results = 68 Files

Best to keep them:

#Include <Date.au3>
$Year = "2011"
For $iMonth = 1 To 12
ProgressOn("Download", _DateToMonth($iMonth) & " " & $Year)
For $iDay = 1 To _DateDaysInMonth($Year, $iMonth)
  $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip"
  ProgressSet($iDay/_DateDaysInMonth($Year, $iMonth)*100, "Downloading " & $sDate)
  Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate)
Next
ProgressOff()
Next
MsgBox(0, "Complete", "All Complete")

Edited September 4, 2011 by rogue5099

[Spiff59]

#include <Date.au3>
Global $StartDate = "2010/01/01", $EndDate = "2011/01/25", $Search
Global $Days = _DateDiff("D", $StartDate, $EndDate)
For $x = 0 to $Days
$WorkDate = _DateAdd("D", $x, $StartDate)
$aWorkDate = StringSplit($WorkDate, "/")
Switch _DateToDayOfWeek($aWorkDate[1],$aWorkDate[2],$aWorkDate[3])
  Case 2 to 6
   $WorkDate = $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2)
   $FileName = "eq" & $WorkDate & "_csv.zip"
;   InetGet("http://www.bseindia.com/Hisbhav/" & $FileName, @DesktopDir & "\" & $FileName)
   $Search += 1; not required
EndSwitch
Next
MsgBox(1,"Search from " & $StartDate & " to " & $EndDate, $Days & " days reduced to " & $Search & " searches")

I see rogue has already made the refinements to reduce internet traffic while I was off making this...

It's basically the same thing...

[safdar19]

It worked as promised, only a folder named test on destop needed to be created. I still am not not able to work out a way to be able to download user defined. dates. that is it should ask for a date range.

Say from 25 feb 2011 to 11 march 2011

It has to said it worked great. downloaded very fast.

[Spiff59]

Since there is no ability to use wildcards with INetGet(), you're stuck doing an individual search for each filename.

Here's a UDF'ed version:

#include <Date.au3>
Download_Files("2010/01/01", "2011/01/25", @DesktopDir & "\Test\")
 
Func Download_Files($StartDate, $EndDate, $DestFolder)
For $x = 0 to _DateDiff("D", $StartDate, $EndDate)
  $WorkDate = _DateAdd("D", $x, $StartDate)
  $aWorkDate = StringSplit($WorkDate, "/")
  Switch _DateToDayOfWeek($aWorkDate[1],$aWorkDate[2],$aWorkDate[3])
   Case 2 to 6 ; ignore Sunday and Saturday
    $WorkDate = $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2)
    $FileName = "eq" & $WorkDate & "_csv.zip"
;   InetGet("[url="http://www.bseindia.com/Hisbhav/"]http://www.bseindia.com/Hisbhav/[/url]" & $FileName, $DestFolder & $FileName)
  EndSwitch
Next
EndFunc

PS - I hate how the new forum software screws around with pasted code removing blank lines and changing the indentation I do know implementing the new version is a lot of work, and am sure eventually all the kinks will be worked out. It is appreciated, Devs/Mods

Edit: It appears Saturday and Sunday data do appear on that site, so maybe just:

#include <Date.au3>
Download_Files("2011/01/01", "2011/01/25", @DesktopDir & "\Test\")
 
Func Download_Files($StartDate, $EndDate, $DestFolder)
  If Not FileExists($DestFolder) Then DirCreate($DestFolder)
  For $x = 0 to _DateDiff("D", $StartDate, $EndDate)
    $aWorkDate = StringSplit(_DateAdd("D", $x, $StartDate), "/")
    $FileName = "eq" & $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2) & "_csv.zip"
    InetGet("[url="http://www.bseindia.com/Hisbhav/"]http://www.bseindia.com/Hisbhav/[/url]" & $FileName, $DestFolder & $FileName)
  Next
EndFunc

Edited September 4, 2011 by Spiff59

[Rogue5099]

I got bored here ya go:

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <Date.au3>
GUICreate("Download Between", 200, 200, 800, 200)
GUICtrlCreateLabel("Start Date:", 10, 10)
$Date1 = GUICtrlCreateDate("", 10, 40, 185, 20)
GUICtrlCreateLabel("End Date:", 10, 70)
$Date2 = GUICtrlCreateDate("", 10, 100, 185, 20)
$Button = GUICtrlCreateButton("OK", 60, 140, 70)
$DTM_SETFORMAT_ = 0x1032
$style = "yyyy/MM/dd"
GUICtrlSendMsg($Date1, $DTM_SETFORMAT_, 0, $style)
GUICtrlSendMsg($Date2, $DTM_SETFORMAT_, 0, $style)
GUISetState()
While 1
 $msg = GUIGetMsg()
 Switch $msg
  Case $GUI_EVENT_CLOSE
   Exit
  Case $Button
   ExitLoop
 EndSwitch
WEnd
$Start = GUICtrlRead($Date1)
$End = GUICtrlRead($Date2)
GUIDelete()
If $Start = "" Or $End = "" Then Exit
Download_Between($Start, $End, @DesktopDir & "\Test\")
Func Download_Between($StartDate, $EndDate, $Destination)
 DirCreate($Destination)
 $sDays = _DateDiff("D", $StartDate, $EndDate)
 ProgressOn("Download", "", "", 0, 0)
 For $x = 0 to $sDays
  $dDate = _DateAdd("D", $x, $StartDate)
  $aDate = StringSplit($dDate, "/")
  $dDate = $aDate[3] & $aDate[2] & StringRight($aDate[1], 2)
  $sFile = "eq" & $dDate & "_csv.zip"
  ProgressSet($x/$sDays*100, "Downloading " & $sFile & @CRLF & Int($x/$sDays*100) & "%", _DateToMonth($aDate[2]) & " " & $aDate[1])
  InetGet("http://www.bseindia.com/Hisbhav/" & $sFile, $Destination & $sFile)
 Next
 ProgressOff()
 Return 0
EndFunc

[safdar19]

rogue5099, the last code works perfect !!

Great work !!

One last request. There is no way to extract the compressed file. So Run() function will have to be applied at the end, to run winrar.

Run("D:\Program Files\WinRAR\WinRAR.exe")

these extracted file need to be processed in a certain way. As these files are .csv, and not .xlsx , these files can not be processed( only a few colums and rows need to adjusted , this can be easily done) .auto it has functions for exec files and not .csv files.

any suggestions?

[Rogue5099]

Check this out instead of using winrar:

Then depending on what data you want depends on the next steps you take.

Edited September 17, 2011 by rogue5099

[Juvigy]

There was a CSV udf so it should be easy. Also you can use the Excel udf - CSV files can be opened by excel.