safdar19 Posted September 4, 2011 Share Posted September 4, 2011 Hello everyoneI want to download data from www.bseindia.com.this is a sample ulr: http://www.bseindia.com/Hisbhav/eq020911_csv.ziphere in eq020911_csv.zip date is 02 september 2011I can download data by using this code Inetget("http://www.bseindia.com/Hisbhav/eq020911_csv.zip","E:\eq020911_csv.zip")Now what I am not able to do is to download data by user defined date ranges.For example, I am not able to download data from 01 jan 2010 to 25 jan 2011 or any user defined range.Only "eq020911_csv.zip" would change in the ulr twise to any other date.Also it should not abort if data for any date is not available. it should simply move on to the next dateThanks in advanceSafdar Link to comment Share on other sites More sharing options...
Rogue5099 Posted September 4, 2011 Share Posted September 4, 2011 (edited) This should download the whole year of 2011 For $iMonth = 1 To 12 For $iDay = 1 To 31 $sDateFile = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip" Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDateFile, @DesktopDir & "\Test\" & $sDateFile) Next Next Edited September 4, 2011 by rogue5099 My projects: Inventory / Mp3 Inventory, Computer Stats Link to comment Share on other sites More sharing options...
safdar19 Posted September 4, 2011 Author Share Posted September 4, 2011 Thanks, It helped. Link to comment Share on other sites More sharing options...
Rogue5099 Posted September 4, 2011 Share Posted September 4, 2011 (edited) Edited post to fix couple of things, added progress bar. #Include <Date.au3> For $iMonth = 1 To 12 ProgressOn("Download", _DateToMonth($iMonth)) For $iDay = 1 To 31 $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip" ProgressSet($iDay/31*100, "Downloading " & $sDate) Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate) Next ProgressOff() Next MsgBox(0, "Complete", "All Complete") Edited September 4, 2011 by rogue5099 My projects: Inventory / Mp3 Inventory, Computer Stats Link to comment Share on other sites More sharing options...
Spiff59 Posted September 4, 2011 Share Posted September 4, 2011 You could probably save a lot of internet traffic searching for invalid dates (Feb 29th, April 31st, etc) by using a loop that goes from your startdate to your endate and uses _dateAdd() to add 1 day each loop. Since this appears to be stock market info, I'm guessing you could cut down the internet calls by another 2/7th's by using a test _DateToDayofWeek() to discard searches that fall on a Saturday or Sunday. Link to comment Share on other sites More sharing options...
Rogue5099 Posted September 4, 2011 Share Posted September 4, 2011 (edited) You could probably save a lot of internet traffic searching for invalid dates (Feb 29th, April 31st, etc) by using a loop that goes from your startdate to your endate and uses _dateAdd() to add 1 day each loop. Since this appears to be stock market info, I'm guessing you could cut down the internet calls by another 2/7th's by using a test _DateToDayofWeek() to discard searches that fall on a Saturday or Sunday. #Include <Date.au3> $Year = "2011" For $iMonth = 1 To 12 ProgressOn("Download", _DateToMonth($iMonth)) For $iDay = 1 To _DateDaysInMonth($Year, $iMonth) If _DateToDayOfWeek($Year, StringFormat("%02d", $iMonth), StringFormat("%02d", $iDay)) > 1 Then If _DateToDayOfWeek($Year, StringFormat("%02d", $iMonth), StringFormat("%02d", $iDay)) < 7 Then $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip" ProgressSet($iDay/_DateDaysInMonth($Year, $iMonth)*100, "Downloading " & $sDate) Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate) EndIf EndIf Next ProgressOff() Next MsgBox(0, "Complete", "All Complete") Edited September 4, 2011 by rogue5099 My projects: Inventory / Mp3 Inventory, Computer Stats Link to comment Share on other sites More sharing options...
Rogue5099 Posted September 4, 2011 Share Posted September 4, 2011 (edited) Negating Sunday and Saturday results = 46 Files Including Sunday and Saturday results = 68 Files Best to keep them: #Include <Date.au3> $Year = "2011" For $iMonth = 1 To 12 ProgressOn("Download", _DateToMonth($iMonth) & " " & $Year) For $iDay = 1 To _DateDaysInMonth($Year, $iMonth) $sDate = StringFormat("%02d", $iMonth) & StringFormat("%02d", $iDay) & "11_csv.zip" ProgressSet($iDay/_DateDaysInMonth($Year, $iMonth)*100, "Downloading " & $sDate) Inetget("http://www.bseindia.com/Hisbhav/eq" & $sDate, @DesktopDir & "\Test\" & $sDate) Next ProgressOff() Next MsgBox(0, "Complete", "All Complete") Edited September 4, 2011 by rogue5099 My projects: Inventory / Mp3 Inventory, Computer Stats Link to comment Share on other sites More sharing options...
Spiff59 Posted September 4, 2011 Share Posted September 4, 2011 #include <Date.au3> Global $StartDate = "2010/01/01", $EndDate = "2011/01/25", $Search Global $Days = _DateDiff("D", $StartDate, $EndDate) For $x = 0 to $Days $WorkDate = _DateAdd("D", $x, $StartDate) $aWorkDate = StringSplit($WorkDate, "/") Switch _DateToDayOfWeek($aWorkDate[1],$aWorkDate[2],$aWorkDate[3]) Case 2 to 6 $WorkDate = $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2) $FileName = "eq" & $WorkDate & "_csv.zip" ; InetGet("http://www.bseindia.com/Hisbhav/" & $FileName, @DesktopDir & "\" & $FileName) $Search += 1; not required EndSwitch Next MsgBox(1,"Search from " & $StartDate & " to " & $EndDate, $Days & " days reduced to " & $Search & " searches") I see rogue has already made the refinements to reduce internet traffic while I was off making this... It's basically the same thing... Link to comment Share on other sites More sharing options...
safdar19 Posted September 4, 2011 Author Share Posted September 4, 2011 It worked as promised, only a folder named test on destop needed to be created. I still am not not able to work out a way to be able to download user defined. dates. that is it should ask for a date range. Say from 25 feb 2011 to 11 march 2011 It has to said it worked great. downloaded very fast. Link to comment Share on other sites More sharing options...
Spiff59 Posted September 4, 2011 Share Posted September 4, 2011 (edited) Since there is no ability to use wildcards with INetGet(), you're stuck doing an individual search for each filename. Here's a UDF'ed version: #include <Date.au3> Download_Files("2010/01/01", "2011/01/25", @DesktopDir & "\Test\") Func Download_Files($StartDate, $EndDate, $DestFolder) For $x = 0 to _DateDiff("D", $StartDate, $EndDate) $WorkDate = _DateAdd("D", $x, $StartDate) $aWorkDate = StringSplit($WorkDate, "/") Switch _DateToDayOfWeek($aWorkDate[1],$aWorkDate[2],$aWorkDate[3]) Case 2 to 6 ; ignore Sunday and Saturday $WorkDate = $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2) $FileName = "eq" & $WorkDate & "_csv.zip" ; InetGet("[url="http://www.bseindia.com/Hisbhav/"]http://www.bseindia.com/Hisbhav/[/url]" & $FileName, $DestFolder & $FileName) EndSwitch Next EndFunc PS - I hate how the new forum software screws around with pasted code removing blank lines and changing the indentation I do know implementing the new version is a lot of work, and am sure eventually all the kinks will be worked out. It is appreciated, Devs/Mods Edit: It appears Saturday and Sunday data do appear on that site, so maybe just: #include <Date.au3> Download_Files("2011/01/01", "2011/01/25", @DesktopDir & "\Test\") Func Download_Files($StartDate, $EndDate, $DestFolder) If Not FileExists($DestFolder) Then DirCreate($DestFolder) For $x = 0 to _DateDiff("D", $StartDate, $EndDate) $aWorkDate = StringSplit(_DateAdd("D", $x, $StartDate), "/") $FileName = "eq" & $aWorkDate[3] & $aWorkDate[2] & StringRight($aWorkDate[1], 2) & "_csv.zip" InetGet("[url="http://www.bseindia.com/Hisbhav/"]http://www.bseindia.com/Hisbhav/[/url]" & $FileName, $DestFolder & $FileName) Next EndFunc Edited September 4, 2011 by Spiff59 Link to comment Share on other sites More sharing options...
Rogue5099 Posted September 4, 2011 Share Posted September 4, 2011 I got bored here ya go: expandcollapse popup#include <GUIConstantsEx.au3> #include <Date.au3> GUICreate("Download Between", 200, 200, 800, 200) GUICtrlCreateLabel("Start Date:", 10, 10) $Date1 = GUICtrlCreateDate("", 10, 40, 185, 20) GUICtrlCreateLabel("End Date:", 10, 70) $Date2 = GUICtrlCreateDate("", 10, 100, 185, 20) $Button = GUICtrlCreateButton("OK", 60, 140, 70) $DTM_SETFORMAT_ = 0x1032 $style = "yyyy/MM/dd" GUICtrlSendMsg($Date1, $DTM_SETFORMAT_, 0, $style) GUICtrlSendMsg($Date2, $DTM_SETFORMAT_, 0, $style) GUISetState() While 1 $msg = GUIGetMsg() Switch $msg Case $GUI_EVENT_CLOSE Exit Case $Button ExitLoop EndSwitch WEnd $Start = GUICtrlRead($Date1) $End = GUICtrlRead($Date2) GUIDelete() If $Start = "" Or $End = "" Then Exit Download_Between($Start, $End, @DesktopDir & "\Test\") Func Download_Between($StartDate, $EndDate, $Destination) DirCreate($Destination) $sDays = _DateDiff("D", $StartDate, $EndDate) ProgressOn("Download", "", "", 0, 0) For $x = 0 to $sDays $dDate = _DateAdd("D", $x, $StartDate) $aDate = StringSplit($dDate, "/") $dDate = $aDate[3] & $aDate[2] & StringRight($aDate[1], 2) $sFile = "eq" & $dDate & "_csv.zip" ProgressSet($x/$sDays*100, "Downloading " & $sFile & @CRLF & Int($x/$sDays*100) & "%", _DateToMonth($aDate[2]) & " " & $aDate[1]) InetGet("http://www.bseindia.com/Hisbhav/" & $sFile, $Destination & $sFile) Next ProgressOff() Return 0 EndFunc My projects: Inventory / Mp3 Inventory, Computer Stats Link to comment Share on other sites More sharing options...
safdar19 Posted September 5, 2011 Author Share Posted September 5, 2011 rogue5099, the last code works perfect !! Great work !! One last request. There is no way to extract the compressed file. So Run() function will have to be applied at the end, to run winrar. Run("D:\Program Files\WinRAR\WinRAR.exe") these extracted file need to be processed in a certain way. As these files are .csv, and not .xlsx , these files can not be processed( only a few colums and rows need to adjusted , this can be easily done) .auto it has functions for exec files and not .csv files. any suggestions? Link to comment Share on other sites More sharing options...
Rogue5099 Posted September 17, 2011 Share Posted September 17, 2011 (edited) Check this out instead of using winrar: Then depending on what data you want depends on the next steps you take. Edited September 17, 2011 by rogue5099 My projects: Inventory / Mp3 Inventory, Computer Stats Link to comment Share on other sites More sharing options...
Juvigy Posted September 19, 2011 Share Posted September 19, 2011 There was a CSV udf so it should be easy. Also you can use the Excel udf - CSV files can be opened by excel. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now