pileot Posted March 12, 2008 Share Posted March 12, 2008 (edited) Edit: Code update March 31 Thanks for the help thus far. For those of you just joining, i am trying to get a working script to sequentially find and download MP3's on www.freeotrshows.com the code thus far is: expandcollapse popup#include <INet.au3> $number = inputbox("","","") Dim $WebSite = "http://www.freeotrshows.com" Dim $mainhtmlSource = _INetGetSource($WebSite);main html source, duh Dim $aLinks = _GetLinks($mainhtmlSource, 0);list of all links For $i = 1 To $aLinks[0] _DownloadLinks(_GetLinks(_INetGetSource($aLinks[$i]), 1));lolwhut? Next Func _DownloadLinks($avArray) If Not $avArray[0] Then Return Local $svMP3Link;setting up link for mp3 For $i = 1 To $avArray[0] $svMP3Link = _DownloadM3U(_INetGetSource($avArray[$i])) If @error Then ContinueLoop $oSize = InetGetSize($svMP3Link) $mp3name = stringsplit($svMP3Link,"/");splits link to get the final name of the file $mp3name = $mp3name[$mp3name[0]] $mp3name = stringreplace($mp3name,"-","") $mp3name = stringreplace($mp3name,"_","") $mp3name = stringreplace($mp3name,",","") tooltip($mp3name,0,$number*20) ConsoleWrite(@lf & $mp3name); shows you the links to the mp3's, you could launch your webbrowser to download them, or store them in a file, or w/e if not FileExists(@ScriptDir & "\Radio\" & $mp3name) Then InetGet($svMP3Link, @ScriptDir & "\Radio\" & $mp3name, 1, 1);final one continues before file completely downloaded, 0 waits for download to complete consolewrite(" Downloading...") While @InetGetActive toolTip($mp3name & " " & Round(@InetGetBytesRead/$oSize*100) & "%", 0, $number*20) sleep(100) WEnd consolewrite(" Done!") endif Next EndFunc Func _DownloadM3U($svString) If Not Stringinstr($svString, ".mp3") Then Return(SetError(1, 0, 0)) ;Code to save the M3U if you want here.. Return StringLeft($svString, StringInStr($svString, ".mp3")+3) EndFunc Func _GetLinks($svHtml, $ivMode = 0) If Not $ivMode Then Local $avArray = StringSplit($svHtml, "href=", 1) Else Local $avArray = StringSplit($svHtml, $WebSite, 1) EndIf Local $avRet[1] = [0] Local $svLink For $i = 1 To $avArray[0] If Not $ivMode Then $svLink = $WebSite & StringLeft($avArray[$i], StringInStr($avArray[$i], ">")-1) Else $svLink = StringReplace($WebSite & StringLeft($avArray[$i], StringInStr($avArray[$i], "m3u")+2), "' + '.' + '", ".") EndIf If (Not $ivMode And StringInStr($svLink, ".html")) Or ($ivMode And StringInStr($svLink, ".m3u") And Not StringInStr($svLink, "/.m3u")) Then $avRet[0] += 1 ReDim $avRet[$avRet[0]+1] $avRet[$avRet[0]] = $svLink EndIf Next Return $avRet EndFunc Whats next is to try adding a check for if the file that exists is the correct size. I have no idea how to make that work. If you could assist with that it would be apprieciated. If you make changes to assist, please keep the following in mind I am a novice coder. This being said, i dont mind learning new things, so PLEASE if you add or modify somthing, say what you changed and why, even if it seems obvious. I want to learn, and i cant do that if you dont explain why you did what you did. Please take note of what vars are being used and continue to use them, dont just make a new set of vars because its easier. If i have to try to keep track of 400 different vars it makes it dam near impossable for me to edit / adapt the script to my needs. Thanks for any/all assistance, i really do appreciate it! Edited March 31, 2008 by pileot Link to comment Share on other sites More sharing options...
DjDeep00 Posted March 12, 2008 Share Posted March 12, 2008 (edited) @pileot...Had some free time on my hands...This is the way I would do it... Note: There are many ways I could have cleaned up the script...feel free expandcollapse popup#include <INet.au3> #include <Array.au3> $Website = "http://www.freeotrshows.com/" $Download_Location = @ScriptDir & "\dl\" DirCreate($Download_Location) $Homepage = _INetGetSource($Website) $Main_Array = StringSplit($Homepage, @LF) For $x = 1 To $Main_Array[0] If StringInStr($Main_Array[$x], "<a href=/otr") Then $URL_Split = StringSplit($Main_Array[$x], ">") $Album_URL = StringReplace($URL_Split[1], "<a href=", $Website) $Album_Songs_Split = _INetGetSource($Album_URL) $Song_Array = StringSplit($Album_Songs_Split, @LF) For $z = 1 To $Song_Array[0] If StringInStr($Song_Array[$z], "document.write('http://www.freeotrshows.com/") <> 0 Then $Song_Split = StringSplit(StringStripWS(StringReplace($Song_Array[$z], "document.write('", ""), 8), "+") $Song_URL = StringTrimRight($Song_Split[1], 1) If $Song_Split[0] <> 3 Then $Song_Ext = "m3u" Else $Song_Ext = StringReplace(StringReplace($Song_Split[3], '"' & ">');", ""), "'", "") EndIf $Song_Location = $Song_URL & "." & $Song_Ext If StringRight($Song_Location, 1) <> "/" Then $Song_Name = StringSplit($Song_Location, "/") ConsoleWrite($Song_Location & @CRLF) $oSize = InetGetSize($Song_Location) If Not FileExists(@ScriptDir & "\dl\" & $Song_Name[$Song_Name[0]]) Then InetGet($Song_Location, $Download_Location & $Song_Name[$Song_Name[0]], 1); Don't use 1 as the last parameter (background)...See the helpfile. $Get_MP3_Location=StringStripWS(StringStripCR(FileReadLine($Download_Location & $Song_Name[$Song_Name[0]],1)),8) $MP3_Name=StringSplit($Get_MP3_Location, "/") ;Msgbox(4096,"",$MP3_Name[$MP3_Name[0]]) InetGet($Get_MP3_Location, $Download_Location & $MP3_Name[$MP3_Name[0]], 1) FileDelete($Download_Location & $Song_Name[$Song_Name[0]]) EndIf While @InetGetActive ToolTip(Round(@InetGetBytesRead / $oSize * 100) & "%", 0, 0) Sleep(100) WEnd EndIf EndIf Next EndIf Next Edited March 13, 2008 by DjDeep00 Link to comment Share on other sites More sharing options...
pileot Posted March 12, 2008 Author Share Posted March 12, 2008 (edited) Full edit Ok one thing that would really assist me is if you defined what different vars were supposed to be... InetGet($Song_Location, $Download_Location & $Song_Name[$Song_Name[0]], 1); Don't use 1 as the last parameter (background)...See the helpfile. what command are you talking about? inetget? $Song_Name[0]? did you already change it so it works so its good or do i have to change it? whats the importance of "FileDelete($Download_Location & $Song_Name[$Song_Name[0]])" thanks, it seems to be working as it is now.... next thing to add is a tooltip that will show how much of the file is being downloaded... and work for mp3's as well as m3u's Edited March 13, 2008 by pileot Link to comment Share on other sites More sharing options...
DjDeep00 Posted March 13, 2008 Share Posted March 13, 2008 I Updated my earlier post... Link to comment Share on other sites More sharing options...
pileot Posted March 27, 2008 Author Share Posted March 27, 2008 (edited) removed, code is in first post. Edited March 31, 2008 by pileot Link to comment Share on other sites More sharing options...
FreeFry Posted March 27, 2008 Share Posted March 27, 2008 (edited) Try this:expandcollapse popup#include <INet.au3> Dim $WebSite = "http://www.freeotrshows.com" Dim $mainhtmlSource = _INetGetSource($WebSite) Dim $aLinks = _GetLinks($mainhtmlSource, 0) For $i = 1 To $aLinks[0] _DownloadLinks(_GetLinks(_INetGetSource($aLinks[$i]), 1)) Next Func _DownloadLinks($avArray) If Not $avArray[0] Then Return Local $svMP3Link For $i = 1 To $avArray[0] $svMP3Link = _DownloadM3U(_INetGetSource($avArray[$i])) If @error Then ContinueLoop ;code to download files goes here, I leave this for you ConsoleWrite($svMP3Link & @LF) ; shows you the links to the mp3's, you could launch your webbrowser to download them, or store them in a file, or w/e Next EndFunc Func _DownloadM3U($svString) If Not Stringinstr($svString, ".mp3") Then Return(SetError(1, 0, 0)) ;Code to save the M3U if you want here.. Return StringLeft($svString, StringInStr($svString, ".mp3")+3) EndFunc Func _GetLinks($svHtml, $ivMode = 0) If Not $ivMode Then Local $avArray = StringSplit($svHtml, "href=", 1) Else Local $avArray = StringSplit($svHtml, "http://www.freeotrshows.com/", 1) EndIf Local $avRet[1] = [0] Local $svLink For $i = 1 To $avArray[0] If Not $ivMode Then $svLink = $WebSite & StringLeft($avArray[$i], StringInStr($avArray[$i], ">")-1) Else $svLink = StringReplace("http://www.freeotrshows.com/" & StringLeft($avArray[$i], StringInStr($avArray[$i], "m3u")+2), "' + '.' + '", ".") EndIf If (Not $ivMode And StringInStr($svLink, ".html")) Or ($ivMode And StringInStr($svLink, ".m3u") And Not StringInStr($svLink, "/.m3u")) Then $avRet[0] += 1 ReDim $avRet[$avRet[0]+1] $avRet[$avRet[0]] = $svLink EndIf Next Return $avRet EndFuncThat should grab the links for the mp3 files from the site... Sorry but I just can't get my head around StringRegExp I leave the downloading of the MP3's too you(I would've preferred doing it with my webbrowser, as it's harder for autoit to download several files at once..).You should beware that this will probably put an enormous load on the webserver though...Edit:Got rid of the _GetM3ULinks function, and combined it in the _GetLinks function instead.. Edited March 27, 2008 by FreeFry Link to comment Share on other sites More sharing options...
pileot Posted March 28, 2008 Author Share Posted March 28, 2008 (edited) Ya, last time i tried downloading from them i killed the server for like, 48 hours... got overzealous with my school's connection downloading at 3 mbps lol. My intention for having it check for files already there was so i could have multiple instances of this program running at once, and each one downloading a single file at a time... this would allow me to control the bandwith in/out easier. I will try messing with the script for a while, and let you all know if i require more assistance. Thanks for the help! i really apprieciate it! *Edit* Ok theres a few things im noticing right off... $website gets dimmed right off at the start, and its only used a few places? other places where it could be used its not... This is one of the things that continues to confuse me, prolly why having multiple coders on the same script is a bad idea... because things can get messy really fast. For $i = 1 To $aLinks[0] _DownloadLinks(_GetLinks(_INetGetSource($aLinks[$i]), 1)) Next what does this do? oh wait, im assuming _downloadlinks is a function? why is there the _ infront of it? im confused about this whole bit... Local $svMP3Link What is this being done for? When dimming vars or whatever (ya i dont totally understand the difference between dim, global, and local, and the help file was rather technical, i didnt understand it much) it would be nice if it was explained what that var will be used for. In this case im assuming its the link to the specific MP3 thats being downloaded? If @error Then ContinueLoop Ok, i understand that @error gets created (or whatever) if theres problems with most of the functions... and its a good way to check if what you jsut did worked... But what is continueloop? wouldnt you want to stop the loop if theres an error? or go past that part and go on to the next part? what if later on you need to rely on a value that would have depended on that function working? is this really what you want to do here? Also the whole last function doesnt make a lot of sense to me, could you please explain whats going on there? what does each part do? the reason i want this is because theres other sites that i will eventually move on to once i have sucked this one dry, and i want to be able to change the program with minimal difficulty, as well as understand how to make this work on my own. Edited March 28, 2008 by pileot Link to comment Share on other sites More sharing options...
FreeFry Posted March 28, 2008 Share Posted March 28, 2008 (edited) Ya, last time i tried downloading from them i killed the server for like, 48 hours... got overzealous with my school\'s connection downloading at 3 mbps lol. My intention for having it check for files already there was so i could have multiple instances of this program running at once, and each one downloading a single file at a time... this would allow me to control the bandwith in/out easier. I will try messing with the script for a while, and let you all know if i require more assistance. Thanks for the help! i really apprieciate it! *Edit* Ok theres a few things im noticing right off... $website gets dimmed right off at the start, and its only used a few places? other places where it could be used its not... This is one of the things that continues to confuse me, prolly why having multiple coders on the same script is a bad idea... because things can get messy really fast.My mistake, it was late at night, and i just whipped the script together without looking over it.., I\'m posting a edited version below. For $i = 1 To $aLinks[0] _DownloadLinks(_GetLinks(_INetGetSource($aLinks[$i]), 1)) Next what does this do? oh wait, im assuming _downloadlinks is a function? why is there the _ infront of it? im confused about this whole bit... This runs the _DownloadLinks function with the return value of the _GetLinks function(which gets html source code from _INetGetSource). The reason for the _ is because I like this naming convention for UDF\'s(User Defined Functions). Local $svMP3Link What is this being done for? When dimming vars or whatever (ya i dont totally understand the difference between dim, global, and local, and the help file was rather technical, i didnt understand it much) it would be nice if it was explained what that var will be used for. In this case im assuming its the link to the specific MP3 thats being downloaded?It creates a local variable called svMP3Link, which I store the link to the MP3 file I read from the M3U file in the _DownloadM3U function. Local means it can only be accessed inside the function it\'s declared in(meaning when the _DownloadLinks function exits, the value is destroyed, and the memory it uses is released). Dim is more like the automated version of Local, and Global, it creates a variable/array in the scope it\'s currently in(If it\'s inside a function, it\'s created Locally, if outside a function, it\'s created Globally. Global creates the variable/array Globally, no matter what(even if it\'s created inside a function). If @error Then ContinueLoop Ok, i understand that @error gets created (or whatever) if theres problems with most of the functions... and its a good way to check if what you jsut did worked... But what is continueloop? wouldnt you want to stop the loop if theres an error? or go past that part and go on to the next part? what if later on you need to rely on a value that would have depended on that function working? is this really what you want to do here?@error is a macro, it\'s normally set to 0(meaning that the last function ran didn\'t produce an error), otherwise(if an error occured in the last function), it\'s set to a positive value.. Basically what \"If @error Then ContinueLoop\" does is that it checks if the _DownloadM3U function failed, and if it did, it Continues the loop(ie. it\'s continues on the next iteration in the loop) Inside the _DownloadM3U function I do a simple check if the M3U file contains .mp3, if it doesn\'t it sets @error to 1 with the SetError function, and returns 0(also done in the SetError (also done with SetError). Reason for this is because I noticed that some links on the page doesn\'t exist, and you get a error page, which would cripple this function if I wouldn\'t do error checking. Also the whole last function doesnt make a lot of sense to me, could you please explain whats going on there? what does each part do? the reason i want this is because theres other sites that i will eventually move on to once i have sucked this one dry, and i want to be able to change the program with minimal difficulty, as well as understand how to make this work on my own. The _GetLinks function? I Send it the html code of the page(retrieved with _INetGetSource), which contains all the links, etc. Then I split the html code on each place where \"href=\" exists with StringSplit, (href= is part of the html code for a link), and then I loop through the array StringSplit returns, and diggs out all the links for the page(storing them in another array). I also do error checking here to determine if the link actually links to a page that has .html in it(which all the music links has on this page) so I don\'t get wrong links(as there\'s some other links leading to non-music pages). Then when I\'ve looped through the array I return the array, which I later use in some other places in the script. The array contains all the links. The _GetLinks function I guess it quite unique for this site, it would probably need some modification to use it on other sites(as the links on this site was actually quite easy to aquire).. Edit: whoops, forgot to include the edited code: expandcollapse popup#include <INet.au3> Dim $WebSite = "http://www.freeotrshows.com" Dim $mainhtmlSource = _INetGetSource($WebSite) Dim $aLinks = _GetLinks($mainhtmlSource, 0) For $i = 1 To $aLinks[0] _DownloadLinks(_GetLinks(_INetGetSource($aLinks[$i]), 1)) Next Func _DownloadLinks($avArray) If Not $avArray[0] Then Return Local $svMP3Link For $i = 1 To $avArray[0] $svMP3Link = _DownloadM3U(_INetGetSource($avArray[$i])) If @error Then ContinueLoop ;code to download files goes here, I leave this for you ConsoleWrite($svMP3Link & @LF) ; shows you the links to the mp3's, you could launch your webbrowser to download them, or store them in a file, or w/e Next EndFunc Func _DownloadM3U($svString) If Not Stringinstr($svString, ".mp3") Then Return(SetError(1, 0, 0)) ;Code to save the M3U if you want here.. Return StringLeft($svString, StringInStr($svString, ".mp3")+3) EndFunc Func _GetLinks($svHtml, $ivMode = 0) If Not $ivMode Then Local $avArray = StringSplit($svHtml, "href=", 1) Else Local $avArray = StringSplit($svHtml, $WebSite & "/", 1) EndIf Local $avRet[1] = [0] Local $svLink For $i = 1 To $avArray[0] If Not $ivMode Then $svLink = $WebSite & StringLeft($avArray[$i], StringInStr($avArray[$i], ">")-1) Else $svLink = StringReplace($WebSite & "/" & StringLeft($avArray[$i], StringInStr($avArray[$i], "m3u")+2), "' + '.' + '", ".") EndIf If (Not $ivMode And StringInStr($svLink, ".html")) Or ($ivMode And StringInStr($svLink, ".m3u") And Not StringInStr($svLink, "/.m3u")) Then $avRet[0] += 1 ReDim $avRet[$avRet[0]+1] $avRet[$avRet[0]] = $svLink EndIf Next Return $avRet EndFunc I haven't added anything, just changed to utilize the $WebSite variable better. On a sidenote, I would advise you to use the webbrowser to download the files instead of using autoit... as autoit can\'t download more than one file at once(without some workarounds).. Edit: For some reason the forum added a lot of backslashes into my code, fixed it... hopefully it won't happen again.. Edited April 1, 2008 by FreeFry Link to comment Share on other sites More sharing options...
pileot Posted March 31, 2008 Author Share Posted March 31, 2008 (edited) Edited code in first post to try keeping this thread somewhat clean That last function you have there has a massive stringreplace function in it... i know how to use stringreplace in the most basic sense, but what you have there is relitively extre,e, could you explain to me how it works so i can use that function more effeceintly? Also if you look at my code above, its working fine, i got the tooltip meter checking how far along the file is... One thing that would really be nice is a check if the file that exists is the proper size, in case the download was interrupted and i only have part of the file there... Im not sure how to do that. As to the one file at a time, thats why i have the check for the file existing, i can run more than one instance at once, each will download a seperate file. Other than that the downloader is working pretty well, the check for existing files is good, it supports multiple instances, its pretty awesome. Thanks again for all the support! Edited March 31, 2008 by pileot Link to comment Share on other sites More sharing options...
FreeFry Posted April 1, 2008 Share Posted April 1, 2008 (edited) You should refrain from editing the first post really, others who might read the topic wont understand the process of 'development' if you do that.Also when you refer to code somewhere you should be more precise than just saying something as "the last function", even if it's obvious what you mean in this case.It will help others to understand you, and help them helping you. About the way I use the StringReplace function, I don't think there's anything special about it, maybe just a little complicated to an untrained eye:What I do is the same as doing this(but with less code, and no need for a temporary variable):; Here we retrive the part that contains the url to the m3u file.. $tmp = StringLeft($avArray[$i], StringInStr($avArray[$i], "m3u")+2) ; problem is that the site uses javascript to add the .m3u part to the file ;(probably for obfuscating it, making it harder to "rip" from it automatically), which we get rid of here $svLink = StringReplace($tmp, "' + '.' + '", ".") ;and here we add the website address to the url(normally the links in the page has the format "/dir/dir/file.ext", while it has to be "http://www.domain.ext/dir/dir/file.ext" $svLink = $WebSite & "/" & $svLinkoÝ÷ Øî²×h¶z'zX§z)ìµæjwpØhºÛazë¡ö¯j¸nWºÚ"µÍÌÍÜÝ[ÈHÝ[ÔXÙJ ÌÍÕÙXÚ]H [È ][ÝËÉ][ÝÈ [ÈÝ[ÓY ÌÍØ]^VÉÌÍÚWKÝ[Ò[Ý ÌÍØ]^VÉÌÍÚWK ][ÝÛLÝI][ÝÊJÌK ][ÝÉÌÎNÈ È ÌÎNËÌÎNÈ È ÌÎNÉ][ÝË ][ÝË][ÝÊI like to keep it short, even if it can be a "debugging nightmare" later on, as finding an error in a code like that can be hell. Edited April 1, 2008 by FreeFry Link to comment Share on other sites More sharing options...
pileot Posted April 1, 2008 Author Share Posted April 1, 2008 New problem! expandcollapse popup#include <INet.au3> $number = inputbox("","","") Dim $WebSite = "http://www.freeotrshows.com" Dim $mainhtmlSource = _INetGetSource($WebSite);main html source, duh Dim $aLinks = _GetLinks($mainhtmlSource, 0);list of all links For $i = 1 To $aLinks[0] _DownloadLinks(_GetLinks(_INetGetSource($aLinks[$i]), 1));lolwhut? Next Func _DownloadLinks($avArray) If Not $avArray[0] Then Return Local $svMP3Link;setting up link for mp3 For $i = 1 To $avArray[0] $svMP3Link = _DownloadM3U(_INetGetSource($avArray[$i])) If @error Then ContinueLoop $oSize = InetGetSize($svMP3Link) $mp3name = stringsplit($svMP3Link,"/");splits link to get the final name of the file $mp3name = $mp3name[$mp3name[0]] $mp3name = stringreplace($mp3name,"-","");remove invalid chars $mp3name = stringreplace($mp3name,"_","") $mp3name = stringreplace($mp3name,",","") tooltip($mp3name,0,$number*20) ConsoleWrite(@lf & $mp3name); shows you the links to the mp3's, you could launch your webbrowser to download them, or store them in a file, or w/e if not FileExists(@ScriptDir & "\Radio\" & $mp3name) or FileGetSize(@ScriptDir & "\Radio\" & $mp3name) < $oSize then InetGet($svMP3Link, @ScriptDir & "\Radio\" & $mp3name, 1, 1);final one continues before file completely downloaded, 0 waits for download to complete consolewrite(" Downloading...") While @InetGetActive toolTip($mp3name & " " & Round(@InetGetBytesRead/$oSize*100) & "%", 0, $number*20) sleep(100) WEnd consolewrite(" Done!") endif Next EndFunc Func _DownloadM3U($svString) If Not Stringinstr($svString, ".mp3") Then Return(SetError(1, 0, 0)) ;Code to save the M3U if you want here.. Return StringLeft($svString, StringInStr($svString, ".mp3")+3) EndFunc Func _GetLinks($svHtml, $ivMode = 0) ;this section still needs comments explaining whats going on If Not $ivMode Then Local $avArray = StringSplit($svHtml, "href=", 1) Else Local $avArray = StringSplit($svHtml, $WebSite, 1) EndIf Local $avRet[1] = [0] Local $svLink For $i = 1 To $avArray[0] If Not $ivMode Then $svLink = $WebSite & StringLeft($avArray[$i], StringInStr($avArray[$i], ">")-1) Else $svLink = StringReplace($WebSite & StringLeft($avArray[$i], StringInStr($avArray[$i], "m3u")+2), "' + '.' + '", ".") EndIf If (Not $ivMode And StringInStr($svLink, ".html")) Or ($ivMode And StringInStr($svLink, ".m3u") And Not StringInStr($svLink, "/.m3u")) Then $avRet[0] += 1 ReDim $avRet[$avRet[0]+1] $avRet[$avRet[0]] = $svLink EndIf Next Return $avRet EndFunc The code above works wonderfully, however theres a few issues. The main issue i find is when i try running multiple instances and because its detecting incomplete files each instance of the downloader is downloading, and so i get 3 different programs downloading the same file. How can i make it so that each instance of the program will know that file is currently being downloaded, and pass it by and go to the next one? Link to comment Share on other sites More sharing options...
FreeFry Posted April 1, 2008 Share Posted April 1, 2008 (edited) I would just use FileExists and ContinueLoop(if FileExist(...) then ContinueLoop. Alternatively, Launch another script to download a file(that way you have the main script controlling the other scripts, sorta like master and slaves. ) Edited April 1, 2008 by FreeFry Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now