download files from website without having their url

[finet]

Is there a way to download all files from a website directory?

E.g

http://www.site.com/docs/1.htm

http://www.site.com/docs/2.htm

http://www.site.com/docs/3.htm

http://www.site.com/docs/x.htm

I would like to download all the files in http://www.site.com/docs/ without knowing how many files there are and without knowing their name.

(I have a file with URLS in it. Via _FileReadToArray and InetGet I can download them all. The problem with that is that I have to know the URL of each file)

Thanks,

Dirk

[SmOke_N]

Is there a way to download all files from a website directory?

E.g

http://www.site.com/docs/1.htm

http://www.site.com/docs/2.htm

http://www.site.com/docs/3.htm

http://www.site.com/docs/x.htm

I would like to download all the files in http://www.site.com/docs/ without knowing how many files there are and without knowing their name.

(I have a file with URLS in it. Via _FileReadToArray and InetGet I can download them all. The problem with that is that I have to know the URL of each file)

Thanks,

Dirk

If you made them, wouldn't you know them?

Edit:

On another note, I do believe someone did this in FTP.

http://www.autoitscript.com/forum/index.ph...st&p=124032

Edited October 6, 2006 by SmOke_N

[finet]

If you made them, wouldn't you know them?

Edit:

On another note, I do believe someone did this in FTP.

http://www.autoitscript.com/forum/index.ph...st&p=124032

==================================================

The problem is making such an url file and keeping it uptodate.

Thanks for your idea on ftp, but I am no programmer, new to AutoIt and don't know anything of ftp.

It would be nice to have something like

InetGet("http://www.site.com/docs/*.*",...

[big_daddy]

If you enter "http://www.site.com/docs/" in a web browser does it give you a list of the files, or does it redirect you to the index page?

Edited October 6, 2006 by big_daddy

[finet]

If you enter "http://www.site.com/docs/" in a web browser does it give you a list of the files, or does it redirect you to the index page?

Neither of both.

It gives: Page not found. Error 404

[big_daddy]

Neither of both.

It gives: Page not found. Error 404

The only other option I can think of is using a loop, but this will only work if the files are as your example above.

$sURL = "http://www.site.com/docs/"
For $i = 1 To 10
    InetGet($sURL & $i & ".htm", @ScriptDir & "\" & $i & ".htm")
    If @error Then ExitLoop
Next

[MrBeatnik]

It seems it would be difficult, considering you can't get a listing of contents. However, if the links to each file is included *somewhere* on the website, then it can be done I guess...

1) Direct your script to go to the search webpage for this site (for example).

2) Get your script to perform a search on this page, to return all results with the required name.

3) Loop through each return for URL including the name.

It really depends on how the server has been set up to deliver content.

[finet]

The only other option I can think of is using a loop, but this will only work if the files are as your example above.

$sURL = "http://www.site.com/docs/"
For $i = 1 To 10
    InetGet($sURL & $i & ".htm", @ScriptDir & "\" & $i & ".htm")
    If @error Then ExitLoop
Next

Thanks for your fast reply! But... I get one doc in the script directory: "1.htm" or "1.html"

BTW the real adress is where all the files are listed in a webpage is (Dutch language site)

http://www2.vlaanderen.be/ned/sites/ruimte...besluiten2.html

The files to download are

ambtenarenmb.html

ambtenarennamen.html

gemachtigdeambtenaar.html

kbgewestplan.html

kleinewerken.html

merplicht.html

normatieve.html

voetgangersverkeer.html

etcetera....

Greetings,

Dirk

[PsaltyDS]

Thanks for your fast reply! But... I get one doc in the script directory: "1.htm" or "1.html"

BTW the real adress is where all the files are listed in a webpage is (Dutch language site)

http://www2.vlaanderen.be/ned/sites/ruimte...besluiten2.html

The files to download are

ambtenarenmb.html

ambtenarennamen.html

gemachtigdeambtenaar.html

kbgewestplan.html

kleinewerken.html

merplicht.html

normatieve.html

voetgangersverkeer.html

etcetera....

Greetings,

Dirk

The administrator of the website can set Apache (or whatever web server service is used) to allow browsing the files. It appears the admin chose not to do so in this case. If you'd like that changed, contact the admin.

[DaleHohm]

The administrator of the website can set Apache (or whatever web server service is used) to allow browsing the files. It appears the admin chose not to do so in this case. If you'd like that changed, contact the admin.

This has nothing to do with AutoIt, but if you want to retrieve the files, check out FlashGet http://www.flashget.com

It is a very nice, free download manager.

Specifically what you want is the "Site Explorer" in the Tools menu.

Trust me, check it out.

Dale

[PsaltyDS]

This has nothing to do with AutoIt, but if you want to retrieve the files, check out FlashGet http://www.flashget.com

It is a very nice, free download manager.

Specifically what you want is the "Site Explorer" in the Tools menu.

Trust me, check it out.

Dale

That link is a 404 for me. Perhaps you meant FlashGot for FireFox?

Still, that's not a security hack. If the web site admin doesn't allow it, I don't think you can list directory files. FlashGot follows the published links on the page(s), it doesn't search directories and list them if the admin didn't permit that already.

Edit: Aha, found reference to FlashGet in Wikipedia.

Edited October 6, 2006 by PsaltyDS

[DaleHohm]

That link is a 404 for me. Perhaps you meant FlashGot for FireFox?

Still, that's not a security hack. If the web site admin doesn't allow it, I don't think you can list directory files. FlashGot follows the published links on the page(s), it doesn't search directories and list them if the admin didn't permit that already.

Edit: Aha, found reference to FlashGet in Wikipedia.

The link works fine for me. Correct - it is not bypassing the security, but they are using some sort of crawling technique to find what is on the site and in the folder.

And as I said, FlashGet, not FlashGot.

Dale

[finet]

Thank you all for your kind assistance and advice!

Dirk