'Html Scraping'

[UnknownWarrior]

Html is too short to search for, so I was kind of forced to ask this question because I couldn't search it .

Is there any way to get HTML code BEFORE client-side processing? Any browser's "View Source" shows only the HTML after the client as processed it. I need it before :/.

Any help is greatly appreciated - I really do not know where to start at.

[Carlo84]

you always get non-edited html with any function. what are you trying to do?

[UnknownWarrior]

you always get non-edited html with any function. what are you trying to do?

I know that, but what I need is the stuff processed BEFORE the client processes. I believe javascript does this - it has an original value before the client processes the script and then that's what you see on your screen (the processed script). I want that value BEFORE my browser processes it.

[KaFu]

You need to tap into the actual data transmission, e.g. if you create an Autoit http proxy and point your browser to it would be another method (though I'm not sure there's a fully working Autoit proxy out there somewhere ). For Firefox you could also use the Greasemonkey plugin.

[Mallie99]

Html is too short to search for, so I was kind of forced to ask this question because I couldn't search it .

Is there any way to get HTML code BEFORE client-side processing? Any browser's "View Source" shows only the HTML after the client as processed it. I need it before :/.

Any help is greatly appreciated - I really do not know where to start at.

If you're talking about server side scripting then it can't be done. That's kind of how it works. When you make a request for data, the server will transmit the contents of the file over the Internet connection. If it comes across any pre-processing tags it will do that processing before transmitting the results.

Client-side action such as Javascript is all performed in the browser before you get to see it on your screen. Your web browser will read the information passed to it and send any tasks to whatever is configured to handle such tasks (such as javascript). By the time you see it in your web browser, it's already been compiled and run for you.

So, in recap, you cannot get PHP or ASP source without downloading the files directly from the host folder using FTP.

[JohnOne]

What about Inet*() functions.

[UnknownWarrior]

You need to tap into the actual data transmission, e.g. if you create an Autoit http proxy and point your browser to it would be another method (though I'm not sure there's a fully working Autoit proxy out there somewhere ). For Firefox you could also use the Greasemonkey plugin.

If you're talking about server side scripting then it can't be done. That's kind of how it works. When you make a request for data, the server will transmit the contents of the file over the Internet connection. If it comes across any pre-processing tags it will do that processing before transmitting the results.

Client-side action such as Javascript is all performed in the browser before you get to see it on your screen. Your web browser will read the information passed to it and send any tasks to whatever is configured to handle such tasks (such as javascript). By the time you see it in your web browser, it's already been compiled and run for you.

So, in recap, you cannot get PHP or ASP source without downloading the files directly from the host folder using FTP.

How does 'FireBug' do it then? (Plugin for Firefox). It allows me to click ANY element on my screen in the FireBug editor and it shows me the element HTML - BEFORE processing.

What about Inet*() functions.

I am aware of this function, but I don't think it gets the source code before client processing, it gets the same results as 'View Source' would.

[JohnOne]

"How does 'FireBug' do it then? (Plugin for Firefox). It allows me to click ANY element on my screen in the FireBug editor and it shows me the element HTML - BEFORE processing."

I dont understand that, firebug needs to be on the webpage, so it has already processed the page, unless I'm missing something.

[UnknownWarrior]

"How does 'FireBug' do it then? (Plugin for Firefox). It allows me to click ANY element on my screen in the FireBug editor and it shows me the element HTML - BEFORE processing."

I dont understand that, firebug needs to be on the webpage, so it has already processed the page, unless I'm missing something.

That's exactly what it does though, it's weird.

You didn't miss anything - Firebug literally tells me the elements HTML BEFORE client processing, but the page is already loaded on my screen. I would assume that it is storing the pre-processed HTML on each webpage loaded when Firebug is activated, so that when you go and use it, it just recalls that code.

[ProgAndy]

If you use "Show Page Source" in Firefox you get the original HTML that was sent to FF without any modifications. FireBug displays the current state of the DOM after the Javascript was executed.

[Tvern]

If you could give an example page and the desired result from that page it might help.

Firebug might do something like the following: (Unfounded speculation ahead!)

It reads any references to files in the header of the page (CSS, JS, Etc), then when you inspect an element in the page it will check if any of the scripts interact with that element and goes from there.

If page does not make a reference to a script in the header, only the results are seen. I believe this comes down to client side only.

Edit:

PS: You can search these forums using google like this

Edited March 22, 2011 by Tvern

[Mallie99]

"How does 'FireBug' do it then? (Plugin for Firefox). It allows me to click ANY element on my screen in the FireBug editor and it shows me the element HTML - BEFORE processing."

I dont understand that, firebug needs to be on the webpage, so it has already processed the page, unless I'm missing something.

This sounds like google chrome's element inspector, all it's doing is breaking down how the browser goes about making a webpage come alive on your screen. If you want to see that then just rip the site or open the webpage source.

You will never see the pre-processing code unless you have the source code. all <?php echo "PHP Elements"; ?> will be stripped and processed as required before transmission.

[DaleHohm]

IE's View Source and INet* both show you the source BEFORE client processing (INet* should be obvious, because there is no client rendering engine involved). The _IE* functions (like _IEDocReadHtml) all work on HTML AFTER client processing. Tools like DebugBar for IE and other DOM inspectors allow you to see HTML either before OR after client processing. Also, don't confuse server-side processing with this discussion... you have no control over that from the client end.

Dale

[UnknownWarrior]

This has nothing at all to do with what I'm doing, but I cannot disclose the exact information that I am working on (client confidentiality).

On this signup page, I found a similar (if not exact) scenario:

http://bigthink.com/signup

You'll notice if you View Source in FF or IE you will see something similar to this code:

(variations in the link probably, but you'll see that line in the source)

<iframe src="http://api.recaptcha.net/noscript?k=6Lc38QQAAAAAAF0mh4k2rLRBKh1L0YNNYN8e0PLe" height="300" width="500" frameborder="0"></iframe><br/>

However, in Firebug, if I use it's 'Trace' feature and click on the image, it returns this to me:

<img width="300" height="57" src="http://www.google.com/recaptcha/api/image?c=03AHJ_VutXlBZb-Mwf2z-KYfvNYUSwGv3R2L0OE0oU2kyMtHQwrB3PTEOgIQV49xOylnlg8c-RWMUZos83pwt31D5SN-2E5JVWVB9uJIDTu-7x_tlptIbSnkpaAgwmyx3ARvPIbdrWK4oWAfgc0d_HWSYfBwiJX90c6g" style="display: block;">

THAT is the code I need... not the javascript stuff in the 'View Source' example.

Any ideas? :/

[Tvern]

<img width="300" height="57" src="http://www.google.com/recaptcha/api/image?c=03AHJ_VutXlBZb-Mwf2z-KYfvNYUSwGv3R2L0OE0oU2kyMtHQwrB3PTEOgIQV49xOylnlg8c-RWMUZos83pwt31D5SN-2E5JVWVB9uJIDTu-7x_tlptIbSnkpaAgwmyx3ARvPIbdrWK4oWAfgc0d_HWSYfBwiJX90c6g" style="display: block;">

THAT is the code I need... not the javascript stuff in the 'View Source' example.

Guess the contents of the javascript if you'd follow the link...

Edit:

havascript. What's that supposed to be.

P.s. If the purpose of this is getting past a CAPTCHA, I wouldn't bother.

Edited March 23, 2011 by Tvern

[UnknownWarrior]

Guess the contents of the javascript if you'd follow the link...

Edit:

havascript. What's that supposed to be.

P.s. If the purpose of this is getting past a CAPTCHA, I wouldn't bother.

What do you mean guess the contents?

And no, captcha has nothing to do with what I'm doing, but it runs on the same kind of system that I am trying to read.

[JohnOne]

You keep banging on about firebug this and that, see if I can put it in simple terms for you.

Firebug is using data from the page it is on, and firefox browser has rendered the page, and the code on it has offered up its secrets.

Firebug is not doing the wizardry you seem to think it is.

What you are seeing is the product of firebug following instructions from code on a page that has been succesfully identified as firefox browser.

It is NOT source before the page is loaded, is is one step after source HAS been loaded.

So what you are asking for makes no sense, you want browser rendered information before the browser renders it, but it must be before the page is loaded? and that wouldnt have the capcha image location anyway even if dark matter had engulfed it..

Whats going on here is you are flogging a dead horse.

[UnknownWarrior]

You keep banging on about firebug this and that, see if I can put it in simple terms for you.

Firebug is using data from the page it is on, and firefox browser has rendered the page, and the code on it has offered up its secrets.

Firebug is not doing the wizardry you seem to think it is.

What you are seeing is the product of firebug following instructions from code on a page that has been succesfully identified as firefox browser.

It is NOT source before the page is loaded, is is one step after source HAS been loaded.

So what you are asking for makes no sense, you want browser rendered information before the browser renders it, but it must be before the page is loaded? and that wouldnt have the capcha image location anyway even if dark matter had engulfed it..

Whats going on here is you are flogging a dead horse.

Notice, I had ASSUMED that Firebug was reading the code before the client processed it... I really don't have a clue HOW Firebug it is, that's why I made this topic.

Whatever it is, I need to know HOW to do it, I don't care how I get there or what comes first or whatever.

[JohnOne]

Whatever it is, I need to know HOW to do it, I don't care how I get there or what comes first or whatever.

INetRead() or INetGet()

[UnknownWarrior]

INetRead() or INetGet()

Doesn't work... it returns the 'api.' code (1st code above), not the 'google.com' code (2nd code above).