Sign in to follow this  
Followers 0
Graywalker

Read Binary index.dat file

4 posts in this topic

#1 ·  Posted (edited)

The goal here is to get the Internet History for endpoints and users in the enterprise to find what business use websites use Java.

When opening index.dat in Notepad, I can see all the data I need. Unfortunatly, trying to read that data in AutoIT is proving diffucult.

I've tried :

Func _ParseIndexdat($indexdatpath)
; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($Bindexdat)
$strIndexdat = BinaryToString($indexdat,2)
MsgBox(0,"String from Binary",$strIndexdat)
$FileArray = StringSplit($strIndexdat,@CRLF)
;This may get complex...
Dim $r = 1 ; to count the records
Dim $e = 0; to count the entries

; Start reading from line 1
For $line In $FileArray
$content = StringStripWS($line,7)
  Select
   Case StringInStr($line,"REDR")
    ; this is the start of a record
    MsgBox(0,"REDR",$content)
   Case StringInStr($line,"URL")
    ; this may be the start of a record
    MsgBox(0,"URL",$content)
   Case StringInStr($line,"LEAK")
    ; this is the start of an error record
    ; I will likely ignore it.
    MsgBox(0,"LEAK",$content)
   Case StringInStr($line,"http://")
    ; this is the line with a couple entries
    MsgBox(0,"http",$content)
   Case StringInStr($line,"Content-Type:")
    ; this is an entry I want
    MsgBox(0,"Content-Type",$content)
   Case StringInStr($line,"X-Powered-By:")
    ; this is an entry I want
    MsgBox(0,"Powered-By",$content)
   Case StringInStr($line,"~U:")
    ; this is an entry I want and it marks the end of a record
    MsgBox(0,"~U",$content)
   Case Else
    ; do nothing with the line
  EndSelect
Next
EndFunc

That doesn't get the info... That code doesn't return anything.

Using

$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($Bindexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$strIndexdat)

the "string from binary" msg box shows a LOT of data... but URL and http case are both blank or gibberish. BinaryToString($indexdat,2) and ,3 return Nothing in the cases - 4 does the same as 1.

Using :

$Bindexdat = FileOpen($indexdatpath, 16)
$indexdat = FileRead($Bindexdat);$indexdatpath)
MsgBox(0,"Index dat",$indexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat,@CRLF)

Shows that $indexdat and $strIndexdat are effectively the same.

Using :

; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
;$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($indexdatpath);$Bindexdat)
;$strIndexdat = BinaryToString($indexdat,4)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($indexdat,@CRLF)

It reads a lot more info... Content-Type is fine. ~U: is fine. URL is mostly blank.

Case http:// it will pop up the msg box, but $content is blank. So the data is there... I just can't figure out how to get it into a string.

Using :

$Bindexdat = FileOpen($indexdatpath,256)
$indexdat = FileRead($Bindexdat);$indexdatpath)
MsgBox(0,"Index dat",$indexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat,@CRLF)

Returns data, but URL and Http:// pop up as blank...

$Bindexdat = FileOpen($indexdatpath,48) - URL and Http are blank.

I've found a VBScript that is supposed to read the files... so far no luck on Win7 - it can't find index.dat file... lol! On remote XP machines, it can't find a history folder. So I don't know if it DOES read index.dat files.

I've attached it.

Any ideas or code help is greatly appreciated!!

IE_Network - Copy.txt

Edited by Graywalker

Share this post


Link to post
Share on other sites



#3 ·  Posted (edited)

I altered the vbscript to point directly to the index.dat file and it gets info, but returns all kinds of gibberish.

... that totally messes up the reply even when pasted as code!

So, I've attached it as a screen cap.

post-38206-0-27179900-1327335952_thumb.p

Edited by Graywalker

Share this post


Link to post
Share on other sites

Okay, I've found out that the data IS there. I've just got to find a way to get TO it.

Func _ParseIndexdat($indexdatpath)
; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
$Bindexdat = FileOpen($indexdatpath, 16)
$indexdat = FileRead($Bindexdat);$indexdatpath)
;MsgBox(0, "Index dat", $indexdat)
$strIndexdat = BinaryToString($indexdat, 1)
$strIndexdat = StringStripWS($strIndexdat, 7)
;MsgBox(0, "String from Binary", $strIndexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat, @CRLF)
;This may get complex...
Dim $r = 1 ; to count the records
Dim $e = 0; to count the entries
; Start reading from line 1
For $line In $FileArray
  $line = StringReplace($line, @CRLF, "")
  $line = StringReplace($line, @CR, "")
  $line = StringReplace($line, @LF, "")
  Select
   Case StringInStr($line, "http://")
    ; this is the line with a couple entries
    ;MsgBox(0, "http", $line)
    ; Check to see if URL is in the line
    If StringInStr($line, "URL") Then
     ; it is a REDR or LEAK, trim to URL
     $urlpos = StringInStr($line, "URL")
     $line = StringTrimLeft($line, $urlpos)
    EndIf
    $httppos = StringInStr($line, "http")
    $line = StringTrimLeft($line, $httppos - 1)
    $dotpos = StringInStr($line, ".", "", 3)
    If $dotpos > 10 Then
     $content = $line
     $linelen = StringLen($line)
     $trimfromright = $linelen - ($dotpos + 3)
     $line = StringTrimRight($line, $trimfromright)
     ;MsgBox(0,"Trim", "string lenght : " & $linelen & @CRLF & "dot position: " & $dotpos & @CRLF & "Trim from Right : " & _
     ;$trimfromright & @CRLF & $line)
    Else
     ; nothing
    EndIf
    $record = $line & ","
   Case StringInStr($line, "Content-Type:")
    ; this is an entry I want
    $line = StringStripWS($line, 7)
    $record = $record & $line & ","
   Case StringInStr($line, "X-Powered-By:")
    ; this is an entry I want
    $line = StringStripWS($line, 7)
    $record = $record & $line & ","
   Case StringInStr($line, "~U:")
    ; this is an entry I want and it marks the end of a record
    $line = StringReplace($line, "~U:", "")
    $line = StringStripWS($line, 7)
    $record = $record & $line
    FileWriteLine($logfile, $record)
    $record = ""
   Case Else
    ; do nothing with the line
  EndSelect
Next
EndFunc   ;==>_ParseIndexdat

I am getting fairly spotty results - leaving in lots of random characters before and after the URLs.

Could anyone come up with a way to use StringRegExp to pull the URLs - all of them - from a line?

I've tried several from : http://regexlib.com/Search.aspx?k=URL&AspxAutoDetectCookieSupport=1

with no consistent luck.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • TheSaint
      By TheSaint
      An adaption of an adaption.
      A good while back, I created a program, KindEbook Wishlist, that I use most days and is still available here at the forum. It works well, keeping tabs on price changes for Kindle ebooks at Amazon ... at least for my modest wants.
      About a year or so later, I struck upon the idea of adapting that program for CDs, DVDs, Blu-rays etc from the JB Hifi store, where there was a limit of about 50 items on the in-store wishlist. JB Wishlist is kind of a niche program, really only suitable for those from AUS, at the AutoIt Forum, who would want such a thing ... so not many I imagine, so it has never been available here. However, if you meet that criteria etc, and are willing to register at the AutoIt4Life Clubrooms, then it is available in the Project Chat section there ... though not the more recent updates (you need to ask me about them). That too, works quite well for my modest needs, and even includes an additional Preview & Summary window, which KindEbook Wishlist doesn't have ... though both programs do have access to a Details window per item, that is like the Preview window ... just doesn't constantly display alongside the Main program window (see later for an example with IonGoG Wishlist).
      So, now we come to IonGoG Wishlist, which is an adaption of the JB Wishlist program, and used for keeping tabs on GOG Games ... prices, history of changes, bought list, etc.
      IonGoG Wishlist is an incomplete adaption, so still in beta ... though all the really important elements work well enough. I have been working on it in dribs and drabs, and I now think it has come to a reasonable enough stage to share it ... just be advised, that some of the features accessed via the right-click menu etc, may not work or give strange results. Any of the options that I have given an Accelerator key facility to, should be working fine ... and most of the others are too ... and I am not even sure at this point, of what remains to be adapted ... not much I imagine.
      The name by the way, in case you were wondering, is a phonetic mangling on the words - Keep an eye on your GOG wishlist.
      Here is a screenshot, with the Preview window on left. The Preview window can be placed at right instead or even turned off.

      Download, includes source files.
      IonGoG Wishlist v0.0_b8.zip
      Enjoy!
      ADDENDUM
      In reality, compared to most stores I have come across, the GOG store is great, well setup and laid out. and for most things, including their Wishlist, is more than adequate. So I rarely do a full list Price Query ... so much quicker and easier to just check the two pages of my in-store wishlist. When I first started work on IonGoG Wishlist, the in-store wishlist wasn't as good as it is now.
      All that said, I still find it handy to use IonGoG Wishlist, for a variety of things - Offline browsing, Cover images to use with my bought & downloaded game folders, price changes history (and patterns of GOG behavior for sales etc), a bought list (with price I paid and date etc), Game Notes & Warnings, etc. And while not as quick as just checking the pages of your in-store wishlist, you can just set the full, favorites or non-favorites Query going, while you go away and do something else, and then later come back and look at the highlighted changes that may have occurred.
    • TheSaint
      By TheSaint
      Formerly known as KindEbook Price Query.
      Please read IMPORTANT ADVICE here at Post #57.  (9th May 2016)
      Reading some sections of the previous topic could be helpful. There is also a Disclaimer reference etc.
      OLDER DOWNLOADS
      KindEbook Wishlist v5.4.zip  (new) 
      BONUS - AZWPlug v2.8.zip (new)
      BONUS - Add Book & All Formats To Calibre v1.4.zip (new)
      See Post #51 for information about the Bonus program (AZWPlug).
      Screenshots in second post.
      (11-11-2017) KindEbook Wishlist updated to v5.4
      Bugfix for '/gp/' and/or '/product/' in ebook URL. Holding down CTRL when the program starts will prevent the last user list from loading (if you are quick enough). Minor improvements to COPY and MOVE options. Accelerator Keys for COPY and MOVE now changed to (Shift-C) and (Shift-M) from (Ctrl-C) and (Ctrl-M) respectively, to avoid possibility of hotkey conflicts. Accelerator Keys added for Shared Comments (Shift-S) and Private Comments (Shift-P).
      AZWPlug updated to v2.8
      Database list can now be set not to display when starting that window, plus a LOAD button added to load that list when desired. CTRL used with the open Kindle Content folder button, will now close the program. If CTRL used when selecting the MODE on the Results window, it will just save the setting without making changes to existing results. Option added for slicker loading of list, plus alternate list lines are now pale pink. Two find options added for the Author and Title, with FIND jumping to each entry found and SHOW just displaying all found.
      NOTE - With LOAD deselected, the list will only show the added ebook title (if any).
      Add Book & All Formats To Calibre v1.4     (NOTE - This has only been tested with calibre v2.x.x, not the newer v3.x.x.)
      Another bonus complimentary program added to the mix, that has a dropbox for adding an ebook and all associated ebook formats to calibre, using the OPF metadata file. See Instructions & Screenshots here.
      (07-10-2017) KindEbook Wishlist updated to v5.3.
      Bugfix for 'Move To Another User' and 'Relocate All Bought Ebooks', when using right-click menu options with the new 'Slicker visuals for low powered PC' option enabled. The bug meant you ended up with a blank list (requiring a sort or load to fix). Added three more Accelerator Keys - Selected Entry (text) Detail (Ctrl-T), Set a SWEET price (Ctrl-S), Set a Warning (Ctrl-W).
      ----------------------------------------------------------------------------------------------------
      KindEbook Wishlist - Most controls get disabled now during a Query, plus the same occurs during an 'Exchange Rate Query'. Plus if 'Please Wait' splash is disabled for Query, then a splash is no longer shown during an 'Exchange Rate Query', with that text shown instead in both the 'Please Wait' and 'Timer' labels, which are temporarily unhidden for that purpose. The Timer label can also be reshown if hidden by the new feature, by clicking the FIND label.
      Mouse cursor now changes during Query and Ebook list loading etc, to indicate when busy working. Extended the 'Please Wait' label to loading & sorting the Ebooks list, which also applies to any other process that re-populates or changes number of list entries (ebook removal, changing user, relocating bought, etc). Those are all dependent on the "Use a flashing 'Please Wait' ..." setting being set. NOTE - This feature was added to this and the previous version, so that having a 'Please Wait' splash displayed on top of all windows, for a lengthy period perhaps, could be avoided, if desired.
      Previous Versions (newer)
      Previous Versions  (older)
      The KindEbook Wishlist, is basically a compliment to your regular store wishlist, but has a history element and allows you to check current and previous prices etc in a better, quicker and more organized fashion. Essentially it assists and hopefully improves with decision making, when it comes to determining whether to make a purchase yet or not.
      You could compare it to manually and painfully creating something like an Excel spreadsheet for the same purpose, without the time, complexity and pain involved.
      You can sort by Title, Author, Current Price, Lowest Price, Add Order, Favorites, etc.
      You can view details - Book Description, Price Changes, etc.
      You can open the ebook URL in your web browser, where you can elect to make the purchase.
      The program supports multiple users, shared titles, shared comments and private comments, etc.
      After purchase, an ebook can be relocated to a bought list (per user).
      Price can be queried on an individual title basis or ALL titles (starting at first or selected or only favorites or not favorites).
      Dates are recorded for most processes, with various reporting options.
      I call it a wishlist on steroids.
       
    • Dragonfighter
      By Dragonfighter
      I'm searching a way to do xor and shift and if possible also other operations. Thanks in advance for the replies.
    • rudi
      By rudi
      Hello.
      I'm too stupid to see my mistake:
      To investigate the internal "dictionary" of TIFF files I'd like to read in the files in binary mode and to check, if there are more than one pages "in" this TIFF.
      Notepad++, "View as Hex" is presenting the first bytes as "49 49 2a 20 08 20 20 20 12" for the TIF attached to this posting
      The "TIFF Header Format" is easy:
      Offset 00h, 2 Byte = Byte Order, "II"=intel, "MM"=motorola. (I = 0x49)
      --> II
      Offset 02h, 2 Byte = Version Nr.
      Offset 04h, 4 Byte = pointer to first IFD entry
      Description of TIFF header: https://www.awaresystems.be/imaging/tiff/faq.html#q3
       

      Howto read and analyse the binary content correctly? This is my messy, not operational code:
       
      $sampleTiff="H:\daten\tif\11\11\111111.TIF" $h=FileOpen($sampleTiff,16) $content=FileRead($h) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $content = ' & $content & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console FileClose($h) $type=VarGetType($content) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $type = ' & $type & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console $ToString=BinaryToString($content) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $ToString = ' & $ToString & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console ConsoleWrite(@CRLF & @CRLF) $content=StringTrimLeft($content,2) ; cut off the leading "0x" ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $content = ' & $content & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console for $i = 1 to 8 step 8 $next=StringMid($content,$i,2) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $next = ' & $next & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console $Chr=BinaryToString($next) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $Chr = ' & $Chr & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console ConsoleWrite(@CRLF & "---" & @CRLF) Next Regards, Rudi.
      111111.TIF
    • ur
      By ur
      When I am trying to compile the autoit files with aut2exe.
      I am getting below error.
      There is no issue in code as the same code is getting compiled on different machine.
      I tried reinstalling the AUtoIT, but the issue replicates.

      Any suggestions?