Jump to content
Sign in to follow this  
Graywalker

Read Binary index.dat file

Recommended Posts

Graywalker

The goal here is to get the Internet History for endpoints and users in the enterprise to find what business use websites use Java.

When opening index.dat in Notepad, I can see all the data I need. Unfortunatly, trying to read that data in AutoIT is proving diffucult.

I've tried :

Func _ParseIndexdat($indexdatpath)
; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($Bindexdat)
$strIndexdat = BinaryToString($indexdat,2)
MsgBox(0,"String from Binary",$strIndexdat)
$FileArray = StringSplit($strIndexdat,@CRLF)
;This may get complex...
Dim $r = 1 ; to count the records
Dim $e = 0; to count the entries

; Start reading from line 1
For $line In $FileArray
$content = StringStripWS($line,7)
  Select
   Case StringInStr($line,"REDR")
    ; this is the start of a record
    MsgBox(0,"REDR",$content)
   Case StringInStr($line,"URL")
    ; this may be the start of a record
    MsgBox(0,"URL",$content)
   Case StringInStr($line,"LEAK")
    ; this is the start of an error record
    ; I will likely ignore it.
    MsgBox(0,"LEAK",$content)
   Case StringInStr($line,"http://")
    ; this is the line with a couple entries
    MsgBox(0,"http",$content)
   Case StringInStr($line,"Content-Type:")
    ; this is an entry I want
    MsgBox(0,"Content-Type",$content)
   Case StringInStr($line,"X-Powered-By:")
    ; this is an entry I want
    MsgBox(0,"Powered-By",$content)
   Case StringInStr($line,"~U:")
    ; this is an entry I want and it marks the end of a record
    MsgBox(0,"~U",$content)
   Case Else
    ; do nothing with the line
  EndSelect
Next
EndFunc

That doesn't get the info... That code doesn't return anything.

Using

$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($Bindexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$strIndexdat)

the "string from binary" msg box shows a LOT of data... but URL and http case are both blank or gibberish. BinaryToString($indexdat,2) and ,3 return Nothing in the cases - 4 does the same as 1.

Using :

$Bindexdat = FileOpen($indexdatpath, 16)
$indexdat = FileRead($Bindexdat);$indexdatpath)
MsgBox(0,"Index dat",$indexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat,@CRLF)

Shows that $indexdat and $strIndexdat are effectively the same.

Using :

; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
;$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($indexdatpath);$Bindexdat)
;$strIndexdat = BinaryToString($indexdat,4)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($indexdat,@CRLF)

It reads a lot more info... Content-Type is fine. ~U: is fine. URL is mostly blank.

Case http:// it will pop up the msg box, but $content is blank. So the data is there... I just can't figure out how to get it into a string.

Using :

$Bindexdat = FileOpen($indexdatpath,256)
$indexdat = FileRead($Bindexdat);$indexdatpath)
MsgBox(0,"Index dat",$indexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat,@CRLF)

Returns data, but URL and Http:// pop up as blank...

$Bindexdat = FileOpen($indexdatpath,48) - URL and Http are blank.

I've found a VBScript that is supposed to read the files... so far no luck on Win7 - it can't find index.dat file... lol! On remote XP machines, it can't find a history folder. So I don't know if it DOES read index.dat files.

I've attached it.

Any ideas or code help is greatly appreciated!!

IE_Network - Copy.txt

Edited by Graywalker

Share this post


Link to post
Share on other sites
Graywalker

I altered the vbscript to point directly to the index.dat file and it gets info, but returns all kinds of gibberish.

... that totally messes up the reply even when pasted as code!

So, I've attached it as a screen cap.

post-38206-0-27179900-1327335952_thumb.p

Edited by Graywalker

Share this post


Link to post
Share on other sites
Graywalker

Okay, I've found out that the data IS there. I've just got to find a way to get TO it.

Func _ParseIndexdat($indexdatpath)
; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
$Bindexdat = FileOpen($indexdatpath, 16)
$indexdat = FileRead($Bindexdat);$indexdatpath)
;MsgBox(0, "Index dat", $indexdat)
$strIndexdat = BinaryToString($indexdat, 1)
$strIndexdat = StringStripWS($strIndexdat, 7)
;MsgBox(0, "String from Binary", $strIndexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat, @CRLF)
;This may get complex...
Dim $r = 1 ; to count the records
Dim $e = 0; to count the entries
; Start reading from line 1
For $line In $FileArray
  $line = StringReplace($line, @CRLF, "")
  $line = StringReplace($line, @CR, "")
  $line = StringReplace($line, @LF, "")
  Select
   Case StringInStr($line, "http://")
    ; this is the line with a couple entries
    ;MsgBox(0, "http", $line)
    ; Check to see if URL is in the line
    If StringInStr($line, "URL") Then
     ; it is a REDR or LEAK, trim to URL
     $urlpos = StringInStr($line, "URL")
     $line = StringTrimLeft($line, $urlpos)
    EndIf
    $httppos = StringInStr($line, "http")
    $line = StringTrimLeft($line, $httppos - 1)
    $dotpos = StringInStr($line, ".", "", 3)
    If $dotpos > 10 Then
     $content = $line
     $linelen = StringLen($line)
     $trimfromright = $linelen - ($dotpos + 3)
     $line = StringTrimRight($line, $trimfromright)
     ;MsgBox(0,"Trim", "string lenght : " & $linelen & @CRLF & "dot position: " & $dotpos & @CRLF & "Trim from Right : " & _
     ;$trimfromright & @CRLF & $line)
    Else
     ; nothing
    EndIf
    $record = $line & ","
   Case StringInStr($line, "Content-Type:")
    ; this is an entry I want
    $line = StringStripWS($line, 7)
    $record = $record & $line & ","
   Case StringInStr($line, "X-Powered-By:")
    ; this is an entry I want
    $line = StringStripWS($line, 7)
    $record = $record & $line & ","
   Case StringInStr($line, "~U:")
    ; this is an entry I want and it marks the end of a record
    $line = StringReplace($line, "~U:", "")
    $line = StringStripWS($line, 7)
    $record = $record & $line
    FileWriteLine($logfile, $record)
    $record = ""
   Case Else
    ; do nothing with the line
  EndSelect
Next
EndFunc   ;==>_ParseIndexdat

I am getting fairly spotty results - leaving in lots of random characters before and after the URLs.

Could anyone come up with a way to use StringRegExp to pull the URLs - all of them - from a line?

I've tried several from : http://regexlib.com/Search.aspx?k=URL&AspxAutoDetectCookieSupport=1

with no consistent luck.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • Stew
      By Stew
      (Edited from original.  Please note that I AM NOT AN AUTOIT EXPERT.  I write code using Autoit frequently but I am no expert, especially when it comes to I/O.  So any remarks that start with "Why did you..." can be answered by referring to the first sentence.  This project was done in Autoit because of an interface I built to display the data.)
      Attached is a program and ascii input file I wrote to read stock price data, convert it to binary and then read it back into the program in binary.  The goal was to show increased performance for reading the files in binary and provide a demo on how to read/write binary for int32, int64, double and strings for anyone who might find it helpful.  The results on my PC show the following:
      Time to read ascii file only: 456.981951167202
      Ascii read & process time: 6061.83075631701
      Binary write file time: 14787.9184635239
      Time just to read binary file: 42.418867292311
      Binary read and process time: 4515.16129830537
      A couple things to note:
      1) The 32 MB ascii file took 10x longer to read than the 15 MB binary file.  Not entirely sure why.  Both were read into a buffer.
      2) The Binary write takes a long time but I made no effort to optimize this because the plan was to write this file one time only so I don't mind if it takes longer to write this file.  I care much more about how long it takes to read the file because I will be reading it many times.
      3) There was a modest gain in converting the ascii file to binary in terms of file size and reading speed.
      So big picture... not sure it's worth the effort to convert the files to binary even though most of the data is numerical data in the binary file.  That was actually surprising as I expected there would be more of a difference.  Any ideas on how to get the binary data to read at a faster rate would be great.
       
      binary.au3
      2019_02_08.zip
    • TheSaint
      By TheSaint
      An adaption of an adaption.
      A good while back, I created a program, KindEbook Wishlist, that I use most days and is still available here at the forum. It works well, keeping tabs on price changes for Kindle ebooks at Amazon ... at least for my modest wants.
      About a year or so later, I struck upon the idea of adapting that program for CDs, DVDs, Blu-rays etc from the JB Hifi store, where there was a limit of about 50 items on the in-store wishlist. JB Wishlist is kind of a niche program, really only suitable for those from AUS, at the AutoIt Forum, who would want such a thing ... so not many I imagine, so it has never been available here. However, if you meet that criteria etc, and are willing to register at the AutoIt4Life Clubrooms, then it is available in the Project Chat section there ... though not the more recent updates (you need to ask me about them). That too, works quite well for my modest needs, and even includes an additional Preview & Summary window, which KindEbook Wishlist doesn't have ... though both programs do have access to a Details window per item, that is like the Preview window ... just doesn't constantly display alongside the Main program window (see later for an example with IonGoG Wishlist).
      So, now we come to IonGoG Wishlist, which is an adaption of the JB Wishlist program, and used for keeping tabs on GOG Games ... prices, history of changes, bought list, etc.
      IonGoG Wishlist is an incomplete adaption, so still in beta ... though all the really important elements work well enough. I have been working on it in dribs and drabs, and I now think it has come to a reasonable enough stage to share it ... just be advised, that some of the features accessed via the right-click menu etc, may not work or give strange results. Any of the options that I have given an Accelerator key facility to, should be working fine ... and most of the others are too ... and I am not even sure at this point, of what remains to be adapted ... not much I imagine.
      The name by the way, in case you were wondering, is a phonetic mangling on the words - keep an Eye On your GOG Wishlist.
      Here is a screenshot, with the Preview window on left. The Preview window can be placed at right instead or even turned off.

      Download, includes source files.
      IonGoG Wishlist v0.0_b21.zip  (see detail at Post#16)
      Enjoy!
      OLDER DOWNLOADS
      SUPPLEMENTARY
      I have another program that GOG users may be interested in, that can assist with getting game images that IonGoG Wishlist gets and doesn't get ... larger versions taken from modified thumbnail links in their GOG Library.
      Downloads Dropbox  (see the GOG example in Post #7)
      ADDENDUM
      In reality, compared to most stores I have come across, the GOG store is great, well setup and laid out. and for most things, including their Wishlist, is more than adequate. So I rarely do a full list Price Query ... so much quicker and easier to just check the two pages of my in-store wishlist. When I first started work on IonGoG Wishlist, the in-store wishlist wasn't as good as it is now.
      All that said, I still find it handy to use IonGoG Wishlist, for a variety of things - Offline browsing, Cover images to use with my bought & downloaded game folders, price changes history (and patterns of GOG behavior for sales etc), a bought list (with price I paid and date etc), Game Notes & Warnings, etc. And while not as quick as just checking the pages of your in-store wishlist, you can just set the full, favorites or non-favorites Query going, while you go away and do something else, and then later come back and look at the highlighted changes that may have occurred.
    • Jemboy
      By Jemboy
      Hi,
      At work we have some proprietary website, users have to login to.
      I have "made" an autoit executable to start IE, go the website, login, so the user do not have input their credentials every time.
      By NDA I am not allowed disclosed the URL of the website nor the login credentials
      So I made a fake website and an autoitscript to illustrate my question.
      #include <ie.au3> $oIE = _IECreate ("about:blank", 0, 1, 1, 1) $HWND = _IEPropertyGet($oIE, "hwnd") WinActivate ($HWND,"") WinSetState($HWND, "", @SW_MAXIMIZE) _IENavigate ($oIE, "http://demo.rkilinc.nl",1) The above start my demo website. The actual website has some links in the footer that I do not want most people click on.
      I contacted the developers of the website and they are thinking of making an option to configure what links to show in the footer, but they said it's not a high priority for them.
       
      I discovered, that by click F12 and deleting the <footer> element the footer is delete from the live page view (until the page is reloaded off course)
      I want to automate the removal of the footer, without using things like send().
      I tried getting the footer with _IEGetObjById and deleting it with _IEAction, but that didn't work.
      Does any one has an idea how I could delete the footer directly from view with an autoit script?
      TIA, Jem.
       
       

    • TLAM
      By TLAM
      Hello,
      I am working on an autoIT script for CyberArk which is running Internet Explorer and connecting the user to a web application.
      For an unknown reason (I cannot explain why), IE has some trouble to start, only after a long disconnecting period (morning or after lunch). If he tries again, no problem, IE starts.
      I open a case with CyberArk but I am also searching a workaround, I thought to kill the process directly, or set a timeout with _IELoadWait but the process is not really created..

      May do you have any ideas for helping me ?

      Thanks in advance
       
    • Seminko
      By Seminko
      Wrote a script that grabs all of the IP addresses from Netflix's IP log, checks the IPs and returns suspicious activity.
      Everything works as it should but only when _IECreate is set to visible. When visible is set to false, it fails to login for some reason.
      Any ideas what might cause it and/or how to circumvent that?
×