Jump to content
Sign in to follow this  
Graywalker

Read Binary index.dat file

Recommended Posts

The goal here is to get the Internet History for endpoints and users in the enterprise to find what business use websites use Java.

When opening index.dat in Notepad, I can see all the data I need. Unfortunatly, trying to read that data in AutoIT is proving diffucult.

I've tried :

Func _ParseIndexdat($indexdatpath)
; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($Bindexdat)
$strIndexdat = BinaryToString($indexdat,2)
MsgBox(0,"String from Binary",$strIndexdat)
$FileArray = StringSplit($strIndexdat,@CRLF)
;This may get complex...
Dim $r = 1 ; to count the records
Dim $e = 0; to count the entries

; Start reading from line 1
For $line In $FileArray
$content = StringStripWS($line,7)
  Select
   Case StringInStr($line,"REDR")
    ; this is the start of a record
    MsgBox(0,"REDR",$content)
   Case StringInStr($line,"URL")
    ; this may be the start of a record
    MsgBox(0,"URL",$content)
   Case StringInStr($line,"LEAK")
    ; this is the start of an error record
    ; I will likely ignore it.
    MsgBox(0,"LEAK",$content)
   Case StringInStr($line,"http://")
    ; this is the line with a couple entries
    MsgBox(0,"http",$content)
   Case StringInStr($line,"Content-Type:")
    ; this is an entry I want
    MsgBox(0,"Content-Type",$content)
   Case StringInStr($line,"X-Powered-By:")
    ; this is an entry I want
    MsgBox(0,"Powered-By",$content)
   Case StringInStr($line,"~U:")
    ; this is an entry I want and it marks the end of a record
    MsgBox(0,"~U",$content)
   Case Else
    ; do nothing with the line
  EndSelect
Next
EndFunc

That doesn't get the info... That code doesn't return anything.

Using

$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($Bindexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$strIndexdat)

the "string from binary" msg box shows a LOT of data... but URL and http case are both blank or gibberish. BinaryToString($indexdat,2) and ,3 return Nothing in the cases - 4 does the same as 1.

Using :

$Bindexdat = FileOpen($indexdatpath, 16)
$indexdat = FileRead($Bindexdat);$indexdatpath)
MsgBox(0,"Index dat",$indexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat,@CRLF)

Shows that $indexdat and $strIndexdat are effectively the same.

Using :

; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
;$Bindexdat = FileOpen($indexdatpath,16)
$indexdat = FileRead($indexdatpath);$Bindexdat)
;$strIndexdat = BinaryToString($indexdat,4)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($indexdat,@CRLF)

It reads a lot more info... Content-Type is fine. ~U: is fine. URL is mostly blank.

Case http:// it will pop up the msg box, but $content is blank. So the data is there... I just can't figure out how to get it into a string.

Using :

$Bindexdat = FileOpen($indexdatpath,256)
$indexdat = FileRead($Bindexdat);$indexdatpath)
MsgBox(0,"Index dat",$indexdat)
$strIndexdat = BinaryToString($indexdat,1)
MsgBox(0,"String from Binary",$indexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat,@CRLF)

Returns data, but URL and Http:// pop up as blank...

$Bindexdat = FileOpen($indexdatpath,48) - URL and Http are blank.

I've found a VBScript that is supposed to read the files... so far no luck on Win7 - it can't find index.dat file... lol! On remote XP machines, it can't find a history folder. So I don't know if it DOES read index.dat files.

I've attached it.

Any ideas or code help is greatly appreciated!!

IE_Network - Copy.txt

Edited by Graywalker

Share this post


Link to post
Share on other sites

I altered the vbscript to point directly to the index.dat file and it gets info, but returns all kinds of gibberish.

... that totally messes up the reply even when pasted as code!

So, I've attached it as a screen cap.

post-38206-0-27179900-1327335952_thumb.p

Edited by Graywalker

Share this post


Link to post
Share on other sites

Okay, I've found out that the data IS there. I've just got to find a way to get TO it.

Func _ParseIndexdat($indexdatpath)
; Parse index.dat file for useable info
; The tools I've seen don't grab all the info I want :(
$Bindexdat = FileOpen($indexdatpath, 16)
$indexdat = FileRead($Bindexdat);$indexdatpath)
;MsgBox(0, "Index dat", $indexdat)
$strIndexdat = BinaryToString($indexdat, 1)
$strIndexdat = StringStripWS($strIndexdat, 7)
;MsgBox(0, "String from Binary", $strIndexdat);$strIndexdat & @CRLF & @error)
$FileArray = StringSplit($strIndexdat, @CRLF)
;This may get complex...
Dim $r = 1 ; to count the records
Dim $e = 0; to count the entries
; Start reading from line 1
For $line In $FileArray
  $line = StringReplace($line, @CRLF, "")
  $line = StringReplace($line, @CR, "")
  $line = StringReplace($line, @LF, "")
  Select
   Case StringInStr($line, "http://")
    ; this is the line with a couple entries
    ;MsgBox(0, "http", $line)
    ; Check to see if URL is in the line
    If StringInStr($line, "URL") Then
     ; it is a REDR or LEAK, trim to URL
     $urlpos = StringInStr($line, "URL")
     $line = StringTrimLeft($line, $urlpos)
    EndIf
    $httppos = StringInStr($line, "http")
    $line = StringTrimLeft($line, $httppos - 1)
    $dotpos = StringInStr($line, ".", "", 3)
    If $dotpos > 10 Then
     $content = $line
     $linelen = StringLen($line)
     $trimfromright = $linelen - ($dotpos + 3)
     $line = StringTrimRight($line, $trimfromright)
     ;MsgBox(0,"Trim", "string lenght : " & $linelen & @CRLF & "dot position: " & $dotpos & @CRLF & "Trim from Right : " & _
     ;$trimfromright & @CRLF & $line)
    Else
     ; nothing
    EndIf
    $record = $line & ","
   Case StringInStr($line, "Content-Type:")
    ; this is an entry I want
    $line = StringStripWS($line, 7)
    $record = $record & $line & ","
   Case StringInStr($line, "X-Powered-By:")
    ; this is an entry I want
    $line = StringStripWS($line, 7)
    $record = $record & $line & ","
   Case StringInStr($line, "~U:")
    ; this is an entry I want and it marks the end of a record
    $line = StringReplace($line, "~U:", "")
    $line = StringStripWS($line, 7)
    $record = $record & $line
    FileWriteLine($logfile, $record)
    $record = ""
   Case Else
    ; do nothing with the line
  EndSelect
Next
EndFunc   ;==>_ParseIndexdat

I am getting fairly spotty results - leaving in lots of random characters before and after the URLs.

Could anyone come up with a way to use StringRegExp to pull the URLs - all of them - from a line?

I've tried several from : http://regexlib.com/Search.aspx?k=URL&AspxAutoDetectCookieSupport=1

with no consistent luck.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • By Yodavish
      My AutoIt script works fine, for the most part, however, if it goes idle for roughly 30 minutes, it will won't be able to send data to a 3rd party application window titled, "Case #". 
      The problem lies within the function "SendFusion", it's able to see the handle for the "WinActivate" and "WinWaitActivate" (I've also tried putting in 10 seconds in the WinWaitActivate as well, doesn't work). However, if we have been idle, it will always fail to send data afterward. On what I've tried so far:
      "Send" function with the data and a {ENTER} i.e. Send($CaseTxt & "{ENTER}"), this won't even hit the enter key  ControlSetText, tried passing the handle from WinActivate, using the title "Case #" that was copied directly from the Autoit Window Info, hard coding the title, Using CLASS:WindowsForms10... always returns a 0 ControlCommand (same as above) always returns a 0 ControlGetFocus always returns a 0 ControlFocus, always returns a 0 Creating a new and simple Au3 script that just sends data to "Case #", also fails NOTE: If I close the Au3 script and just scan the data directly into the "Case #" or even type it, it works fine with no issues. I'm confused as to why it would not work after a long idle period?
      Below are the essential parts of the script, I can provide the entire thing if needed. Any suggestions would be greatly appreciated!
      #include <AutoItConstants.au3> #include <GUIConstantsEx.au3> #include <MsgBoxConstants.au3> #include <Process.au3> #include <Misc.au3> #include <IE.au3> ;--------------------------------------------------------------- ; Only one instance can run ;--------------------------------------------------------------- If _Singleton("gross.exe", 1) = 0 Then MsgBox($MB_SYSTEMMODAL, "Warning", "Gross.exe is already running. Please exit the existing version first (check the icons in the lower right corner of your screen) before running it again.") Exit EndIf ;--------------------------------------------------------------- ; AutoIt Options ;--------------------------------------------------------------- Opt("GUIOnEventMode", 1) ; Change to OnEvent mode Opt("WinWaitDelay", 0) ; Alters how long a script should briefly pause after a successful window-related operation. Time in milliseconds to pause (default=250). Opt("WinTextMatchMode", 1) ; Alters the method that is used to match window text during search operations. 2 = Quick mode Opt("SendKeyDelay", 0) ; Alters the length of the brief pause in between sent keystrokes. A value of 0 removes the delay completely. Time in milliseconds to pause (default=5). if ($fusionWindowTitle="NONE" or $fusionCaseNumWinTitle="NONE" or $appWindowTitle="NONE" or $winActivateTimeout="NONE" or $winCloseTimeout="NONE" or $winWaitTimeout="NONE" or $pdfViewerWidth="NONE" or $pdfViewerHeight="NONE" or $pdfViewerLeft="NONE" or $pdfViewerWindowName="NONE" or $pdfViewerExeName="NONE" or $pdfUrl="NONE" or $pdfDownloadCommand="NONE") Then Msgbox(0,"ERROR", "gross.ini is missing or does not contain all settings. Please contact Help Desk with this information") Exit -1 EndIf ;--------------------------------------------------------------- ; application settings/constants ;--------------------------------------------------------------- Global Const $[VARIABLE] = IniRead("gross.ini", "default", "[TitleOfVariable]", "NONE") Global Const $[VARIABLE] = IniRead("gross.ini", "default", "[TitleOfVariable]", "NONE") Global Const $[VARIABLE] = IniRead("gross.ini", "default", "[TitleOfVariable]", "NONE") Global Const $[VARIABLE] = IniRead("gross.ini", "default", "[TitleOfVariable]", "NONE") ;--------------------------------------------------------------- ; create the main window ;--------------------------------------------------------------- Local $mainWindow = GUICreate($appWindowTitle, 380, 190) ; create the main GUI window GUISetOnEvent($GUI_EVENT_CLOSE, "handleCloseClick") ; when the click the windows close button call handleCloseClick() WinSetOnTop($appWindowTitle, '', 1) ; Sets main GUI always on top ;--------------------------------------------------------------- ; create the controls on the main window ;--------------------------------------------------------------- Local $okButton = GUICtrlCreateButton("OK", 160, 160, 54,24) ; create OK button GUICtrlSetOnEvent($okButton, "validateInput") ; when they click ok, call handleInputProcessRequest() GUICtrlCreateLabel("Scan Input", 10, 10) ; create a label Global $inputScan = GUICtrlCreateInput("", 10,134,358) ; create the input scan box GUICtrlSetOnEvent($inputScan, "validateInput") ; when they press enter in the scan input box, call handleInputProcessRequest() GUICtrlSetState($inputScan,$GUI_FOCUS) ; automatically sets focus on the input field GUISetState(@SW_SHOW) ; main GUI loop that runs at all times ;--------------------------------------------------------------- While 1 ;---------------------------------------------------------- ; If sumatraPDF active, resets focus back to GUI ;---------------------------------------------------------- If WinActive($pdfViewerWindowName) Then consoleWrite('While Loop sumatraPDF was active' & @CRLF) setMainWindow() ;---------------------------------------------------------- ; After data sent to ProTracker, check for mismatch ; save button to trigger the event handler when clicked ;---------------------------------------------------------- ElseIf WinActive($proTrackerWindowTitle) Then $ie = _IEAttach("ProTracker") $mismatchButton = _IEGetObjByName($ie, $proTrackerMisMatchButtonId) $oEvent = ObjEvent($mismatchButton, "mismatchButton_") If @error Then setMainWindow() ;--------------------------------------------------------------------- ; If Fusion is prompting for the case #, focus back to main GUI input ; If second GUI 'Unknown' detect do nothing ;--------------------------------------------------------------------- ElseIf WinExists($fusionWindowTitle) and WinExists($fusionCaseNumWinTitle) and Not WinActive($mainWindow) and Not WinExists('Unknown Input') Then setMainWindow() EndIF ;--------------------------------------------------------------------- ; Checks if $processFlag = Done, if so, shows PDF viewer and clears ; processFlag for next iteration ;--------------------------------------------------------------------- IF WinExists($pdfViewerWindowName) and $processFlag == 'Done' Then WinSetState ($pdfViewerWindowName, '', @SW_SHOW) $processFlag = '' setMainWindow() EndIF Sleep(100) ; Sleep to reduce CPU usage WEnd Func handleInputProcessRequest($input) $processFlag = True Local $hTimer = TimerInit() consoleWrite('> handleInputProcessRequest: ' & @CRLF) Local $idInput = identifyInput($input) ; Select a proccess to run based on id input Select Case $idInput = "container" ; regex that captures only the case number $caseTxt = StringUpper(StringRegExpReplace($strippedInputData, '\??(\w*\d*-\d*|\d*).*$','$1')) If WinExists($fusionWindowTitle) and WinExists($fusionCaseNumWinTitle) Then sendProTracker(StringUpper($strippedInputData)) sendFusion($caseTxt) ; gets pdf for sumatraPDF in seperate script, since that function is the slowest Run("viewPDF.exe " & $caseTxt) ;getSumatraPDF($caseTxt) $lastCaseNum = $caseTxt Else ; Checks to make sure the previous "Container" case num is the same ; to the current Container case num, before sending to ProTracker If($lastCaseNum <> $caseTxt) Then ; If not the same case number send alert sound SoundPlay("error.wav") Else sendProTracker($strippedInputData) EndIf EndIf Case $idInput = "user" sendProTracker($strippedInputData) Case $idInput = "cassette" sendProTracker(StringUpper($strippedInputData)) Case $idInput = "unknown" selectUnknown() EndSelect Local $fDiff = TimerDiff($hTimer) $processFlag = 'Done' consoleWrite('> handleInputProcessRequest Completed Total time: ' & $fDiff & ' ' &@CRLF&@CRLF&@CRLF) EndFunc Func sendFusion($caseTxt)     Local $hTimer = TimerInit()     consoleWrite('+ sendFusion initiated: ' & $caseTxt & @CRLF)     If $caseTxt = "" Then         MsgBox(0, "Error", "Not a valid case number")     Else         ;$fusionCaseNumWinTitle = "Case #"         $retVal1 = WinActivate($fusionCaseNumWinTitle, "")         consoleWrite("ReturnValue WinActivate " & $retVal1 & @CRLF)         $retVal2 = WinWaitActive($fusionCaseNumWinTitle,"",$winActivateTimeout)         consoleWrite("ReturnValue WinWaitActive: " & $retVal2 & @CRLF)         $retVal3 = ControlSetText($fusionCaseNumWinTitle, "", "", $caseTxt)         ;$retVal3 = ControlCommand($fusionCaseNumWinTitle, "", "", "EditPaste", $caseTxt)         consoleWrite("ReturnValue ControlSetText: " & $retVal3 & @CRLF)         ;Send($caseTxt & "{ENTER}")         WinWaitClose($fusionCaseNumWinTitle, "", $winCloseTimeout)         consoleWrite('+ sendFusion WinWaitClose: ' & @CRLF)     EndIf     Local $fDiff = TimerDiff($hTimer)     ConsoleWrite('+ sendFusion Completed Total time:' & $fDiff & ' ' & @CRLF) EndFunc
      Console logs from the SciTLE
      Window (x86) Info matches the correct handle for the WinActivate and WinWaitActivate

      Window Control (x86) Info, I've tried the CLASS, the ID, also I just noticed that the "handle" in "Control" and "Window" appear to be different as well.

       
      So far the only work-around is to close down the 3rd party application and the AutoIt script, re-open them and it works all completely fine. But this is a pain for the end-user since it's all touch screen and it slows down their workflow, which they can be extremely sensitive about.
       
    • By TheSaint
      An adaption of an adaption.
      A good while back, I created a program, KindEbook Wishlist, that I use most days and is still available here at the forum. It works well, keeping tabs on price changes for Kindle ebooks at Amazon ... at least for my modest wants.
      About a year or so later, I struck upon the idea of adapting that program for CDs, DVDs, Blu-rays etc from the JB Hifi store, where there was a limit of about 50 items on the in-store wishlist. JB Wishlist is kind of a niche program, really only suitable for those from AUS, at the AutoIt Forum, who would want such a thing ... so not many I imagine, so it has never been available here. However, if you meet that criteria etc, and are willing to register at the AutoIt4Life Clubrooms, then it is available in the Project Chat section there ... though not the more recent updates (you need to ask me about them). That too, works quite well for my modest needs, and even includes an additional Preview & Summary window, which KindEbook Wishlist doesn't have ... though both programs do have access to a Details window per item, that is like the Preview window ... just doesn't constantly display alongside the Main program window (see later for an example with IonGoG Wishlist).
      So, now we come to IonGoG Wishlist, which is an adaption of the JB Wishlist program, and used for keeping tabs on GOG Games ... prices, history of changes, bought list, etc.
      IonGoG Wishlist is an incomplete adaption, so still in beta ... though all the really important elements work well enough. I have been working on it in dribs and drabs, and I now think it has come to a reasonable enough stage to share it ... just be advised, that some of the features accessed via the right-click menu etc, may not work or give strange results. Any of the options that I have given an Accelerator key facility to, should be working fine ... and most of the others are too ... and I am not even sure at this point, of what remains to be adapted ... not much I imagine.
      The name by the way, in case you were wondering, is a phonetic mangling on the words - keep an Eye On your GOG Wishlist.
      Here is a screenshot, with the Preview window on left. The Preview window can be placed at right instead or even turned off.

      Download, includes source files.
      IonGoG Wishlist v0.0_b21.zip  (see detail at Post#16)
      Enjoy!
      OLDER DOWNLOADS
      SUPPLEMENTARY
      I have another program that GOG users may be interested in, that can assist with getting game images that IonGoG Wishlist gets and doesn't get ... larger versions taken from modified thumbnail links in their GOG Library.
      Downloads Dropbox  (see the GOG example in Post #7)
      ADDENDUM
      In reality, compared to most stores I have come across, the GOG store is great, well setup and laid out. and for most things, including their Wishlist, is more than adequate. So I rarely do a full list Price Query ... so much quicker and easier to just check the two pages of my in-store wishlist. When I first started work on IonGoG Wishlist, the in-store wishlist wasn't as good as it is now.
      All that said, I still find it handy to use IonGoG Wishlist, for a variety of things - Offline browsing, Cover images to use with my bought & downloaded game folders, price changes history (and patterns of GOG behavior for sales etc), a bought list (with price I paid and date etc), Game Notes & Warnings, etc. And while not as quick as just checking the pages of your in-store wishlist, you can just set the full, favorites or non-favorites Query going, while you go away and do something else, and then later come back and look at the highlighted changes that may have occurred.
    • By Stew
      (Edited from original.  Please note that I AM NOT AN AUTOIT EXPERT.  I write code using Autoit frequently but I am no expert, especially when it comes to I/O.  So any remarks that start with "Why did you..." can be answered by referring to the first sentence.  This project was done in Autoit because of an interface I built to display the data.)
      Attached is a program and ascii input file I wrote to read stock price data, convert it to binary and then read it back into the program in binary.  The goal was to show increased performance for reading the files in binary and provide a demo on how to read/write binary for int32, int64, double and strings for anyone who might find it helpful.  The results on my PC show the following:
      Time to read ascii file only: 456.981951167202
      Ascii read & process time: 6061.83075631701
      Binary write file time: 14787.9184635239
      Time just to read binary file: 42.418867292311
      Binary read and process time: 4515.16129830537
      A couple things to note:
      1) The 32 MB ascii file took 10x longer to read than the 15 MB binary file.  Not entirely sure why.  Both were read into a buffer.
      2) The Binary write takes a long time but I made no effort to optimize this because the plan was to write this file one time only so I don't mind if it takes longer to write this file.  I care much more about how long it takes to read the file because I will be reading it many times.
      3) There was a modest gain in converting the ascii file to binary in terms of file size and reading speed.
      So big picture... not sure it's worth the effort to convert the files to binary even though most of the data is numerical data in the binary file.  That was actually surprising as I expected there would be more of a difference.  Any ideas on how to get the binary data to read at a faster rate would be great.
       
      binary.au3
      2019_02_08.zip
    • By Jemboy
      Hi,
      At work we have some proprietary website, users have to login to.
      I have "made" an autoit executable to start IE, go the website, login, so the user do not have input their credentials every time.
      By NDA I am not allowed disclosed the URL of the website nor the login credentials
      So I made a fake website and an autoitscript to illustrate my question.
      #include <ie.au3> $oIE = _IECreate ("about:blank", 0, 1, 1, 1) $HWND = _IEPropertyGet($oIE, "hwnd") WinActivate ($HWND,"") WinSetState($HWND, "", @SW_MAXIMIZE) _IENavigate ($oIE, "http://demo.rkilinc.nl",1) The above start my demo website. The actual website has some links in the footer that I do not want most people click on.
      I contacted the developers of the website and they are thinking of making an option to configure what links to show in the footer, but they said it's not a high priority for them.
       
      I discovered, that by click F12 and deleting the <footer> element the footer is delete from the live page view (until the page is reloaded off course)
      I want to automate the removal of the footer, without using things like send().
      I tried getting the footer with _IEGetObjById and deleting it with _IEAction, but that didn't work.
      Does any one has an idea how I could delete the footer directly from view with an autoit script?
      TIA, Jem.
       
       

    • By TLAM
      Hello,
      I am working on an autoIT script for CyberArk which is running Internet Explorer and connecting the user to a web application.
      For an unknown reason (I cannot explain why), IE has some trouble to start, only after a long disconnecting period (morning or after lunch). If he tries again, no problem, IE starts.
      I open a case with CyberArk but I am also searching a workaround, I thought to kill the process directly, or set a timeout with _IELoadWait but the process is not really created..

      May do you have any ideas for helping me ?

      Thanks in advance
       
×
×
  • Create New...