How to search for text in a .txt file and open a URL using it

[gr8BigGeek]

I'm trying to make a GUI with one textbox and one button that will always stay on top of other applications. When you enter text (ex. "c123") in the textbox and press the 'enter' button it will conduct a search of a .txt file (see the attached file).

I have an SQL set up already that is pinging c123's GPS (lat/long) and dropping them (live) into this .txt file. It delivers the information from top to bottom, so I need to search this .txt file (without opening it) from bottom to top (so I have the most recent GPS). This "messages.txt" file stays in the same location on the C:\ drive and doesn't move.

Once the enter button had been pressed and the search conducted, I needed to open a new instance of the default Internet browser using that URL (with the added lat/long inserted) so I can map where c123 is.

The other issue I'm having is the lat/long shows up without decimal places in my .txt file after my SQL delivers the info. Even if I could get the latitude/Longitude to attach to the URL it would show up like this:

"http://maps.google.com/maps?q=+33480260,-11174307"

which will return with an error, unless it has those decimals in there like this:

"http://maps.google.com/maps?q=+33.480260,-111.74307"

How can I extract the data and add in those decimal places at the right points each time (along with all of these other things I'm having issues with)?

I know these is a lot of questions to ask all at once, but any help you could provide would be awesome!!

messages.txt

[BrettF]

Hi gr8BigGeek,

To extract the data you can use FileReadLine ("FILE", -1) which will return the last line in the file. To open a URL in the default browser use ShellExectute.

As for changing the URL, I'm not 100% sure on how to do that, maybe look at the criteria for the GPS coordinates and see if there is any patterns and the like?

Cheers,

Brett

[ssubirias3]

I know these is a lot of questions to ask all at once, but any help you could provide would be awesome!!

Hi, a few months back a wrote a script for a buddy who wanted to use Microsoft MapPoint with his TomTom GPS. That was a fun project and taught me a bit about working with these strings that aren't always in the "correct" format we need. Here's some feedback that hopefully will help you moving forward.

1) You won't be able to read any file without opening it, at least in the background.

2) You realize the file is writing long/lat and your script with have to reorder those numbers in ADDITION to inserting the "."

3) Searching the Help file a little more could have given you the answer BrettF gave you about the FileReadLine() & ShellExecute().

Good job putting some effort in describing your goal and provided more info than the usual newbie. Play around with this script and see what you can learn by looking up the commands you don't recognize. This should get you started.

Cheers!!!

expandcollapse popup

Global $gmap = "http://maps.google.com/maps?q=+", $lat, $lon
Global $sfile = "c:\messages.txt"


$file = FileOpen($sfile, 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

$eof = FileReadLine($sfile, -1)

FileClose($file)

; parses last line ($eof) into an array
$aString = StringRegExp($eof, "\w+[^\|]", 3)

$lat = $aString[10]
$lon = $aString[9]

; splits $lat string into two parts to allow insert of "." then rejoins
$lat1 = StringLeft($lat, 2)
$lat2 = StringRight($lat, 6)
$lat = $lat1 & "." & $lat2

; splits $lon string into two parts to allow insert of "." then rejoins
$lon1 = StringLeft($lon, 3)
$lon2 = StringRight($lon, 6)
$lon = $lon1 & "." & $lon2

; put it all together in a URL that any browser and Google with like
$myURL = $gmap & $lat & ",-" & $lon


ShellExecute($myURL)

[gr8BigGeek]

ssubirias3,

Thank you very much for your help with that code! What you have sent works wonderfully and I'm very grateful that you have helped me this far.

I have been playing with the code, searching the help file, reading on forums and basically just rearranged the concept of what you had sent me (because it's the best code I've found). I am running into a problem with viewing the last line in the messages.txt file though. This is because the units I am searching for change from time to time. I am monitoring multiple GPS units at a time so the following may occur:

[sat Apr 10 05:53:31 2010] |AVL1|C121|-112065260|33428810|0800|0800|064000|064000|||||46414000|0|2|1000|3|20||

[sat Apr 10 05:53:52 2010] |AVL1|C122|-112059120|33428540|1000|1000|064000|064000|||||46434000|0|2|1000|3|20||

[sat Apr 10 05:54:11 2010] |AVL1|C123|-112053440|33427640|1020|1020|054000|054000|||||46454000|0|2|1000|3|20||

Then I need to know where c121 is at a given time. This unit may not be at the bottom of the document and depending on how my other program is pinging their GPS this unit I'm searching for may be up in the document quiet a bit. If I can search for it from bottom > top then I will find the most recent because the SQL is set up to auto-populate this messages.txt file which (opens the file, dumps the update inside) places the most recent information to the bottom (and then saves).

If there is an input box option where it can compare the strings that would be what is needed. This way it could search from bottom > top, then once the strings (or unit identifiers such as "c121") are matched it will begin something like:

"$aString = StringRegExp($eof, "\w+[^\|]", 3)"

Thus, returning the URL and opening it up in the default browser. Also, the end user may be using cases. The .txt document is ALL CAPS on the unit codes so if they search using lowercase will I run into a problem?

I'm much more proficient using VBA that is for sure! I am, however, enjoying Autoit! I like how you can compress right away and begin using a .exe file. More importantly, I am enjoying how everyone is so very friendly and helpful to one another on these forums. I really enjoy the learning environment.

Thank you in advance for any further help.

Your Geek,

Matt~

My current code:

expandcollapse popup

#AutoIt3Wrapper_au3check_parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <GUIConstantsEx.au3>
#include <GuiButton.au3>
#include <WindowsConstants.au3>

Global $gmap = "http://maps.google.com/maps?q=+", $lat, $lon
Global $sfile = "e:\messages.txt"

;dim $sfile, $gmap

Opt("MustDeclareVars", 1) ;doesn't really have to be here (yet)

Global $iMemo

_Main()

Func _Main()
    Local $btn2, $msg, $label_1
    
    GUICreate("Finder", 150, 40) ; will create a dialog box that when displayed is centered
    
    ;I want to make this an input box but I don't know how to use the input to search
    ;the messages.txt file for that specific bit of data.
    
    $label_1 = GuiCtrlCreateLabel ("Needs input box here", 2, 10)
    GuiCtrlSetColor (-1, 0xcc00cc)
    
;Create button on the form
$btn2 = GUICtrlCreateButton("Find", 115, 5, 30, 25)

    GUISetState()  

;when button is pressed do this:
While 1
        $msg = GUIGetMsg()
        Select
            Case $msg = $GUI_EVENT_CLOSE
                ExitLoop
            Case $msg =$btn2
            ;when you push the button go to this command please:
                UnitLookup()
                ExitLoop
        EndSelect
    WEnd

    Exit
EndFunc   ;==>_Main

func UnitLookup()
    Local $file, $eof, $aString, $lat, $lon, $lat1, $lat2, $lon1, $lon2, $myURL, $gmap
Global $gmap = "http://maps.google.com/maps?q=+", $lat, $lon
;Global $sfile = "e:\messages.txt"
$file = FileOpen($sfile, 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to find that unit.")
    Exit
else
;Looks at the last line in the text ONLY
$eof = FileReadLine($sfile, -1)

FileClose($file)

; parses last line ($eof) into an array
$aString = StringRegExp($eof, "\w+[^\|]", 3)

$lat = $aString[10]
$lon = $aString[9]

; splits $lat string into two parts to allow insert of "." then rejoins
$lat1 = StringLeft($lat, 2)
$lat2 = StringRight($lat, 6)
$lat = $lat1 & "." & $lat2

; splits $lon string into two parts to allow insert of "." then rejoins
$lon1 = StringLeft($lon, 3)
$lon2 = StringRight($lon, 6)
$lon = $lon1 & "." & $lon2

; put it all together in a URL that any browser and Google with like
$myURL = $gmap & $lat & ",-" & $lon

;couldn't get $gmap to combine with $myURL so I manually typed in the URL here with a + $myURL
;which containes the two latitude and longitude combined:
ShellExecute( "http://maps.google.com/maps?q=+"& $myURL)
EndIf
EndFunc

Searching txt Doc.au3

[jchd]

You say that your SQL will append data in more or less random order with respect to different units. But why not query directly the database engine yourself from AutoIt to obtain what you need (wihout having to search by linear scan using regexps)?

Having two processes, one writer (the SQL output) and one unbound reader (you don't know in advance how many lines you will need to read from the bottom, a slow operation) using the same file at the same time without sync is likely to produce errors or deadlocks.

That's precisely what database are for!

[gr8BigGeek]

jchd,

Very good point! I'm very glad you posted. Thank you for pointing that out and yes I am aware that the SQL/database could do what you have suggested. The problem is I do not have access to updating this SQL because it would violate our warranty on the mapping product that produces this SQL generated "messages.txt" document.

I have been asked to find out where it's dumping the information and search it for specific units. Reproducing the SQL and database is possible but it would require far more coding than finding one that would compare strings I believe (and my boss would not be happy if I violated a EULA or warranty).

Using the code that pulls the last line in the document seems to be working well with the actual SQL produced document without any errors so far.

Is there a command that will allow you to use an input box to search a specific document (Bottom>Top) and match two strings then return certain characters based on your search?

Matt~

[jchd]

Querying a database is not updating it. You wouldn't have to change the existing DB to read it. But if you have to do it in a much harder, less reliable way, why not! Such EULA would make a fun reading btw...

Anyway, the issue is that it isn't obvious that you can open that file and keep it open for the time you need to collect your information, and have another process open it for writing asynchronously. That could be the problem. Try opening the file (for reading) in AutoIt then pausing with InputBox or something and check that the other process can still append its data, but do so only if you can afford loosing some information during the test (in case the other process can't open for write).

Now is the message.txt file growing forever or is it purged or what?

Anyway, I would keep a running pointer of the FilePos of my last read, then when requested, read from there down to EOF and store information in my own DB or something.

Is that a fleet monitoring or similar?

[gr8BigGeek]

Alright, I opened messages.txt and it did not interrupt the SQL from running. I opened numerous instances of the .txt document and it appears that each time I opened the document it appeared as though I was opening a snapshot in time, if you will. Opening the file did not stop it from being updated either which leads me to believe that I could keep it open as long as I needed to while I searched for data.

The program begins storing the information in the document until the program is turned off and then restarted again. Loosing data from this file is not entirely important from a storage standpoint as the data is dumped upon the program beginning again anyway.

[jchd]

That's a good point. Hope future versions will keep on behaving this way (or allow querying the base ... :ahem: ).

Is the log file typically big, or rather is it too large to parse from the beginning every time you need to fetch data in it?

If it is, go with the FileGetPos and FileSetPos method, but again, that will need you code for your own storage: you store data for "device" 1 to N, and/or you store file pointers from where to read in file for device 1 to N. After that, storing in memory or on disk is mainly up to you except if volume data requires you do it on disk.

[gr8BigGeek]

This file can get pretty large (like 4 - 7GB+), however, I can restart it to purge the document and start over. It gets too large eventually and notepad cannot open it and you get an error message that advises you it is too large to open. Regardless, this happens only after it has been running for extended lengths of time (days or even a week as we restart the computers every week).

I can simply restart the program at the beginning of my shift and there should be no problem with it. As far as searching the document, it looks like I may only need to go back say 100 lines (from bottom > up)? Would that be too much? Also, would the case be matched exactly (some unit numbers are similar with one another ie. e646 and 646)? I'm not familiar with these code options do you know of some coding examples I could look at so I may be able to better understand their functions (and the necessary declaration I would need to add)?

You are being so very helpful and I'm thankful for you conversing with me about this. I do hope that I am being informative enough to satisfy your requests and I do apologize for my lack of knowledge in this area.

I look forward to hearing more from you!

Matt~

[jchd]

No need to apologize anyhow Matt, you're welcome. (I had to get some sleep at indecent hours so couldn't get back to you at once.)

OK, that's what I feared: data is produced at a fairly high rate, so it's out of question to read the file and parse from the start. Restarting is IMO only a last resort option.

We're left with searching from the end of file. No problem, AutoIt can do that as well.

There is no issue with the partial match of unit ids either, we'll take care of that.

You say 100 line could be enough, so let's take a _large_ margin and say we need to read as much as 500 lines (that's nothing). You can change that but don't cut corners too short.

Try this proof of concept for the search:

expandcollapse popup

Local $str[500] = [ _
    "[Sat Apr 10 05:50:27 2010] |AVL1|C123|-112108220|33444780|1800|1800|078000|078000|||||46230000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:50:48 2010] |AVL1|C623|-112108200|33438240|1800|1800|076000|076000|||||46251000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:08 2010] |AVL1|C158|-112108070|33432140|1640|1640|072000|072000|||||46271000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:28 2010] |AVL1|C173|-112103200|33428650|0970|0970|071000|071000|||||46291000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:49 2010] |AVL1|123|-112096360|33429070|0840|0840|062000|062000|||||46312000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:09 2010] |AVL1|C423|-112090400|33429560|0860|0860|063000|063000|||||46332000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:29 2010] |AVL1|C023|-112084560|33429390|0960|0960|065000|065000|||||46351000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:30 2010] |AVL1|C153|-112084240|33429360|0960|0960|065000|065000|||||46352000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:10 2010] |AVL1|C193|-112071820|33428230|0960|0960|066000|066000|||||46393000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:30 2010] |AVL1|C126|-112065870|33428710|0780|0780|064000|064000|||||46412000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:31 2010] |AVL1|C463|-112065260|33428810|0800|0800|064000|064000|||||46414000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:52 2010] |AVL1|C146|-112059120|33428540|1000|1000|064000|064000|||||46434000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:54:11 2010] |AVL1|C423|-112053440|33427640|1020|1020|054000|054000|||||46454000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:50:27 2010] |AVL1|C823|-112108220|33444780|1800|1800|078000|078000|||||46230000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:50:48 2010] |AVL1|C163|-112108200|33438240|1800|1800|076000|076000|||||46251000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:08 2010] |AVL1|C103|-112108070|33432140|1640|1640|072000|072000|||||46271000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:28 2010] |AVL1|C621|-112103200|33428650|0970|0970|071000|071000|||||46291000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:49 2010] |AVL1|C124|-112096360|33429070|0840|0840|062000|062000|||||46312000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:09 2010] |AVL1|D123|-112090400|33429560|0860|0860|063000|063000|||||46332000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:29 2010] |AVL1|C523|-112084560|33429390|0960|0960|065000|065000|||||46351000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:30 2010] |AVL1|h423|-112084240|33429360|0960|0960|065000|065000|||||46352000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:10 2010] |AVL1|C173|-112071820|33428230|0960|0960|066000|066000|||||46393000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:30 2010] |AVL1|C125|-112065870|33428710|0780|0780|064000|064000|||||46412000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:31 2010] |AVL1|C125|-112065260|33428810|0800|0800|064000|064000|||||46414000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:52 2010] |AVL1|C323|-112059120|33428540|1000|1000|064000|064000|||||46434000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:54:11 2010] |AVL1|C723|-112053440|33427640|1020|1020|054000|054000|||||46454000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:50:27 2010] |AVL1|C123|-112108220|33444780|1800|1800|078000|078000|||||46230000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:50:48 2010] |AVL1|C123|-112108200|33438240|1800|1800|076000|076000|||||46251000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:08 2010] |AVL1|2123|-112108070|33432140|1640|1640|072000|072000|||||46271000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:28 2010] |AVL1|123|-112103200|33428650|0970|0970|071000|071000|||||46291000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:51:49 2010] |AVL1|6123|-112096360|33429070|0840|0840|062000|062000|||||46312000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:09 2010] |AVL1|2123|-112090400|33429560|0860|0860|063000|063000|||||46332000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:29 2010] |AVL1|5923|-112084560|33429390|0960|0960|065000|065000|||||46351000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:52:30 2010] |AVL1|D184|-112084240|33429360|0960|0960|065000|065000|||||46352000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:10 2010] |AVL1|C111|-112071820|33428230|0960|0960|066000|066000|||||46393000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:30 2010] |AVL1|C141|-112065870|33428710|0780|0780|064000|064000|||||46412000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:31 2010] |AVL1|C198|-112065260|33428810|0800|0800|064000|064000|||||46414000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:53:52 2010] |AVL1|C143|-112059120|33428540|1000|1000|064000|064000|||||46434000|0|2|1000|3|20||", _
    "[Sat Apr 10 05:54:11 2010] |AVL1|C113|-112053440|33427640|1020|1020|054000|054000|||||46454000|0|2|1000|3|20||" _
]

; say you are looking for this device (actual value can be passed on the command-line, input in a CUI or GUI, choosen randomly, ...)
Local $units[3] = ["c423", "123", "c125"]

For $dev In $units
    Locate($dev)
Next

Func Locate($device)
    Local $time, $devlat, $devlon
    If Lookup($device, $str, $time, $devlat, $devlon) Then
        MsgBox(0, "Device position found", "Unit " & $device & " was last located " & $time & @LF & "at position (" & $devlat & ', ' & $devlon & ')')
    Else
        MsgBox(0, "Device can't be found", "Unit " & $device & " can't currently be located")
    EndIf
EndFunc

Func Lookup($unit, ByRef $data, ByRef $when, ByRef $lat, ByRef $lon)
    Local $res
    For $i = UBound($data) - 1 To 0 Step -1
        $res = StringRegExp($data[$i], "(?i)\[([^\]]+)\]\s+\|[^|]+\|" & $unit & "\|([^|]+)\|([^|]+)", 1)
        If @error Then ContinueLoop
        $when = $res[0]
        $lat = StringRegExpReplace($res[1], "(-?\d{3})(\d*)", "$1\.$2")
        $lon = StringRegExpReplace($res[2], "(-?\d{2})(\d*)", "$1\.$2")
        Return 1    ; success
    Next
    Return 0        ; fail
EndFunc

And tell if that can be tweeked into somrthing useful later.

[gr8BigGeek]

Thank you for this new code. I will now begin looking through it and seeing how I can make it work using an input box to compare for a line within these lines of text in this new code. I'm terribly sorry for asking so many questions but how would I import this data (or these last 500 +/- lines) for each time someone wishes to query the messages.txt document?

More specifically, I noticed you had quotation marks after the "Local $str[500] = [ _" part and then it looks like the text from the document there, however, I am wondering how that data is filled between those quotation marks to be searched?

You are certainly a very good programmer and your skill is so very helpful to me right now. I'm glad you have gotten your rest and I do look forward to learning more from you. I do hope you have a very wonderful day.

[jchd]

You're right, I cheated: the data was in an array (it was just to check that the match was what you were after).

But the following will actually read the file. Still barebones, but functional as far as I can tell.

expandcollapse popup

; my test dummy file (it is the file you posted in your first post + my modified version)
Global Const $datafile = @ScriptDir & "\messages.txt"       ;; change as needed
Global Const $MaxLines = 300                                ;; read up to last $MaxLines lines of above file
Global Const $LineSize = 113                                ;; estimated line size in bytes

Local $device, $devdata

; a simple loop to input unit IDs and fetch information. Exit with Escape or Cancel
While 1
    $device = InputBox("Unit locator", "Enter a device ID", "c123")
    If $device = '' Then ExitLoop
    $devdata = Lookup($device, $datafile)
    Switch @error
        Case 0
            Local $devcoord = StringSplit($devdata, '|', 2)
            MsgBox(64, "Device position found", "Unit " & $devcoord[1] & " was last located " & $devcoord[0] & @LF & _
                                                "at position (" & $devcoord[2] & ', ' & $devcoord[3] & ')')
        Case 1
            MsgBox(48, "Device can't be found", "Unit " & $device & " can't currently be located")
        Case 2
            MsgBox(16, "Fatal error!", $datafile & " can't be found")
    EndSwitch
WEnd
MsgBox(0, "Social message", "Have a nice day!", 2)

; Since we don't know exactly where to start reading the tail of the file, let's
; assume the format is essentially fixed (or at least well bounded). We can compute
; a good approximation of the file position we need to read from, in order to read
; about $MaxLines of useful data.
;
; A line of the example file is 111 characters, plus 2 (CRLF) = 113 characters.
; assuming UTF-8 or ANSI encoding, that is 113 bytes. If we position the reading point
; few bytes before $MaxLines * 113, and throw away the next line read (probably incomplete)
; we should be able to read about $MaxLines full records.
;
; The above strategy may have to be adapted if any assumption made doesn't hold for some
; reason.
;
; It's better to open the file at every lookup so that we minimize chance of missing
; newly added data by the logging process.
Func Lookup($unit, $filename)
    Local $hdl = FileOpen($filename)
    If $hdl = -1 Then Return(SetError(2, 0, ''))    ; no such file
    Local $pos = FileGetSize($hdl) - $MaxLines * $LineSize - 20
    If $pos > 0 Then
        FileSetPos($hdl, $pos, 0)                   ; set position, starting from begin of file
    EndIf
    FileReadLine($hdl)                              ; throw away until line break
    Local $data = FileRead($hdl)                    ; read to EOF
    FileClose($hdl)
    $data = StringSplit($data, @CRLF, 3)            ; make it an array of lines
    Local $res
    $devcoord = ''
    ; search from bottom up
    For $i = UBound($data) - 1 To 0 Step -1
        $res = StringRegExpReplace($data[$i], "(?i)\[([^\]]+)\]\s+\|[^|]+\|(" & $unit & ")\|(-?\d{3})(\d+)\|(-?\d{2})(\d+).*", "$1\|$2\|$3\.$4\|$5\.$6")
        If @extended <> 6 Then ContinueLoop
        $devcoord = $res
        Return(SetError(0, 0, $res))                ; return device data found
    Next
    Return(SetError(1, 0, ''))                      ; device not found
EndFunc

Read the comments and adapt to your case.

[gr8BigGeek]

I tip my hat to you! What a splendid job. This code works better than perfect! You have helped me out so much and this has been such a great learning experience. I have made a few adjustments to the code you provided but this code is working very well.

Thank you so much for standing by me during this time.

Humbly & Thankfully,

Matt~

[jchd]

Matt, you now "just" have to build a nice-looking GUI around, or make it accept command-line arguments, then plug it to your Google map thing, etc.

Happy to see this helped you.

FYI, here is how the extraction part (regexp) works, I mean the (ugly-looking) black magic:

$res = StringRegExpReplace($data[$i], "(?i)\[([^\]]+)\]\s+\|[^|]+\|(" & $unit & ")\|(-?\d{3})(\d+)\|(-?\d{2})(\d+).*", "$1\|$2\|$3\.$4\|$5\.$6")

Let's breaks it down into successive parts this way:

(?i) instruct the regexp engine to work case-insensitive

\[ matches a plain [ (this character has to be escaped in regexps)

([^\]]+) makes a capture (capture $0 is the global match, so this will be referred to by $1 in the replace part). That part works as follows:

[^\]]+ means one or more (the ending +) character which is not (is not is the caret ^) the character ] (again it is escaped)

\] followed by a [ (escaped)

\s+ followed by one or more (the + sign) whitespace (\s = one horizontal or vertical spacing character: space, tab, line feed, ...)

\| followed by a | (needs escaping as well)

[^|]+ followed by one or more character not equal to | (starting to get it?)

\| followed by |

(" & $unit & ") gets replaced by AutoIt into (c123) if $unit contains 'c123'. That not only requires a match but also captures the device number as it is actually in the file (to be able to display it with the case it has in file)

note that we could have spared capturing that since the goal is only cosmetic here. Anyway, this will be capture referred to a $2 in the replacement part

\| followed by |

(-?\d{3}) here we capture (that will be capture $3) an optional (? sign) negative leading sign - followed by exactly 3 (the {3} part) decimal digits (\d)

(\d+) now we capture the rest of the lattitude (or was it longitude?) fractional part as one or more (+) decimal digit (\d). This will be capture $4

\| followed by |

(-?\d{2})(\d+) now we capture the lattitude while breaking it into fixed integer and fractional parts, which will be $5 and $6 resp. in the replace string

.* finally we "consume" (match without capturing) the rest of the string as zero or more times (the * repeater) any character (the . dot stands for any character)

and we replace what we have matched (and we have matched exactly the whole string) by a concatenation of:

$1 was the timestamp

\| a litteral | character (we'll use it to split the returned string into individual components, so it's easy to use this char as it was already a separator in the input string, so no clash)

$2 was the unit (device ID) as writen in the file

\| our | separator

$3 integral part of lattitude

\. litteral decimal point

$4 fractional part of lattitude

\| again a separator

$5\.$6 finally the longitude (int.frac)

and this is what we return in case of match.

StringRegExpReplace returns the number of replaces made in @extended language-wide variable. If we didn't make 6 replacements, then $unit didn't match so we return a "device not found" error code.

In the calling function, when a match is found, we StringSplit the result, exploding it into the components we have little difficulty displaying, passing away to Google, etc.

I hope the comments are clear enough about how the file positionning, reading to EOF, storing in an array of lines and processing backwards (from the bottom) until a match is found.

Edit: a couple of typos + I realize I probably mixed up lattitude and longitude (honest, I didn't really care which was what) but that you will have corrected if needed.

Cheers

Edited April 14, 2010 by jchd

[gr8BigGeek]

FYI, here is how the extraction part (regexp) works, I mean the (ugly-looking) black magic:

Oh, very good! Yeah, your code was very complicated and I found myself going through it line by line to make sure I understood what was doing what. Much thanks again for this added explanation of what is happening. This helps me better understand what is happening as well.

If you wouldn't mind me troubling you for just one more question I was searching around to find the answer as well but have been trying to get this program to stay on top of everything else running on the PC. There is a link here.

This is a forum discussion with a link to WinSetOnTop . The only thing is that I cannot get that to work properly with this program. If I try to set it using WinSetOnTop I get an error message when the code runs up to that point.

Either way, you have helped me so much already. If this is a bit too much to ask here after you have done so much work for me I completely understand. You have been very gracious toward me thus far.

Your friend,

Matt~

[jchd]

Glad that you enjoy a bit of enlightning to how I massage your own data (don't get jealous).

You need a GUI (graphical User Interface, aka "window") for "stay on top" to be meaningful. That's what I meant saying it's now your task to wrap the baby in a fashionable GUI.

I've cranked up a very primitive skeleton:

expandcollapse popup

#include <ButtonConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>


#Region ### START Koda GUI section ### Form=
$Form1 = GUICreate("GPS locator", 330, 330, 192, 124, 0, $WS_EX_TOPMOST)
$Label1 = GUICtrlCreateLabel("Device to search", 56, 24, 85, 17)
$ibDevice = GUICtrlCreateInput("", 56, 40, 217, 21)
$btSearch = GUICtrlCreateButton("Search", 96, 80, 137, 33, $WS_GROUP)
$Label2 = GUICtrlCreateLabel("Time of last position", 56, 136, 97, 17)
$ibTime = GUICtrlCreateInput("", 56, 152, 217, 21, BitOR($ES_CENTER,$ES_AUTOHSCROLL,$ES_READONLY))
$Label3 = GUICtrlCreateLabel("Lattitude", 56, 184, 45, 17)
$ibLatt = GUICtrlCreateInput("", 56, 200, 97, 21, $ES_READONLY)
$Label4 = GUICtrlCreateLabel("Longitude", 176, 184, 51, 17)
$ibLong = GUICtrlCreateInput("", 176, 200, 97, 21, $ES_READONLY)
$btLocate = GUICtrlCreateButton("Locate", 200, 240, 73, 33, $WS_GROUP)
$btExit = GUICtrlCreateButton("Exit", 56, 240, 73, 33, $WS_GROUP)
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

GUICtrlSetState($btLocate, $GUI_DISABLE)


Global Const $datafile = @ScriptDir & "\messages.txt"       ;; change as needed
Global Const $MaxLines = 300                                ;; read up to last $MaxLines lines of above file
Global Const $LineSize = 113                                ;; estimated line size in bytes

Local $device, $devdata

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $btExit, $GUI_EVENT_CLOSE
            Exit
        Case $btSearch
            GUICtrlSetState($btLocate, $GUI_DISABLE)
            GUICtrlSetData($ibTime, "")
            GUICtrlSetData($ibLatt, "")
            GUICtrlSetData($ibLong, "")
            $device = GUICtrlRead($ibDevice)
            $devdata = Lookup($device, $datafile)
            Switch @error
                Case 0
                    Local $devcoord = StringSplit($devdata, '|', 2)
                    GUICtrlSetData($device, $devcoord[1])
                    GUICtrlSetData($ibTime, $devcoord[0])
                    GUICtrlSetData($ibLatt, $devcoord[2])
                    GUICtrlSetData($ibLong, $devcoord[3])
                    GUICtrlSetState($btLocate, $GUI_ENABLE)
                Case 1
                    GUICtrlSetData($ibTime, "-- no recent known position --")
                    GUICtrlSetData($ibLatt, "")
                    GUICtrlSetData($ibLong, "")
                Case 2
                    MsgBox(16, "Fatal error!", $datafile & " can't be found")
            EndSwitch
        Case $btLocate
            ; invoke Google Map or go play blackjack
    EndSwitch
WEnd

; Since we don't know exactly where to start reading the tail of the file, let's
; assume the format is essentially fixed (or at least well bounded). We can compute
; a good approximation of the file position we need to read from, in order to read
; about $MaxLines of useful data.
;
; A line of the example file is 111 characters, plus 2 (CRLF) = 113 characters.
; assuming UTF-8 or ANSI encoding, that is 113 bytes. If we position the reading point
; few bytes before $MaxLines * 113, and throw away the next line read (probably incomplete)
; we should be able to read about $MaxLines full records.
;
; The above strategy may have to be adapted if any assumption made doesn't hold for some
; reason.
;
; It's better to open the file at every lookup so that we minimize chance of missing
; newly added data by the logging process.
Func Lookup($unit, $filename)
    Local $hdl = FileOpen($filename)
    If $hdl = -1 Then Return(SetError(2, 0, ''))    ; no such file
    Local $pos = FileGetSize($hdl) - $MaxLines * $LineSize - 20
    If $pos > 0 Then
        FileSetPos($hdl, $pos, 0)                   ; set position, starting from begin of file
    EndIf
    FileReadLine($hdl)                              ; throw away until line break
    Local $data = FileRead($hdl)                    ; read to EOF
    FileClose($hdl)
    $data = StringSplit($data, @CRLF, 3)            ; make it an array of lines
    Local $res
    $devcoord = ''
    ; search from bottom up
    For $i = UBound($data) - 1 To 0 Step -1
        $res = StringRegExpReplace($data[$i], "(?i)\[([^\]]+)\]\s+\|[^|]+\|(" & $unit & ")\|(-?\d{3})(\d+)\|(-?\d{2})(\d+).*", "$1\|$2\|$3\.$4\|$5\.$6")
        If @extended <> 6 Then ContinueLoop
        $devcoord = $res
        Return(SetError(0, 0, $res))                ; return device data found
    Next
    Return(SetError(1, 0, ''))                      ; device not found
EndFunc

[gr8BigGeek]

Wonderful again. Very impressive. That works beautifully! It is everything I was needing and honestly I am jealous! thank you again.

[gr8BigGeek]

[^|]+ followed by one or more character not equal to | (starting to get it?)

Not yet. I've been going through this code line by line and I think I understand most all of it. I've come to try and test it myself to see something. If I can modify it and watch what my changes do then I will understand it.

My goal is try try and change the code:

$res = StringRegExpReplace($data[$i], "(?i)\[([^\]]+)\]\s+\|[^|]+\|(" & $unit & ")\|(-?\d{3})(\d+)\|(-?\d{2})(\d+).*", "$1\|$2\|$3\.$4\|$5\.$6")

So that I can read the first 3 characters of the 7th block of numbers:

[Mon Apr 12 17:46:18 2010] |AVL1|C119|-112127750|33762840|3500|3500|068000|068000|||||02805000|0|2|1000|4|20||

In this string I want to read the "068", which is 7 of these little "|" in from the right. If I can understand how to get that to happen (and understand how you are displaying it into text boxes) then I believe I will understand all of the code.

How can I add to the code so that it will read this number for me?

Sorry to be bothersome, just trying to learn everything in this code.

Many thanks.

[jchd]

No problem.

Try to locate in the regexp breakdown above a sub-pattern that will match "a bar | character followed by a sequence of one or more characters which are not equal to a bar |, followed by a bar". That's not too difficult.

Now try to insert it near the end of the regexp pattern so that it will cause a match of the next field we want to skip (ignore).

Do that another time, then devise a sub-pattern that will 'capture a sequence of exactly 3 decimal digits'.

After that, you're home with the regexp matching pattern.

Ask yourself how to refer to the new capture made and how to include it in the string we'll return, as we've already made for the previous fields.

Change the check we made to verify we did actually match the required number of fields.

Try to add the required GUI components into the GUI design part to display the new field.

Don't forget to clear the new field as needed, the same way we clear other displayed fields.

Display the new field in the place where it belongs.

TADA!

Did you need to hit this button?

expandcollapse popup

#include <ButtonConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>


#Region ### START Koda GUI section ### Form=
$Form1_1 = GUICreate("GPS locator", 331, 361, 192, 124, -1, BitOR($WS_EX_TOPMOST,$WS_EX_WINDOWEDGE))
$Label1 = GUICtrlCreateLabel("Device to search", 56, 24, 85, 17)
$ibDevice = GUICtrlCreateInput("", 56, 40, 217, 21)
$btSearch = GUICtrlCreateButton("Search", 96, 80, 137, 33, $WS_GROUP)
$Label2 = GUICtrlCreateLabel("Time of last position", 56, 136, 97, 17)
$ibTime = GUICtrlCreateInput("", 56, 152, 217, 21, BitOR($ES_CENTER,$ES_AUTOHSCROLL,$ES_READONLY))
$Label3 = GUICtrlCreateLabel("Lattitude", 56, 184, 45, 17)
$ibLatt = GUICtrlCreateInput("", 56, 200, 97, 21, BitOR($ES_CENTER,$ES_AUTOHSCROLL,$ES_READONLY))
$Label4 = GUICtrlCreateLabel("Longitude", 176, 184, 51, 17)
$ibLong = GUICtrlCreateInput("", 176, 200, 97, 21, BitOR($ES_CENTER,$ES_AUTOHSCROLL,$ES_READONLY))
$btLocate = GUICtrlCreateButton("Locate", 200, 288, 73, 33, $WS_GROUP)
$btExit = GUICtrlCreateButton("Exit", 56, 288, 73, 33, $WS_GROUP)
; label and display box for the new field
$Label5 = GUICtrlCreateLabel("Secret value", 130, 232, 64, 17)
$ibSecret = GUICtrlCreateInput("", 128, 248, 73, 21, BitOR($ES_CENTER,$ES_AUTOHSCROLL,$ES_READONLY))
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

GUICtrlSetState($btLocate, $GUI_DISABLE)


Global Const $datafile = @ScriptDir & "\messages.txt"       ;; change as needed
Global Const $MaxLines = 300                                ;; read up to last $MaxLines lines of above file
Global Const $LineSize = 113                                ;; estimated line size in bytes

Local $device, $devdata

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $btExit, $GUI_EVENT_CLOSE
            Exit
        Case $btSearch
            GUICtrlSetState($btLocate, $GUI_DISABLE)
            GUICtrlSetData($ibTime, "")
            GUICtrlSetData($ibLatt, "")
            GUICtrlSetData($ibLong, "")
            GUICtrlSetData($ibSecret, "")                               ; we need to clean it here, like other fields
            $device = GUICtrlRead($ibDevice)
            $devdata = Lookup($device, $datafile)
            Switch @error
                Case 0
                    Local $devcoord = StringSplit($devdata, '|', 2)     ; this splits the string into an array, using | as separators
                    GUICtrlSetData($device, $devcoord[1])
                    GUICtrlSetData($ibTime, $devcoord[0])
                    GUICtrlSetData($ibLatt, $devcoord[2])
                    GUICtrlSetData($ibLong, $devcoord[3])
                    GUICtrlSetData($ibSecret, $devcoord[4])             ; our new field gets displayed here
                    GUICtrlSetState($btLocate, $GUI_ENABLE)
                Case 1
                    GUICtrlSetData($ibTime, "-- no recent known position --")
                    GUICtrlSetData($ibLatt, "")
                    GUICtrlSetData($ibLong, "")
                    GUICtrlSetData($ibSecret, "")                       ; we clear it here as well
                Case 2
                    MsgBox(16, "Fatal error!", $datafile & " can't be found")
            EndSwitch
        Case $btLocate
            ; invoke Google Map or go play blackjack
    EndSwitch
WEnd


; Since we don't know exactly where to start reading the tail of the file, let's
; assume the format is essentially fixed (or at least well bounded). We can compute
; a good approximation of the file position we need to read from, in order to read
; about $MaxLines of useful data.
;
; A line of the example file is 111 characters, plus 2 (CRLF) = 113 characters.
; assuming UTF-8 or ANSI encoding, that is 113 bytes. If we position the reading point
; few bytes before $MaxLines * 113, and throw away the next line read (probably incomplete)
; we should be able to read about $MaxLines full records.
;
; The above strategy may have to be adapted if any assumption made doesn't hold for some
; reason.
;
; It's better to open the file at every lookup so that we minimize chance of missing
; newly added data by the logging process.
Func Lookup($unit, $filename)
    Local $hdl = FileOpen($filename)
    If $hdl = -1 Then Return(SetError(2, 0, ''))    ; no such file
    Local $pos = FileGetSize($hdl) - $MaxLines * $LineSize - 20
    If $pos > 0 Then
        FileSetPos($hdl, $pos, 0)                   ; set position, starting from begin of file
    EndIf
    FileReadLine($hdl)                              ; throw away until line break
    Local $data = FileRead($hdl)                    ; read to EOF
    FileClose($hdl)
    $data = StringSplit($data, @CRLF, 3)            ; make it an array of lines
    Local $res
    $devcoord = ''
    ; search from bottom up
    For $i = UBound($data) - 1 To 0 Step -1

        ; The regexp is becoming a bit messy and painful to read
        ; let's break it down into individual fields, each including the | separator at its right
        ; and use the example line of your last post as a guideline
        ;
        ; [Mon Apr 12 17:46:18 2010] |AVL1|C119|-112127750|33762840|3500|3500|068000|068000|||||02805000|0|2|1000|4|20||
        ; it's ambiguous which 068 do you want? ______________________________^^^ or ^^^
        ;
        ; so remove ambiguity by changing the example a bit there--------------------vvv
        ; [Mon Apr 12 17:46:18 2010] |AVL1|C119|-112127750|33762840|3500|3500|068000|082000|||||02805000|0|2|1000|4|20||

        $res = StringRegExpReplace($data[$i], "(?i)\[([^\]]+)\]\s*\|" & _               ; [Mon Apr 12 17:46:18 2010] |      captured as $1
                                                    "[^|]*\|" & _                       ; AVL1|                             skipped
                                                    "(" & $unit & ")\|" & _             ; C119|                             captured as $2
                                                    "(-?\d{3})(\d+)\|" & _              ; -112127750|                       captured as $3 and $4
                                                    "(-?\d{2})(\d+)\|" & _              ; 33762840|                         captured as $5 and 6
                                                    "[^|]*\|" & _                       ; 3500|                             skipped
                                                    "[^|]*\|" & _                       ; 3500|                             skipped
                                                    "(\d{3})" & _                       ; 068                               captured as $7
                                                    ".*", _
                                                    "$1\|$2\|$3\.$4\|$5\.$6\|$7")       ; this builds the result string, separating fields with |
        ; with this example, the $res string would be: [Mon Apr 12 17:46:18 2010]|C119|-112.127750|33762840|068

;~      ; if you want the block where the is 082 in the modified example, use this instead
;~      $res = StringRegExpReplace($data[$i], "(?i)\[([^\]]+)\]\s*\|" & _               ; [Mon Apr 12 17:46:18 2010] |      captured as $1
;~                                                  "[^|]*\|" & _                       ; AVL1|                             skipped
;~                                                  "(" & $unit & ")\|" & _             ; C119|                             captured as $2
;~                                                  "(-?\d{3})(\d+)\|" & _              ; -112127750|                       captured as $3 and $4
;~                                                  "(-?\d{2})(\d+)\|" & _              ; 33762840|                         captured as $5 and 6
;~                                                  "[^|]*\|" & _                       ; 3500|                             skipped
;~                                                  "[^|]*\|" & _                       ; 3500|                             skipped
;~                                                  "[^|]*\|" & _                       ; 068000|                           skipped
;~                                                  "(\d{3})" & _                       ; 082                               captured as $7
;~                                                  ".*", _
;~                                                  "$1\|$2\|$3\.$4\|$5\.$6\|$7")       ; this builds the result string, separating fields with |
;~      ; with this example, the $res string would be: [Mon Apr 12 17:46:18 2010]|C119|-112.127750|33762840|082

            If @extended <> 7 Then ContinueLoop     ; we now expect to get all 7 partial matches, not 6 as before
        $devcoord = $res
        Return(SetError(0, 0, $res))                ; return device data found
    Next
    Return(SetError(1, 0, ''))                      ; device not found
EndFunc

I've realized a few details needed to be changed: it's better to require "zero or more characters..." instead of "one or more characters ..." when there is the possibility that a field would appear empty. Zero or more is * in regexp parlance, while one or more is + as you now know

Also I wanted to clarify something. Look, we have input fields separated with | and we find it convenient to separate our output fields (those we have selected and formatted) with the same character. First, we know the since | was used as a separator in the input, there can be no problem using it for output as well. Then you'd say that it's kind of dumb to capture the value inside separators, without capturing the end separator, and only in the replacement part, add again the same separator.

For instance, we capture longitude value. We could as well capture the ending | and add nothing else than the capture in the replacement part. That would be a "nanosecond optimization", i.e. gain nothing.

This is a question of style and caution. The way I coded is in my view clearer and that matters since regexps can become highly tricky very quickly. Then it separates the input format (we have generally no control over) and the output format, we should be able to change at will. For instance, I could as well define a variable $separ = '@' and insert $separ in place of \| in the output string and in StringSplit transparently. Finally, it's a good habit: if you ever practice regexps in more complex stuff and use advanced constructs, then it will make full sense.

Edited April 17, 2010 by jchd