Sign in to follow this  
Followers 0
SpotCheckBilly

Can I Query a Multidimensional Array?

9 posts in this topic

Hey everyone,

I have what I am assuming is a 5d array as follows:

Column 0 ="X", "L", "O", "?"

Column 1= "CLSID"

Column 2= "Filename"

Column 3= "Description"

Column 4= "Reference Number"

Actually, I have a whole list of arrays of similar construction. The only difference would be one less column (the CLSID). For this particular instance I need to match all rows in which CLSID+Filename=X.

I've been working with Authenticity on a line parser to extract certain information from each line of a log file. For instance in the following line:

O2 - BHO: Windows Live Toolbar Helper - {E15A8DC0-8516-42A1-81EA-DC94EC1ACF10} - C:\Program Files (x86)\Windows Live\Toolbar\wltcore.dll

I am extracting the CLSID (everything between "{" and "}"), and the file name (file path is not important at this point). So the results I'm looking for would be a new array in which all rows have a CLSID and a file name (if present) with an "X" in the first (Column 0). Hope that makes sense.

In all of the help files I only find information on querying 1d or 2d arrays. Is it possible in AutoIt to perform the kind of query I described?

I am still quite the newbie, so he kind. Thanks so very much in advance. -- SCB


[font="Tahoma"]"I was worried 'bout rich and skinny, 'til I wound up poor and fat."-- Delbert McClinton[/font]

Share this post


Link to post
Share on other sites



Hi,

Like Jos said, your data is clearly two-dimensional. You shouldn't have any problem dealing with such using the array functions documented.

BTW, if you have complex queries on large volume data (some logs can get huge), then it would be beneficial to ask yourself if using a lightweight database like SQLite wouldn't be a good idea. Fortunately, AutoIt has a decent SQLite support built-in an UDF and documented in the help.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Thanks for the responses. It would seem that my concept of arrays is way off course. My next question would then be what would constitute an array of more than two dimensions?

@jchd -- first off, I know absolutely nothing about SQL databases, much less how to construct one. Second, if you are familiar with diagnostic logs used to pinpoint malware, currently my project only includes HijackThis, RSIT and DDS. You're absolutely correct, some of the other types of logs can really get huge. I will actually be using a collection of about 20 to 25 smaller databases, each geared towards one section of the log file. That's my current thinking, at least. Depending on how this project goes I may add other logs in the future.

At any rate, it looks like I need to do some more homework before proceeding. Looks like I need to get my line parser perfected first before moving on to the databases.

Thanks again for the responses. I really AM a rookie when it comes to AutoIt and programming in general. This project is pretty big for someone at my experience level but what I'm learning along the way makes it all worthwhile. -- SCB


[font="Tahoma"]"I was worried 'bout rich and skinny, 'til I wound up poor and fat."-- Delbert McClinton[/font]

Share this post


Link to post
Share on other sites

My next question would then be what would constitute an array of more than two dimensions?

Go read the wiki link offered.

If you're maintaining the PCs of some department, you can do so in 2-D: each PC is a row in your array, each hardware and software characteristic is a column.

But at the time you're promoted to maintain the PCs of the company headquarters, then you need to promote your array to 3-D: departments are now rows, individual PCs in a department are now columns and each characteristic comes in a Z-axis, a third dimension.

If ever you're promoted to maintain the PCs of every establishment of that company inside a continent, then ... you don't need PCs arrays anymore because you've got much more important tasks and leave that to your employees.

OTOH, don't believe one can maintain such things with simple arrays!

In short, database engines are tools that allow you to easily store really large collections of data and perform complex queries on it very efficiently without having to devise low-level storage and search strategies by yourself.

I really AM a rookie when it comes to AutoIt and programming in general. This project is pretty big for someone at my experience level but what I'm learning along the way makes it all worthwhile.

A fairly appreciated attitude. If a some point you're stuck with something that escapes you, you'll certainly find help from this forum.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

@AdmiralAlkex -- thanks for the link. You'll have to pardon my ignorance, I didn't even know that AutoIt had a wiki. I'm quite severely vision impaired due to complications from diabetes so it's easy for me to miss things. I have it bookmarked now.

@jchd -- actually, I'm a member of several security sites that help people clean their PCs. Strictly volunteer and there just never are enough of us to keep up with the demand for help. The project I am working on is intended to help me analyze the log files more quickly. Not to mention minimizing the amount of reading that I will have to do. End result being that I will be able to assist more people in the same amount of time.

Thanks again for the info and the link. I'm sure I'll be back with more questions. LOL -- SCB


[font="Tahoma"]"I was worried 'bout rich and skinny, 'til I wound up poor and fat."-- Delbert McClinton[/font]

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

@jchd -- actually, I'm a member of several security sites that help people clean their PCs. Strictly volunteer and there just never are enough of us to keep up with the demand for help. The project I am working on is intended to help me analyze the log files more quickly. Not to mention minimizing the amount of reading that I will have to do. End result being that I will be able to assist more people in the same amount of time.

Good thing to help laymen get rid of nasties. You may want to consider (later) using some kind of database since I believe it could greatly help sorting out fast and reliably legitimate system & software components and unwanted/nasty one.

Ignorance is never a problem, only lack of willingness to cure it is. You're welcome here anytime.

Edit: a sidenote to something I wrote before

But at the time you're promoted to maintain the PCs of the company headquarters, then you need to promote your array to 3-D: departments are now rows, individual PCs in a department are now columns and each characteristic comes in a Z-axis, a third dimension.

You may read this and say to yourself "Eh, I could as well put the owning department as another PC characteristic".

That's true, but I would say it's very bad design. Contrary to serial numbers, hardware and software configuration, the fact that a given PC belongs to a given department can't be infered from the PC examination. In other terms, this characteristic is beyond the PC perimeter. We use to call this "meta-data".

That's why (even if it's possible to agglomerate such characteristic in your old 2-D table), it is best avoided.

Another bad (even worse) design would be to create one 2-D PC table for every department...

My remark isn't just for chating: it can help you make the right design choices for scanning logs from different sources.

If you devise a common "super-format" where you can store log entries from say, AdAware, Hijack-This, ... then you can scan those various logs with one single core engine. Should you have to support another anti-malware tool tomorow, then you just would have to write a new formatting routine to process the new log format and make entries fit you "super-format". But the core engine (pointing out nasties traces) will most likely remain untouched.

If you stick with a naive approach, you'll have to write one separate core engine for each log format. Much harder to maintain!

Of course, this approach only makes sense if all the logs are in the same perimeter and carry similar information under slightly varied formats. I believe that's the case for said programs. But it would be terrible to even think of merging, say, unrelated registry-cleaning information with logs from anti-malware tools: those two things are like apples and shoes.

My 2 cents.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

...You may want to consider (later) using some kind of database since I believe it could greatly help sorting out fast and reliably legitimate system & software components and unwanted/nasty one....

The table I mentioned in my first post is actually created from a section of the database at systemlookup.com. Each section relates to one part of (originally just) a HijackThis log. These lists were originally maintained at one of the premier security sites called CastleCops (now defunct). Since then the lists have been updated to include parts of the DDS and RSIT log files. The lines from all three of these logs are formatted very similar to each other and the line parser I have been working on with Authenticity returns the same results from all three.

That's why (even if it's possible to agglomerate such characteristic in your old 2-D table), it is best avoided.

Another bad (even worse) design would be to create one 2-D PC table for every department...

My remark isn't just for chating: it can help you make the right design choices for scanning logs from different sources.

If you devise a common "super-format" where you can store log entries from say, AdAware, Hijack-This, ... then you can scan those various logs with one single core engine. Should you have to support another anti-malware tool tomorow, then you just would have to write a new formatting routine to process the new log format and make entries fit you "super-format". But the core engine (pointing out nasties traces) will most likely remain untouched.

If you stick with a naive approach, you'll have to write one separate core engine for each log format. Much harder to maintain!

Of course, this approach only makes sense if all the logs are in the same perimeter and carry similar information under slightly varied formats. I believe that's the case for said programs. But it would be terrible to even think of merging, say, unrelated registry-cleaning information with logs from anti-malware tools: those two things are like apples and shoes.

My 2 cents.

Personally, I think you're two cents worth is actually more like a dollars worth. LOLFortunately for me, and my project, working with logs from programs such as Ad-Aware isn't necessary. Neither is maintaining "whitelist" information from the programs I will be working with. Currently the only drawback with these three logs is that they are not compatible with 64-bit OS's. The programs that are compatible format the output a little bit differently so at some point I would have to put together another module to parse them. On the plus side they are, they still use the same database(s) from systemlookup.

I may be off base here but my original intention was to make this program easily updatable by adding or removing modules to deal with changes in the log files e.g. removing those no longer in use, and the never ending changes in malware. -- SCB


[font="Tahoma"]"I was worried 'bout rich and skinny, 'til I wound up poor and fat."-- Delbert McClinton[/font]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0