Can I Query a Multidimensional Array?

[SpotCheckBilly]

Hey everyone,

I have what I am assuming is a 5d array as follows:

Column 0 ="X", "L", "O", "?"

Column 1= "CLSID"

Column 2= "Filename"

Column 3= "Description"

Column 4= "Reference Number"

Actually, I have a whole list of arrays of similar construction. The only difference would be one less column (the CLSID). For this particular instance I need to match all rows in which CLSID+Filename=X.

I've been working with Authenticity on a line parser to extract certain information from each line of a log file. For instance in the following line:

O2 - BHO: Windows Live Toolbar Helper - {E15A8DC0-8516-42A1-81EA-DC94EC1ACF10} - C:\Program Files (x86)\Windows Live\Toolbar\wltcore.dll

I am extracting the CLSID (everything between "{" and "}"), and the file name (file path is not important at this point). So the results I'm looking for would be a new array in which all rows have a CLSID and a file name (if present) with an "X" in the first (Column 0). Hope that makes sense.

In all of the help files I only find information on querying 1d or 2d arrays. Is it possible in AutoIt to perform the kind of query I described?

I am still quite the newbie, so he kind. Thanks so very much in advance. -- SCB

[Jos]

I would call this a 2 dimensional array as you have Rows and Columns.

[jchd]

Hi,

Like Jos said, your data is clearly two-dimensional. You shouldn't have any problem dealing with such using the array functions documented.

BTW, if you have complex queries on large volume data (some logs can get huge), then it would be beneficial to ask yourself if using a lightweight database like SQLite wouldn't be a good idea. Fortunately, AutoIt has a decent SQLite support built-in an UDF and documented in the help.

[SpotCheckBilly]

Thanks for the responses. It would seem that my concept of arrays is way off course. My next question would then be what would constitute an array of more than two dimensions?

@jchd -- first off, I know absolutely nothing about SQL databases, much less how to construct one. Second, if you are familiar with diagnostic logs used to pinpoint malware, currently my project only includes HijackThis, RSIT and DDS. You're absolutely correct, some of the other types of logs can really get huge. I will actually be using a collection of about 20 to 25 smaller databases, each geared towards one section of the log file. That's my current thinking, at least. Depending on how this project goes I may add other logs in the future.

At any rate, it looks like I need to do some more homework before proceeding. Looks like I need to get my line parser perfected first before moving on to the databases.

Thanks again for the responses. I really AM a rookie when it comes to AutoIt and programming in general. This project is pretty big for someone at my experience level but what I'm learning along the way makes it all worthwhile. -- SCB

[AdmiralAlkex]

There is a really good tutorial on arrays on the wiki, link

[jchd]

My next question would then be what would constitute an array of more than two dimensions?

Go read the wiki link offered.

If you're maintaining the PCs of some department, you can do so in 2-D: each PC is a row in your array, each hardware and software characteristic is a column.

But at the time you're promoted to maintain the PCs of the company headquarters, then you need to promote your array to 3-D: departments are now rows, individual PCs in a department are now columns and each characteristic comes in a Z-axis, a third dimension.

If ever you're promoted to maintain the PCs of every establishment of that company inside a continent, then ... you don't need PCs arrays anymore because you've got much more important tasks and leave that to your employees.

OTOH, don't believe one can maintain such things with simple arrays!

In short, database engines are tools that allow you to easily store really large collections of data and perform complex queries on it very efficiently without having to devise low-level storage and search strategies by yourself.

I really AM a rookie when it comes to AutoIt and programming in general. This project is pretty big for someone at my experience level but what I'm learning along the way makes it all worthwhile.

A fairly appreciated attitude. If a some point you're stuck with something that escapes you, you'll certainly find help from this forum.

[SpotCheckBilly]

@AdmiralAlkex -- thanks for the link. You'll have to pardon my ignorance, I didn't even know that AutoIt had a wiki. I'm quite severely vision impaired due to complications from diabetes so it's easy for me to miss things. I have it bookmarked now.

@jchd -- actually, I'm a member of several security sites that help people clean their PCs. Strictly volunteer and there just never are enough of us to keep up with the demand for help. The project I am working on is intended to help me analyze the log files more quickly. Not to mention minimizing the amount of reading that I will have to do. End result being that I will be able to assist more people in the same amount of time.

Thanks again for the info and the link. I'm sure I'll be back with more questions. LOL -- SCB

[jchd]

@jchd -- actually, I'm a member of several security sites that help people clean their PCs. Strictly volunteer and there just never are enough of us to keep up with the demand for help. The project I am working on is intended to help me analyze the log files more quickly. Not to mention minimizing the amount of reading that I will have to do. End result being that I will be able to assist more people in the same amount of time.

Good thing to help laymen get rid of nasties. You may want to consider (later) using some kind of database since I believe it could greatly help sorting out fast and reliably legitimate system & software components and unwanted/nasty one.

Ignorance is never a problem, only lack of willingness to cure it is. You're welcome here anytime.

Edit: a sidenote to something I wrote before

But at the time you're promoted to maintain the PCs of the company headquarters, then you need to promote your array to 3-D: departments are now rows, individual PCs in a department are now columns and each characteristic comes in a Z-axis, a third dimension.

You may read this and say to yourself "Eh, I could as well put the owning department as another PC characteristic".

That's true, but I would say it's very bad design. Contrary to serial numbers, hardware and software configuration, the fact that a given PC belongs to a given department can't be infered from the PC examination. In other terms, this characteristic is beyond the PC perimeter. We use to call this "meta-data".

That's why (even if it's possible to agglomerate such characteristic in your old 2-D table), it is best avoided.

Another bad (even worse) design would be to create one 2-D PC table for every department...

My remark isn't just for chating: it can help you make the right design choices for scanning logs from different sources.

If you devise a common "super-format" where you can store log entries from say, AdAware, Hijack-This, ... then you can scan those various logs with one single core engine. Should you have to support another anti-malware tool tomorow, then you just would have to write a new formatting routine to process the new log format and make entries fit you "super-format". But the core engine (pointing out nasties traces) will most likely remain untouched.

If you stick with a naive approach, you'll have to write one separate core engine for each log format. Much harder to maintain!

Of course, this approach only makes sense if all the logs are in the same perimeter and carry similar information under slightly varied formats. I believe that's the case for said programs. But it would be terrible to even think of merging, say, unrelated registry-cleaning information with logs from anti-malware tools: those two things are like apples and shoes.

My 2 cents.

Edited January 12, 2010 by jchd

[SpotCheckBilly]

...You may want to consider (later) using some kind of database since I believe it could greatly help sorting out fast and reliably legitimate system & software components and unwanted/nasty one....

The table I mentioned in my first post is actually created from a section of the database at systemlookup.com. Each section relates to one part of (originally just) a HijackThis log. These lists were originally maintained at one of the premier security sites called CastleCops (now defunct). Since then the lists have been updated to include parts of the DDS and RSIT log files. The lines from all three of these logs are formatted very similar to each other and the line parser I have been working on with Authenticity returns the same results from all three.

That's why (even if it's possible to agglomerate such characteristic in your old 2-D table), it is best avoided.

Another bad (even worse) design would be to create one 2-D PC table for every department...

My remark isn't just for chating: it can help you make the right design choices for scanning logs from different sources.

If you devise a common "super-format" where you can store log entries from say, AdAware, Hijack-This, ... then you can scan those various logs with one single core engine. Should you have to support another anti-malware tool tomorow, then you just would have to write a new formatting routine to process the new log format and make entries fit you "super-format". But the core engine (pointing out nasties traces) will most likely remain untouched.

If you stick with a naive approach, you'll have to write one separate core engine for each log format. Much harder to maintain!

Of course, this approach only makes sense if all the logs are in the same perimeter and carry similar information under slightly varied formats. I believe that's the case for said programs. But it would be terrible to even think of merging, say, unrelated registry-cleaning information with logs from anti-malware tools: those two things are like apples and shoes.

My 2 cents.

Personally, I think you're two cents worth is actually more like a dollars worth. LOLFortunately for me, and my project, working with logs from programs such as Ad-Aware isn't necessary. Neither is maintaining "whitelist" information from the programs I will be working with. Currently the only drawback with these three logs is that they are not compatible with 64-bit OS's. The programs that are compatible format the output a little bit differently so at some point I would have to put together another module to parse them. On the plus side they are, they still use the same database(s) from systemlookup.

I may be off base here but my original intention was to make this program easily updatable by adding or removing modules to deal with changes in the log files e.g. removing those no longer in use, and the never ending changes in malware. -- SCB