Pulling information from a running program

[FlashpointBlack]

Hi!

I'm still fairly new to the forums, but I've been using AutoIT off and on for about 3-4 years. I haven't gotten serious about using it until lately though. Primarily, I tend to use it for web scraping using mostly custom-written scripts, but a few well-written UDFs available here. Now, I've gotten a task that's eluded me, and I'm at the end of my rope at how to accomplish it.

The company I used to work for has software that's used to run a huge refinery. While I was working there, I made many custom adjustments to the code there in order to accomidate data logging, automatic controls, etc... Part of the problem has always been that the software running the place is around 20 years old and all of the data logging really bogs down the software (but not the computer itself). Because of memory leaks in the software, the computer must be restarted on a daily basis. While it's restarting, data from that time period is lost along with the ability to control the refinery (everything keeps running as it was, just as if there was nobody watching it... very dangerous). They've since asked me to come up with something different to prevent restarting daily.

My initial thought was to just use AutoIT to directly pull the information from the program itself using memory reading, but I can't find the right pointers to the memory locations.

Ideally, I'd like to use AutoIT to read the information from the software, keep the data that's pulled every 3 seconds in arrays, and every minute write that data to a MySQL database. I can do everything except how exactly to pull the information from the program.

This is all a long-winded explanation. What it comes down to is this:

-While scanning for the memory addresses, I was using CheatEngine 6.3 (I think) because I couldn't find any other software that did something similar. Is there anything you might recommend?

-Can you think of any similar alternative to using direct memory reading?

While scanning the memory, the numbers on the screen jump around by an unpredictable amount. "freeze game while scanning" causes the software to crash, so I'm not even sure that it's properly locating the memory addresses as you would expect it to.

For argument's sake, the software is Wonderware InTouch 7.1, but it uses custom windows that are coded on a per facility basis. The software is $13,000 last time I checked, so I'm afraid I can't really get much more specific. If any users with a good reputation so desire, I'd be happy to let you remote-view the program with teamviewer, but I'm not asking you to go that far, only for advice.

Thanks for reading this absurdly long post!

PS: I should have mentioned that these are not standard windows controls, so the window spy included with AutoIT will not work.

Edited January 25, 2014 by FlashpointBlack

[orbs]

maybe a workaround: can you configure your program to dump logging data to disk, so the memory is cleared?

[FlashpointBlack]

maybe a workaround: can you configure your program to dump logging data to disk, so the memory is cleared?

I'm afraid it already does that. The way the program works is to fill up about 200 variables with data and dump that information into a MS Access database via ODBC every 10 seconds. Next time the 10 second mark comes around, it simply overwrites the information in the 200 variables and repeats.

I'll be completely honest and say that I'm only a hobbist programmer, but I don't see how rewriting the information in the variables would cause a memory leak, so I'd have to believe it's something with the way the information is written to the disk that has the memory leak in it, and unfortunately, I don't have access to those functions since they're hard-coded into the software. Also unfortunate, the language the software provides does not include a means of unassigning a variable to free system resources. the variables that are defined on system startup are the ones that you will have throughout the entire process, so even temporary variables are permanent. It's more than a little bit outdated...

Thank you for your reply!

[MilesAhead]

If it can't run for 24 hours without a reboot then my feeling is the 200 variables is likely the memory leak.  I would bet the "overwrite" actually creates new instances and abandons any reference to the previous set of instances.

I take it the company doesn't want to pay the original developers for maintenance?

My thought would be to change the info dump to every 15 seconds or some similar interval and see how long it can run without crashing.  If there's a direct relationship then you found the leak.

[MilesAhead]

The other thing is, you may want to check out the licensing for this software.  If there's an anti-reverse engineering clause you may be left bag holding if it hits the fan.

[DW1]

Your best bet is to open a support ticket with the developer of the software experiencing a memory leak.  Provide a dump from ProcDump to them while the slow down issue is happening.

Have you tried calling EmptyWorkingSet on the process experiencing the leak to see if it changes the frequency at which it gets bogged down?

Func _ReduceMemory($i_PID = -1)
    
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1f0fff, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    
    Return $ai_Return[0]
EndFunc;==> _ReduceMemory()

[MilesAhead]

If whatever it is written in has automatic garbage collection the 10 second cycle likely overloads it.  Chances are it wasn't designed for such a short turn-around.

[FlashpointBlack]

I take it the company doesn't want to pay the original developers for maintenance?

[...snip...]

The other thing is, you may want to check out the licensing for this software.  If there's an anti-reverse engineering clause you may be left bag holding if it hits the fan.

 

When the programmers came in, they said that data logging wouldn't work without purchasing a costly upgrade. They must not have known what they were talking about, because even though it has some issues, it does function. Data logging is a REQUIREMENT for the company in order to impliment some sort of QC checks for the end product. Previous to me adding in the code that logs to an access database, the company relied on hand-written numbers on printed spreadsheets that people just pulled out and looked at to see if they noticed a trend off the top of their head. Nobody saw this as problematic for some reason. Anyhow, long story short, the company didn't pay them. The company has a history of that.

(Edit: They were right not to pay them. I'm an novice at best, and the code they wrote was horrible. My guess is that they did the best they could do with a language that hadn't been updated or taught for 5 years, but it wound up being a huge failure, and I had to fix many of their mistakes)

Thankfully, even if the system did have such a thing as an anti-reverse engineering clause, it wouldn't apply to the situation that I'm in. The program is actually a development platform which you can use to write your own programs to control equipment through a programmed logic controller. The part I edited was part of what was "paid" for by my company by independant programmers 15 years ago. (yes. the software was outdated when they implimented it)

 

Your best bet is to open a support ticket with the developer of the software experiencing a memory leak.  Provide a dump from ProcDump to them while the slow down issue is happening.

Have you tried calling EmptyWorkingSet on the process experiencing the leak to see if it changes the frequency at which it gets bogged down?

 

Unfortunately, support ended for the version that we currently have ages ago, and the only alternative they gave us when I asked them about the problem was to upgrade. Upgrading comes at it's own cost... a (graciously) reduced price to ~$8,000 along with the facility being down while I try to muddle my way through it... or hire professionals for god knows how much money, all of which they can't afford.

I haven't tried that solution, and I've never heard of it. over the next few days, I'll do that and see how it turns out.

Something I didn't mention though... It seems like the amount of variables being written doesn't seem to affect it one way or another. When I was playing around with it trying to get it to work, I started off small and had it log only a few ~5-10 variables at a time and the problem was immediately evident then, with the same ~24 hour period before it started to slow down. By slow down, I mean writes to the database begin to take longer and longer and all the numbers on the screen stop moving while the write is taking place, with the operator unable to do anything during that time. That leads me to believe that it's something with how it's using ODBC and not the variables themselves that cause the problem.

Another example of how variables work in the program: the numbers that are being displayed on the screen are using pressure and temperature probes throughout the facility to constantly update real-time data displayed to the user through the same type of variable I'm using to temporarily "capture" the data and send it to ODBC. these variables that are being displayed on the screen can update as much as 2-3 times per second, so I find it hard to believe it's the variables themselves which are causing the slowdown. The slowdown only occurs when connecting to one of their inbuilt ODBC functions which is routed to a database.

The program connects to the ODBC database, sends (basically) SQL text containing columns headings and numbers, then the database is disconnected.

I know this is a project for someone high above my calibre. And I know that the perfect case scenario would be to have the company update the program to the most up-to-date version, update the source code for our "program in a program", and start pushing it into a different type of database, but none of those are going to happen. My goal isn't to try to figure out why the program is doing what it's doing, but to maintain functionality of something that I consider to be essential for this sort of business (data logging) and to maintain control of the facility without having to restart every day by preferably moving the functionality into a versitile programming language that I'm familiar with (AutoIT) to hopefully open the opportunity to not just maintain the standards that are set by logging the data, but to expand upon them as well.

Edited January 25, 2014 by FlashpointBlack

[Inverted]

1) Maybe you should try updating the ODBC drivers? Or changing the configuration ? Or updating MS Access ?

2) I also suggest you also use Process Hacker to monitor the applications running and check when and why the degradation begins. Check out if CPU usage is spiking and whether some application's memory usage keeps increasing (mem leaking)

3) Anyway if you resort to memory reading with AutoIt, it can be done and Cheat Engine should allow you to find the addresses with consecutive searches. If the numbers change too fast you can use Process Hacker to suspend the target program while searching.

I hope you have some backups in case things so south 

[MilesAhead]

Perhaps the data logging side effects are to ensure it "won't work without a costly upgrade." Your idea may be the only feasible one in that case.

[FlashpointBlack]

2) I also suggest you also use Process Hacker to monitor the applications running and check when and why the degradation begins. Check out if CPU usage is spiking and whether some application's memory usage keeps increasing (mem leaking)

3) Anyway if you resort to memory reading with AutoIt, it can be done and Cheat Engine should allow you to find the addresses with consecutive searches. If the numbers change too fast you can use Process Hacker to suspend the target program while searching.

I hope you have some backups in case things so south 

 

Thanks! Very good suggestion, using Process Hacker. I've used it before, but I'd completely forgotten about it. and I didn't know it could suspend applications as well. that certainly helps!

Backups are a must, and it's something I've been pretty vigilant about. Every day I work on it, I start by making a backup up the current build. I learned my lesson the second time I had to update the program from the "default" installed version that was originally put in place. yes, second time. I do l earn from my mistakes, sometimes it just takes more than once!

Perhaps the data logging side effects are to ensure it "won't work without a costly upgrade." Your idea may be the only feasible one in that case.

 

That's actually something I hadn't considered before, and it seems like it might be a possibility. Considering the need for professional programmers to come in and program a custom solution each time, it seems unlikely that none of them would have thought to do what I did. I'm getting information about those programmers from a years-long game of telephone from people who heard it from other people who got fired, etc... so who knows what truth is any more, but this is a real possibility I hadn't considered. thanks!

[orbs]

as much as i agree that properly this should be resolved by the product support, i do sympathize with your case. one thing i may have missed in your description: is the memory leak resolved after program quit or only after computer restart?

instead of restart, try quit the program, perform memory cleaning (there are some freeware that do it rather well), then start the program again and check the performance.

one more thought: if the database writing is what hogs you, consider is the Access database becoming too big over time? (by too big i mean over 2000 records)

within the 10s interval between writes, try to rename the database file. that (hopefully) will make the program create a new, empty database for the next writes.

[FlashpointBlack]

as much as i agree that properly this should be resolved by the product support, i do sympathize with your case. one thing i may have missed in your description: is the memory leak resolved after program quit or only after computer restart?

instead of restart, try quit the program, perform memory cleaning (there are some freeware that do it rather well), then start the program again and check the performance.

one more thought: if the database writing is what hogs you, consider is the Access database becoming too big over time? (by too big i mean over 2000 records)

within the 10s interval between writes, try to rename the database file. that (hopefully) will make the program create a new, empty database for the next writes.

 

To be completely honest, I actually don't know if the computer needs to be restarted or if the application needs to be restarted. We've always just restarted it since there are about 5-6 different programs that all function together as a single unit with one program being the "output" of their combined effort and it's a pain to manually shut them all down and restart. Next time it happens though, I will manually shut them all down and restart them without rebooting the computer and get back to you.

Edited January 27, 2014 by FlashpointBlack