Error: Mem Allocating when using very big log file

[Fire]

Hello, Dear AutoIT Community.

Guys i need help:(

Problem in that i'm writing small log parser which accepts very big apache raw logs.

Raw Logs size: ~ 400~500 MB (wc -l logfile.txt => 34785431 lines)

And i want to $Progress % this process in GUI.

So what i'm doing? Simply for calculation:

-----snippet from code------
$maxline=_FileCountLines($logfile)
Loop goes here & incrementing of $x and some stuff.
GUICtrlSetData($Progress1,Int($x / $maxline * 100))
GUICtrlSetData($Label11,Int($x / $maxline * 100) & " %")
end of loop and some stuff.
-----end of snippet from code-----

I'm getting Error: Error Allocating memory.

Arrays is not best solution in this situation too:(

In fact _FileCountLines() funcsion fails with very big log files.(after 30-40 seconds)

I searched forum too + google too and i found some 3'r party utilities like "wc" for windows.

Also i know i can use cygwin.dll +wc utility for calculation in my program.

Is is possible programmatically calculate that $maxline in faster way and without "Error Allocationg Memory" in Autoit?

Please any solutions?

Thank You Very Much!

Edited July 13, 2011 by Fire

[Exit]

$percent = filepos() / filesize() * 100

Edited July 13, 2011 by forumer100

[jchd]

This is a memoy shortage, nothing close to a memory leak.

The above post gives you a working code.

[Fire]

Thank you very much forumer100 & jchd for your reply.

But for now i'm going completely remove that $progress from my script.

Because i can't handle it in this way.

Also i found another solution:(MS-DOS)

findstr /R /N "^" log.txt | find /C ":"

from:

Count of Lines in file with DOS

This is similar to _FileCountLines($logfile) and will output maximum numbers of lines in file.

It may be usefull i think.But it is very slow in my situation too:( (reason: Very Big log file)

BTW: What a reason for Error Allocationg Memory? In fact _FileCountLines() is unable to parse that large file.

Any limitations for _FileCountLines() ?

Thanks to all.

edit: link

Edited July 13, 2011 by Fire

[AdmiralAlkex]

There's no memory leak, you just ran out of memory.

Lets start by looking at this table:

Memory and Address Space Limits

Now, how much would be needed for _FileCountLines() on a 4-500 MB big file?

I did it on a 477 MB big file and the peak working set (taskmgr >processes>view>select columns) ended on 2.9 GB. A bit more than I expected, but okay.

If you looked at the table earlier as instructed, you know the problem already (trying to use way more than the limit allows).

So... The solution?

• Get more memory
  • Use the boot-flags to increase the limit to 3 GB (may still not be enough)
  • Install a x64 OS
• Use a tool dedicated for this (that doesn't load the whole file at the same time)

This is all based on the assumption that you are on a x86 OS, which is one of the many details you conveniently left out.

[Fire]

Great thanks AdmiralManHairAlkex.

Yes i'm using Win XP SP2 32 bit

RAM: 1.5 GB DDR2

Thanks to all again.

I greatly Appreciate helps!