Jump to content
Sign in to follow this  
Fire

Error: Mem Allocating when using very big log file

Recommended Posts

Fire

Hello, Dear AutoIT Community.

Guys i need help:(

Problem in that i'm writing small log parser which accepts very big apache raw logs.

Raw Logs size: ~ 400~500 MB (wc -l logfile.txt => 34785431 lines)

And i want to $Progress % this process in GUI.

So what i'm doing? Simply for calculation:

-----snippet from code------
$maxline=_FileCountLines($logfile)
Loop goes here & incrementing of $x and some stuff.
GUICtrlSetData($Progress1,Int($x / $maxline * 100))
GUICtrlSetData($Label11,Int($x / $maxline * 100) & " %")
end of loop and some stuff.
-----end of snippet from code-----

I'm getting Error: Error Allocating memory.

Arrays is not best solution in this situation too:(

In fact _FileCountLines() funcsion fails with very big log files.(after 30-40 seconds)

I searched forum too + google too and i found some 3'r party utilities like "wc" for windows.

Also i know i can use cygwin.dll +wc utility for calculation in my program.

Is is possible programmatically calculate that $maxline in faster way and without "Error Allocationg Memory" in Autoit?

Please any solutions?

Thank You Very Much!

Edited by Fire

[size="5"] [/size]

Share this post


Link to post
Share on other sites
Exit

$percent = filepos() / filesize() * 100

Edited by forumer100

My UDFs, Abbrevs and Snippets
If you like my post, just click the like button :) 

Share this post


Link to post
Share on other sites
jchd

This is a memoy shortage, nothing close to a memory leak.

The above post gives you a working code.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
Fire

Thank you very much forumer100 & jchd for your reply.

But for now i'm going completely remove that $progress from my script.

Because i can't handle it in this way.

Also i found another solution:(MS-DOS)

findstr /R /N "^" log.txt | find /C ":"

from:

Count of Lines in file with DOS

This is similar to _FileCountLines($logfile) and will output maximum numbers of lines in file.

It may be usefull i think.But it is very slow in my situation too:( (reason: Very Big log file)

BTW: What a reason for Error Allocationg Memory? In fact _FileCountLines() is unable to parse that large file.

Any limitations for _FileCountLines() ?

Thanks to all.

edit: link

Edited by Fire

[size="5"] [/size]

Share this post


Link to post
Share on other sites
AdmiralAlkex

There's no memory leak, you just ran out of memory.

Lets start by looking at this table:

Memory and Address Space Limits

Now, how much would be needed for _FileCountLines() on a 4-500 MB big file?

I did it on a 477 MB big file and the peak working set (taskmgr >processes>view>select columns) ended on 2.9 GB. A bit more than I expected, but okay.

If you looked at the table earlier as instructed, you know the problem already (trying to use way more than the limit allows).

So... The solution?

  • Get more memory

    • Use the boot-flags to increase the limit to 3 GB (may still not be enough)
    • Install a x64 OS
  • Use a tool dedicated for this (that doesn't load the whole file at the same time)
This is all based on the assumption that you are on a x86 OS, which is one of the many details you conveniently left out.

Share this post


Link to post
Share on other sites
Fire

Great thanks AdmiralManHairAlkex.

Yes i'm using Win XP SP2 32 bit

RAM: 1.5 GB DDR2

Thanks to all again.

I greatly Appreciate helps!


[size="5"] [/size]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×