Simple File Splitter

[blademonkey]

Hey all,

Noob King is here again with another noobish script.

Basically we had a IIS log file that was over 1.6 GB that a user needed to read. Every application would crash while trying to read this thing and most splitting utililties would max out at 5 MB output files.

So i wrote this quick and Dirty.

I will be adding a REAL GUI to it later on, when my workload lessens.

anyway, here goes:

#include<file.au3>
$path = "c:\path\"
$file_name1 = "logfile"
$orig_file = $file_name1 & ".txt"
$splitsize = 50000000;50 MB
$x = 0
$y = 1
SplashTextOn("Bla","",-1,-1,-1,-1,16+2)
ProgressOn("Progress1","Currentoutput file progress","",-1,-1,16+2)
While 1
    
    ControlSetText("Bla","","Static1","lines completed : " & $x)
    $x = $x +1 
    $outfile = $file_name1 &"_"& $y &"_split.txt"
    $line = FileReadLine($path & $orig_file,$x)
    FileWriteLine($path&$outfile,$line)
    if @error = -1 Then
        Exitloop
    EndIf
    if  FileGetSize($path& $outfile)> $splitsize Then
    $y = $y +1
    EndIf

    ProgressSet((FileGetSize($path& $outfile)/$splitsize)*100,round((FileGetSize($path& $outfile)/$splitsize)*100,1))
WEnd
Splashoff()
MsgBox(0,"done",$y & " output files")

Any and all feedback is appreciated.

edit: added feedback comment.{= D

Edited February 1, 2006 by blademonkey

[SmOke_N]

Hey all,

Noob King is here again with another noobish script.

Basically we had a IIS log file that was over 1.6 GB that a user needed to read. Every application would crash while trying to read this thing and most splitting utililties would max out at 5 MB output files.

So i wrote this quick and Dirty.

I will be adding a REAL GUI to it later on, when my workload lessens.

anyway, here goes:

#include<file.au3>
$path = "c:\path\"
$file_name1 = "logfile"
$orig_file = $file_name1 & ".txt"
$splitsize = 50000000;50 MB
$x = 0
$y = 1
SplashTextOn("Bla","",-1,-1,-1,-1,16+2)
ProgressOn("Progress1","Currentoutput file progress","",-1,-1,16+2)
While 1
    
    ControlSetText("Bla","","Static1","lines completed : " & $x)
    $x = $x +1 
    $outfile = $file_name1 &"_"& $y &"_split.txt"
    $line = FileReadLine($path & $orig_file,$x)
    FileWriteLine($path&$outfile,$line)
    if @error = -1 Then
        Exitloop
    EndIf
    if  FileGetSize($path& $outfile)> $splitsize Then
    $y = $y +1
    EndIf

    ProgressSet((FileGetSize($path& $outfile)/$splitsize)*100,round((FileGetSize($path& $outfile)/$splitsize)*100,1))
WEnd
Splashoff()
MsgBox(0,"done",$y & " output files")

Any and all feedback is appreciated.

edit: added feedback comment.{= D

Got something to put them back together?

[blademonkey]

Got something to put them back together?

I could write something for it potentially after I make a GUI for it.

Though, I'm looking for a process that is faster than filewriteline/filereadline. it's pretty slow considering its text.

any ideas?

Edited February 1, 2006 by blademonkey

[blademonkey]

Ok I found a way to speed this bad boy up.

time it took to complete operation before change 2 Days.

Time it takes now: Less than 15 Minutes (on a 3.4Ghz PC).

here's the code

#include<file.au3>


$path = "c:\wlcluster2\"
$file_name1 = "wlcluster2" 
$orig_file = $file_name1 & ".txt"
$splitsize = 50000000
$openorig = FileOpen($path & $orig_file,0)
$x = 0
$y = 1
SplashTextOn("Text File Splitter","",-1,-1,-1,-1,16+2)
ProgressOn("Currentoutput file progress","","",-1,-1,16+2)
while 1
    $x = $x +1 
    $outfile = $file_name1 &"_"& $y &"_split.txt"
    
    $line = FileRead($openorig,$x)
    $error= @error
    FileWrite($path&$outfile,$line)
    if $error = -1 Then
        Exitloop
    EndIf
    
    if  FileGetSize($path& $outfile)> $splitsize Then
    $y = $y +1
    EndIf
    ProgressSet((FileGetSize($path& $outfile)/$splitsize)*100,round((FileGetSize($path& $outfile)/$splitsize)*100,1)&"%")
Wend
Splashoff()
MsgBox(0,"done",$y & " output files")

I'm still working on the GUI and I will post the code when it's finalized.

-Blademonkey

[randallc]

Hi,

So you just gave the open file a handle...?

does it help to give the output file a handle too (I can see it may not)

Does it help to use smaller splits (eg 5000 or 50000)?;

Just interested...

Randall

[blademonkey]

Hi,

So you just gave the open file a handle...?

does it help to give the output file a handle too (I can see it may not)

Does it help to use smaller splits (eg 5000 or 50000)?;

Just interested...

Randall

I would think it would, but based on my current code, there was no easy way to say open a file handle for each split file. I ran into some var declaration issues. Haven't had time to work on it as much as I would have liked, I'm restarting the GUI, since I lost my train of thought. >= |

The smaller file chunks didnt matter much. I picked 50MBs because I would it would be a good basis for measuring speed of computation.

Edited February 14, 2006 by blademonkey

[blademonkey]

Alrighty,

I slapped on a little gui, embedded the progress bar and added a few buttons.

let me know what you all think.

Oh and I renamed it to LogSplitter.

It is a Beta (hence the non 1.0 version), so don't expect much.

Compiles with stable or beta.

LogSplitter_0.5g.au3