Sign in to follow this  
Followers 0
blademonkey

Simple File Splitter

7 posts in this topic

#1 ·  Posted (edited)

Hey all,

Noob King is here again with another noobish script.

Basically we had a IIS log file that was over 1.6 GB that a user needed to read. Every application would crash while trying to read this thing and most splitting utililties would max out at 5 MB output files.

So i wrote this quick and Dirty.

I will be adding a REAL GUI to it later on, when my workload lessens.

anyway, here goes:

#include<file.au3>
$path = "c:\path\"
$file_name1 = "logfile"
$orig_file = $file_name1 & ".txt"
$splitsize = 50000000;50 MB
$x = 0
$y = 1
SplashTextOn("Bla","",-1,-1,-1,-1,16+2)
ProgressOn("Progress1","Currentoutput file progress","",-1,-1,16+2)
While 1
    
    ControlSetText("Bla","","Static1","lines completed : " & $x)
    $x = $x +1 
    $outfile = $file_name1 &"_"& $y &"_split.txt"
    $line = FileReadLine($path & $orig_file,$x)
    FileWriteLine($path&$outfile,$line)
    if @error = -1 Then
        Exitloop
    EndIf
    if  FileGetSize($path& $outfile)> $splitsize Then
    $y = $y +1
    EndIf

    ProgressSet((FileGetSize($path& $outfile)/$splitsize)*100,round((FileGetSize($path& $outfile)/$splitsize)*100,1))
WEnd
Splashoff()
MsgBox(0,"done",$y & " output files")

Any and all feedback is appreciated.

edit: added feedback comment.{= D

Edited by blademonkey

---"Educate the Mind, Make Savage the Body" -Mao Tse Tung

Share this post


Link to post
Share on other sites



Hey all,

Noob King is here again with another noobish script.

Basically we had a IIS log file that was over 1.6 GB that a user needed to read. Every application would crash while trying to read this thing and most splitting utililties would max out at 5 MB output files.

So i wrote this quick and Dirty.

I will be adding a REAL GUI to it later on, when my workload lessens.

anyway, here goes:

#include<file.au3>
$path = "c:\path\"
$file_name1 = "logfile"
$orig_file = $file_name1 & ".txt"
$splitsize = 50000000;50 MB
$x = 0
$y = 1
SplashTextOn("Bla","",-1,-1,-1,-1,16+2)
ProgressOn("Progress1","Currentoutput file progress","",-1,-1,16+2)
While 1
    
    ControlSetText("Bla","","Static1","lines completed : " & $x)
    $x = $x +1 
    $outfile = $file_name1 &"_"& $y &"_split.txt"
    $line = FileReadLine($path & $orig_file,$x)
    FileWriteLine($path&$outfile,$line)
    if @error = -1 Then
        Exitloop
    EndIf
    if  FileGetSize($path& $outfile)> $splitsize Then
    $y = $y +1
    EndIf

    ProgressSet((FileGetSize($path& $outfile)/$splitsize)*100,round((FileGetSize($path& $outfile)/$splitsize)*100,1))
WEnd
Splashoff()
MsgBox(0,"done",$y & " output files")

Any and all feedback is appreciated.

edit: added feedback comment.{= D

Got something to put them back together? :lmao:

[center]Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.[/center]

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Got something to put them back together? :lmao:

I could write something for it potentially after I make a GUI for it.

Though, I'm looking for a process that is faster than filewriteline/filereadline. it's pretty slow considering its text.

any ideas?

Edited by blademonkey

---"Educate the Mind, Make Savage the Body" -Mao Tse Tung

Share this post


Link to post
Share on other sites

Ok I found a way to speed this bad boy up.

time it took to complete operation before change 2 Days.

Time it takes now: Less than 15 Minutes (on a 3.4Ghz PC).

here's the code

#include<file.au3>


$path = "c:\wlcluster2\"
$file_name1 = "wlcluster2" 
$orig_file = $file_name1 & ".txt"
$splitsize = 50000000
$openorig = FileOpen($path & $orig_file,0)
$x = 0
$y = 1
SplashTextOn("Text File Splitter","",-1,-1,-1,-1,16+2)
ProgressOn("Currentoutput file progress","","",-1,-1,16+2)
while 1
    $x = $x +1 
    $outfile = $file_name1 &"_"& $y &"_split.txt"
    
    $line = FileRead($openorig,$x)
    $error= @error
    FileWrite($path&$outfile,$line)
    if $error = -1 Then
        Exitloop
    EndIf
    
    if  FileGetSize($path& $outfile)> $splitsize Then
    $y = $y +1
    EndIf
    ProgressSet((FileGetSize($path& $outfile)/$splitsize)*100,round((FileGetSize($path& $outfile)/$splitsize)*100,1)&"%")
Wend
Splashoff()
MsgBox(0,"done",$y & " output files")

I'm still working on the GUI and I will post the code when it's finalized.

-Blademonkey


---"Educate the Mind, Make Savage the Body" -Mao Tse Tung

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

Hi,

So you just gave the open file a handle...?

does it help to give the output file a handle too (I can see it may not)

Does it help to use smaller splits (eg 5000 or 50000)?;

Just interested...

Randall

I would think it would, but based on my current code, there was no easy way to say open a file handle for each split file. I ran into some var declaration issues. Haven't had time to work on it as much as I would have liked, I'm restarting the GUI, since I lost my train of thought. >= |

The smaller file chunks didnt matter much. I picked 50MBs because I would it would be a good basis for measuring speed of computation.

Edited by blademonkey

---"Educate the Mind, Make Savage the Body" -Mao Tse Tung

Share this post


Link to post
Share on other sites

Alrighty,

I slapped on a little gui, embedded the progress bar and added a few buttons.

let me know what you all think.

Oh and I renamed it to LogSplitter.

It is a Beta (hence the non 1.0 version), so don't expect much.

Compiles with stable or beta.

:o

LogSplitter_0.5g.au3


---"Educate the Mind, Make Savage the Body" -Mao Tse Tung

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0