Sign in to follow this  
Followers 0
Kayage

Splitting File to a few files

4 posts in this topic

Dear script gurus,

I have a text file as big as 1GB++ (too big to be opened by notepad and would consume too much memory if opened up under wordpad. What I would like to do now, is to write a scrip to break that big file into a few text files (eg. to split that 1GB file to 4 250MB files or even more) for it to be managable.

I am new in AutoIT scripting and from what I read (the help function and sampels of scripts posted), I can either choose to read the 1GB file, line by line and write it to another file, setting criteria to save, close and open up new file to populate, thus breaking the file down to a few chunks; or

Read the whole file into an array and then break it out into mane smaller files.

Not too sure the latter method would work as my PC runs on 2GB of RAM (and from experience, the last time I attempted to open up the 1GB file, my RAM utilisation went up to 2.5GB!! almost crashing my OS!) Where else, the former method, could take ages to complete.

Is there another method to work on this?... pls advise. thank you.

Share this post


Link to post
Share on other sites



$f = Fileopen('file')
Do
FileRead($f,4096);4096 means read 4096 characters(or bytes in binary mode) each time
Until @error
FileClose($f)

So if

file.txt <-- "This is a string."

$f = Fileopen('file.txt')
$read = FileRead($f,5);$read = "This "
$read = FileRead($f,5);$read = "is a "
$read = FileRead($f,5);$read = "strin"
$read = FileRead($f,5);$read = "g."
$read = FileRead($f,5);<---- This will produce an @error,bec. it's end of file.
FileClose($f)

Share this post


Link to post
Share on other sites

Dear script gurus,

I have a text file as big as 1GB++ (too big to be opened by notepad and would consume too much memory if opened up under wordpad. What I would like to do now, is to write a scrip to break that big file into a few text files (eg. to split that 1GB file to 4 250MB files or even more) for it to be managable.

I am new in AutoIT scripting and from what I read (the help function and sampels of scripts posted), I can either choose to read the 1GB file, line by line and write it to another file, setting criteria to save, close and open up new file to populate, thus breaking the file down to a few chunks; or

Read the whole file into an array and then break it out into mane smaller files.

Not too sure the latter method would work as my PC runs on 2GB of RAM (and from experience, the last time I attempted to open up the 1GB file, my RAM utilisation went up to 2.5GB!! almost crashing my OS!) Where else, the former method, could take ages to complete.

Is there another method to work on this?... pls advise. thank you.

You could just copy a chunk at a time rather than read a line at a time or read the whole file to an array.

#include <winapi.au3>
Global $nbytes = 0, $chunk = 2000;2^28 for 250Mb
$tBuffer = DllStructCreate("byte[" & $chunk & "]")
$sFile = @ScriptFullPath
$hFile = _WinAPI_CreateFile ($sFile, 2, 2)
$size = _WinAPI_GetFileSizeEx ($hFile)

_WinAPI_SetFilePointer ($hFile, 0)
Global $Index = 1
While 1
    _WinAPI_ReadFile ($hFile, DllStructGetPtr($tBuffer), $chunk, $nbytes)
    
    If $nbytes = 0 Then ExitLoop
    ConsoleWrite("Read " & $nbytes & @CRLF)
    $tBuffer2 = DllStructCreate("byte[" & $nbytes & "]", DllStructGetPtr($tBuffer))
    $hF = FileOpen("File" & $Index & ".doc", 2)
    FileWrite($hF, DllStructGetData($tBuffer2, 1))
    FileClose($hF)
    $Index += 1
WEnd
_WinAPI_CloseHandle ($hFile)

Serial port communications UDF Includes functions for binary transmission and reception.printing UDF Useful for graphs, forms, labels, reports etc.Add User Call Tips to SciTE for functions in UDFs not included with AutoIt and for your own scripts.Functions with parameters in OnEvent mode and for Hot Keys One function replaces GuiSetOnEvent, GuiCtrlSetOnEvent and HotKeySet.UDF IsConnected2 for notification of status of connected state of many urls or IPs, without slowing the script.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0