_FileSplitByLine

[Mat]

Someone asked on h+s how to split a file by lines, so I wrote this. Since it might come in handy to someone, I decided it belonged here.

It may be buggy, and Its not perfect. But it works, and is decently efficient.

5001 lines:

Splitting took: 201.340216424154 milliseconds

Joining took: 16.5691198183009 milliseconds

100001 lines:

Splitting took: 4944.86980887236 milliseconds

Joining took: 533.011318477628 milliseconds

1000000 Lines: (1 million!)

Splitting took: 50457.6548555752 milliseconds

Joining took: 6317.87354258712 milliseconds

There is still a bug with JoinFromTemplate, it doesn't add the last file. which is bugging me cause I can't see the problem. FromArray has not been tested as much as the other 2, and I haven't timed it.

Code:

Edit: code snipped, use this link: _FileSplitByLine

MDiesel

Edited May 26, 2009 by mdiesel

[sensalim]

Hi,

I get error allocating memory when trying to open 628MB (lol, yes I know it's gigantic) text file.

[Mat]

And what do you want me to do about it? There is a solution though, i'm working on.

MDiesel

[sensalim]

Well I was wondering if you were aware of that and since you are working on a solution...

[Mat]

Well I was wondering if you were aware of that and since you are working on a solution...

yer... no problem! This project is very much on a shelf gathering dust, partly due to lack of interest, and also due to me having other stuff that needed doing... then I started a few things... So basically... This isn't going to be on the top of my list for a while.

If you want to - the solution will probably be writing to a file line by line, rather than adding it to a variable. Although considerably slower, and still probably not perfect, it should work

MDiesel