Sign in to follow this  
Followers 0
Mat

_FileSplitByLine

5 posts in this topic

#1 ·  Posted (edited)

Someone asked on h+s how to split a file by lines, so I wrote this. Since it might come in handy to someone, I decided it belonged here.

It may be buggy, and Its not perfect. But it works, and is decently efficient.

5001 lines:

Splitting took: 201.340216424154 milliseconds

Joining took: 16.5691198183009 milliseconds

100001 lines:

Splitting took: 4944.86980887236 milliseconds

Joining took: 533.011318477628 milliseconds

1000000 Lines: (1 million!)

Splitting took: 50457.6548555752 milliseconds

Joining took: 6317.87354258712 milliseconds

There is still a bug with JoinFromTemplate, it doesn't add the last file. which is bugging me cause I can't see the problem. FromArray has not been tested as much as the other 2, and I haven't timed it.

Code:

Edit: code snipped, use this link: _FileSplitByLine

MDiesel

Edited by mdiesel

Share this post


Link to post
Share on other sites



Hi,

I get error allocating memory when trying to open 628MB (lol, yes I know it's gigantic) text file.

Share this post


Link to post
Share on other sites

And what do you want me to do about it? There is a solution though, i'm working on.

MDiesel

Share this post


Link to post
Share on other sites

Well I was wondering if you were aware of that and since you are working on a solution... :)

Share this post


Link to post
Share on other sites

Well I was wondering if you were aware of that and since you are working on a solution... :)

yer... no problem! This project is very much on a shelf gathering dust, partly due to lack of interest, and also due to me having other stuff that needed doing... then I started a few things... So basically... This isn't going to be on the top of my list for a while.

If you want to - the solution will probably be writing to a file line by line, rather than adding it to a variable. Although considerably slower, and still probably not perfect, it should work :)

MDiesel

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0