How to split a 100 MB binary file?

[Kram3r]

well i don't believe its a limitation of the filesystem, but the tools that we have at our dispose at the current time.

Thats does not meen that some other way as skeep us, i really believe that it was.

Its a very interesting challenge , i hope someone else can put a light on it.

[footswitch]

well i don't believe its a limitation of the filesystem, but the tools that we have at our dispose at the current time.

Thats does not meen that some other way as skeep us, i really believe that it was.

Its a very interesting challenge , i hope someone else can put a light on it.

Kram3r, you're portuguese, didn't notice that earlier good to see another portuguese soul

thanks for your support

obrigado pelo apoio

[Kram3r]

O mundo é pequeno , fico contente por haver outro portugues a saber o que é o AutoIt.

Estou um bocado farto de ouvir Auto quê ?

É bom saber que andamos por ai .

Espero que encontres a solução para o teu desafio, pressinto que vais ter que recorrer a dllcalls.

Translation:

Small world , I'm glad to know that there is another portuguese out there that knows AutoIt.

I'm a little bored to always heard Auto what ?

It's nice to know we are out there .

I hope you find the solution for the chalenge, i feel that you have to use dllcalls.

Sorry to all non-portuguese speakers but the temptation to speak on native language was too high .

We are not many here

Edited September 8, 2007 by Kram3r

[footswitch]

http://student.dei.uc.pt/~luisad/split%20file.swf

[RDSchaefer]

... I'm also NOT looking for things impossible to accomplish...

But you ARE asking for the impossible. There is NO WAY to copy a file, or any part of it, without opening and reading it. Even one of those commercial file-splitting programs HAVE to do it that way.

If you really need to do this and AutoIt is too slow, learn ASM and do it yourself!

Ralph

[PsaltyDS]

EDIT: I still believe that what I wanted to do is possible (although this can sound a little low level):

- discard the first xxx bytes of the file;

(this is where I start to sound stupid; please bear in mind I'm not that familiar with filesystems)

- move (copy?) a portion of the file (until it reaches the next cluster) to the beginning of a free cluster, and set the beginning of the file to that new cluster;

- create a pointer from the new cluster to the rest of the file.

(I know this might not be exactly what happens in reality, but I think it's close )

You could download the HxD disk editor and go crazy on the low level file tables... but you better do it on your "spare" computer... with no data you care about on it...

[footswitch]

LOL i was just messing around... Of course I realised that AutoIt is not the tool for this sort of file 'modification'.

But that's not the point anyway. I just thought it was odd that there is no application that does this for you.

With all those bittorrent and emule stuff, full of hash checkings for 4MB pieces in a 4GB file... When you look at that, say you could 'easily' do anything with what's in your hard disk.

Lol let's say I tend to imagine stuff. Sorry about that Thanks for your tips! Low-level... haha

[footswitch]

But you ARE asking for the impossible. There is NO WAY to copy a file, or any part of it, without opening and reading it. Even one of those commercial file-splitting programs HAVE to do it that way.

If you really need to do this and AutoIt is too slow, learn ASM and do it yourself!

Ralph

Just to make things clear... If you had read my post and take a look at the .swf file, you'd see that I do NOT want to "copy a file". If I wanted to split a file and save BOTH parts of it, then I would need to copy at least one of its parts.

And that's available through randallc's tail read/write UDFs, I think.

But still it makes so much sense to perform it like it appears in the .swf, don't you think? It's, like... logical. IF this is possible, would there be a better way?

And even if I wanted to keep BOTH parts of the splitted file, I'd try to do the whole same thing... Just move (copy) as minimum clusters as possible, one pointer here and there... lol

thanks again

Edited September 13, 2007 by footswitch

[randallc]

Hi,

I don't really have an answer; logically, here are my thoughts;

00.==================================================

I don't know about changing "StartOfFile" address; I imagine that would need modification to the File Table somewhere; if possible at all.

0.==================================================

What sort of data is in the binary?

A. ==================================================

If there is some sort of "null" byte which would be ignored by the filereader program; could you just;

1. remove unwanted data (or ignore it other than its length, so it will be overwritten...).

2. read beginning of file data you want to move (say 100 bytes) ( after any identifying data which would be expected?)

3. write that 100 bytes where you want it to be (say 300 bytes into the file)

4. write null bytes to the intervening data from start of file, after any identifying data which would be expected (?296 nulls)?

I guess it depends what sort of data it is; and I have no idea if this would work!

B. ===========================================

If the binary file is a database, you would need more structure detail, but this would not be necessary, anyway, presumably; what are you doing it for!?

C.========================================

Othewise, the program itself would need some sort of internal pointer system in its file structure; if it has such, then you could change the pointer; again, what sort of binary file is it!?

best, randall

[Buey]

File fragmentation makes what you want to do, the way you want to do it, impossible without directly playing around with the File Allocation Table. Since files may not be contiguous, you can't just add 512 to your initial pointer.

Solution?

DllCall

On what? I don't know.

Edited September 14, 2007 by Buey

[footswitch]

Hi,

I don't really have an answer; logically, here are my thoughts;

00.==================================================

I don't know about changing "StartOfFile" address; I imagine that would need modification to the File Table somewhere; if possible at all.

0.==================================================

What sort of data is in the binary?

A. ==================================================

If there is some sort of "null" byte which would be ignored by the filereader program; could you just;

1. remove unwanted data (or ignore it other than its length, so it will be overwritten...).

2. read beginning of file data you want to move (say 100 bytes) ( after any identifying data which would be expected?)

3. write that 100 bytes where you want it to be (say 300 bytes into the file)

4. write null bytes to the intervening data from start of file, after any identifying data which would be expected (?296 nulls)?

I guess it depends what sort of data it is; and I have no idea if this would work!

B. ===========================================

If the binary file is a database, you would need more structure detail, but this would not be necessary, anyway, presumably; what are you doing it for!?

C.========================================

Othewise, the program itself would need some sort of internal pointer system in its file structure; if it has such, then you could change the pointer; again, what sort of binary file is it!?

best, randall

Hi randallc, thanks for your thoughts.

This problem first occurred to me when I was trying to save a http 1.1 server response to file.

Sometimes the response (and body) comes in binary, and sometimes in ASCII.

The point WAS, whatever the response's format, when I was saving to a file, the file would contain BOTH http 1.1 response header AND body (the supposed file itself).

Now the filereader program can still read the file even if it has a http 1.1 header... but that changes the file's sumcheck (CRC32).

Well the solution was simple as hell, just parse the header BEFORE saving to a file and you're done. There you have your split.

But then I was wondering... If I actually wanted to split a file... And this comes for any purpose, although each purpose could require certain extra actions and a bit of file parsing, but how would I do it?

With time I realised that splitting a file is a bad idea. I mean like "use it if you've got NO OTHER OPTION but to do it". If I knew more about files and how they're stored, I probably would've had thought more about it before starting the topic.

Because normally when people want to split files, it's for backup purposes. And when you're doing a backup, you keep the original.

So basically this topic started with an intent that no longer is required (splitting the http 1.1 header response from the body). And I've reached the same point as you: "File splitting... What do I want it for?"

In my mind this eventually evolved into:

- removing a line from the beginning of a file, assuming that i can:

- add a line to the beginning of a file - off-topic

- changing an amount of bytes in the middle of a file - off-topic

And that's why I think it's time to "close" the topic:

- Personally, RIGHT NOW, I don't REALLY need to split a file with all that "logic" as I was trying to explain in the .swf.

- Because NOW, I can save a 4GB file already without the header, so I'm totally happy

If someone else does need to split a file like this, well... they can always undig the topic at anytime.

And curse our souls for that

Edited September 14, 2007 by footswitch