Insert Characters In File

[flyingboz]

What is the preferred methodology these days for Inserting text at an arbitrary location within a file?

I've searched for prepend and insert and filewrite, but haven't found an elegant solution yet.

I want to avoid having to

• utilize temp files (not elegant, wasteful of resources)
• open the file more than once (it could be huge / slow )
• read the entire file into memory (it could be huge / slow )

Anyone using a solution / UDF / technique that they would recommend?

[PsaltyDS]

What is the preferred methodology these days for Inserting text at an arbitrary location within a file?

I've searched for prepend and insert and filewrite, but haven't found an elegant solution yet.

I want to avoid having to

• utilize temp files (not elegant, wasteful of resources)
• open the file more than once (it could be huge / slow )
• read the entire file into memory (it could be huge / slow )

Anyone using a solution / UDF / technique that they would recommend?

You seem to want to avoid all the possible solutions... where does that leave you?

[GEOSoft]

_FileReadToArray() comes to mind but I'm pretty sure that you have disqualified it in one of your conditions.

[Siao]

Unless the insertion does not modify the size of the file (in which case it's a byte replace, not insertion), how do you expect to avoid all of the things you've listed?

The bigger the file, the slower it will be, and you will have to save the end part somehow.

Edited February 22, 2008 by Siao

[randallc]

What is the preferred methodology these days for Inserting text at an arbitrary location within a file?

I've searched for prepend and insert and filewrite, but haven't found an elegant solution yet.

I want to avoid having to

• utilize temp files (not elegant, wasteful of resources)
• open the file more than once (it could be huge / slow )
• read the entire file into memory (it could be huge / slow )

Anyone using a solution / UDF / technique that they would recommend?

Hi,

You will see FileWriteLine funcs in my TailRW UDF from sig link;

"Insert" means you will, at least, have to read the remainder of the file and add it on after the inserted line; at least not the whole file! - not so bad if it is near the tail

But Replace [or Delete line] is sometimes possible;

As Siao says, tricky if you want to replace only the bytes;

There are options there for;

1. Replace with exact trim, so it does not change length of the line; it either pads it or trims it to size;

2. else, as with Insert, you must , at least, have to read the remainder of the file and add it on after the inserted line; at least not the whole file! - not so bad if it is near the tail.

Best, Randall

Edited February 23, 2008 by randallc

[flyingboz]

You seem to want to avoid all the possible solutions... where does that leave you?

I am hoping that it leaves me with a methodology of putting the pointer for the file at an arbitrary location in the file, and inserting text, regardless of what comes before or after where I choose to place the pointer. This is not rocket science; it is primarily a matter of getting the write api call exposing the proper functionality.

The contents of the file as they already exist don't matter (other than they not be corrupted), therefore, I don't want the expense of having to read /rewrite them. If you deal with files that are huge, this matters.

FileWrite() and FileWriteLine() put the pointer at the end of the file in append mode, an excellent design decision for must purposes; but the fact remains that doing so, as well as choosing not to expose the pointer location to the programmer/scripter are language design decisions, and should not limit the scope of this conversation. <Aside: I greatly respect the design team, and am not insulting their choices in any fashion.>

I guess I could write this up as a feature request for FileWrite() and FileWriteLine() ; an optional third parameter that would enable you to specify the pointer location.

The point of my post was twofold:

(1) to ensure that I had not missed something obvious in my forum search, and

(2) to request helpful commentary regarding methods to accomplish the task.

FileWrite($filehandle or "filename","text",[pointer location])
FileWriteLine($filehandle or "filename","line",[pointer location])

if Pointer Location is omitted or less than zero, use current behavior, else position pointer at byte offset location, 0 = beginning of file.

Won't break scripts and will give logfile functions and other apps that want to prepend file information or insert information into a file arbitrarily the means to do so w/o jumping through hoops.

[flyingboz]

@randall,

Interesting stuff in the TailRW.au3 -- The file pointer is exposed , and I can arbitrarily overwrite any location in a file nearly instantaneously, I haven't gone over the API functions with a finetooth comb yet, but am hopeful that there is a flag that will turn "overwrite" into "insert" without having to manually keep up with the remainder of the file's contents.

Maybe I'm just being completely wrongheaded about this, but it seems that the filesystem (fat/ntfs/what have you) is responsible for keeping track of what data goes with what files, and in what order, such that the filesystem api's shouldn't have to...

I have seen this functionality referred to as "file seeking" and "rewinding" , moving the pointer to the desired insertion point.

Of course, I've never designed a filesystem from scratch, either.

Edited February 23, 2008 by flyingboz

[Siao]

and inserting text, regardless of what comes before or after where I choose to place the pointer. This is not rocket science; it is primarily a matter of getting the write api call exposing the proper functionality.

It's true that file I/O is no rocket science, but that doesn't mean you'll be handed everything on a silver platter either. There is no "insert" mode as far as Windows API is concerned. You SetFilePointer() where you want, and write starting from that point. Preserving what you overwrite is your business.

If you change data without changing filesize (replace), then it's pretty fast and simple regardless of filesize. Map view file, write new data at specified pointer, unmap file.

If you change filesize, then you have to read/rewrite everything from your new data to the end of the file (if you're inserting near the beginning of the file, that means pretty much whole file, and it will be relatively slow), and set the new EOF with SetEndOfFile API (if your insertion extends file, you can do this in the beginning of operation).

[flyingboz]

There is no "insert" mode as far as Windows API is concerned. You SetFilePointer() where you want, and write starting from that point. Preserving what you overwrite is your business.

@Siao, thanks for the interesting and informative post.

If the API doesn't support the functionality directly, I guess we can see how much interest there would be in implementing the functionality by further wrapping the API at the language level as a FRQ.

FileWrite($filehandle,"stuff to write",[$pointer_position],[$insert_mode])

Because, of course, a silver platter is exactly what I'm looking for!

[randallc]

@Siao, thanks for the interesting and informative post.

If the API doesn't support the functionality directly, I guess we can see how much interest there would be in implementing the functionality by further wrapping the API at the language level as a FRQ.

FileWrite($filehandle,"stuff to write",[$pointer_position],[$insert_mode])oÝ÷ Ù«­¢+Ù}
¡É%¹ÍÉÑ   å  åÑ ÀÌØíÍ}¥±I°ÀÌØíÑáÑ%¹ÍÉÑ°ÀÌØí¥}MÑÉÑ  åÑ°ÀÌØí¥}IÁ±ôÄ°ÀÌØí¥}Q¥°ôÀ°ÀÌØí¥}   ÕôÔÀÀÀÀÀÀÀ¤

Thanks, Randall

[flyingboz]

What is "FRQ"?;

Sorry, my software development / presales background rearing its head. FRQ is the TLA (Three Letter Acronym) for Feature ReQuest.

would you handle the insert by some other method than by

1. Reading rest of file at pointer to add after insert ?

Not positive. Presuming that Siao is correct, and there is no method of specifying *insert* in the Windows API (though I have no reason to doubt his veracity and accuracy) , then it or something similar would have to occur. The function calls and helper functions using the API may well be as efficient as we can get in windows. Nonetheless, I would argue that incorporating the functionality within the language enhances the language while minimizing the number of functions that need to be dealt with to handle file i/o.

Presumably (though not demonstrably yet) , the functionality being hidden from the user would be advantageous due to the reduction in the number of variables that would need to be created / populated / read, as well as the ability to utilize the standard filehandles being opened / closed by au3. Assuming the tailrw udf's are coded as efficiently as possible, integrating the operations would still likely be slightly faster. If there are any inefficiencies, the differences would be more dramatic.

I have created the FRQ (Bug Trac 135) and Valik did not reject it out of hand, though he didn't go running in joy over the idea either. While I guess I should be over the inefficiencies of Windows by now, it still is somewhat surprising to me that this isn't happening at the filesystem level. The automated defrag routine should take care of files being spread all over the disk later. A modern filesystem should not make the programmer have to worry about rewriting pieces of data that are already on disk.

edit: added bug trac ticket

Edited April 5, 2008 by flyingboz