Sign in to follow this  
Followers 0
flyingboz

Insert Characters In File

11 posts in this topic

What is the preferred methodology these days for Inserting text at an arbitrary location within a file?

I've searched for prepend and insert and filewrite, but haven't found an elegant solution yet.

I want to avoid having to

  • utilize temp files (not elegant, wasteful of resources)
  • open the file more than once (it could be huge / slow )
  • read the entire file into memory (it could be huge / slow )
Anyone using a solution / UDF / technique that they would recommend?

Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites



What is the preferred methodology these days for Inserting text at an arbitrary location within a file?

I've searched for prepend and insert and filewrite, but haven't found an elegant solution yet.

I want to avoid having to

  • utilize temp files (not elegant, wasteful of resources)
  • open the file more than once (it could be huge / slow )
  • read the entire file into memory (it could be huge / slow )
Anyone using a solution / UDF / technique that they would recommend?

You seem to want to avoid all the possible solutions... where does that leave you?

:)


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites

_FileReadToArray() comes to mind but I'm pretty sure that you have disqualified it in one of your conditions. :)


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Unless the insertion does not modify the size of the file (in which case it's a byte replace, not insertion), how do you expect to avoid all of the things you've listed?

The bigger the file, the slower it will be, and you will have to save the end part somehow.

Edited by Siao

"be smart, drink your wine"

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

What is the preferred methodology these days for Inserting text at an arbitrary location within a file?

I've searched for prepend and insert and filewrite, but haven't found an elegant solution yet.

I want to avoid having to

  • utilize temp files (not elegant, wasteful of resources)
  • open the file more than once (it could be huge / slow )
  • read the entire file into memory (it could be huge / slow )
Anyone using a solution / UDF / technique that they would recommend?
Hi,

You will see FileWriteLine funcs in my TailRW UDF from sig link;

"Insert" means you will, at least, have to read the remainder of the file and add it on after the inserted line; at least not the whole file! - not so bad if it is near the tail

But Replace [or Delete line] is sometimes possible;

As Siao says, tricky if you want to replace only the bytes;

There are options there for;

1. Replace with exact trim, so it does not change length of the line; it either pads it or trims it to size;

2. else, as with Insert, you must , at least, have to read the remainder of the file and add it on after the inserted line; at least not the whole file! - not so bad if it is near the tail.

Best, Randall

Edited by randallc

Share this post


Link to post
Share on other sites

You seem to want to avoid all the possible solutions... where does that leave you?

:)

I am hoping that it leaves me with a methodology of putting the pointer for the file at an arbitrary location in the file, and inserting text, regardless of what comes before or after where I choose to place the pointer. This is not rocket science; it is primarily a matter of getting the write api call exposing the proper functionality.

The contents of the file as they already exist don't matter (other than they not be corrupted), therefore, I don't want the expense of having to read /rewrite them. If you deal with files that are huge, this matters.

FileWrite() and FileWriteLine() put the pointer at the end of the file in append mode, an excellent design decision for must purposes; but the fact remains that doing so, as well as choosing not to expose the pointer location to the programmer/scripter are language design decisions, and should not limit the scope of this conversation. <Aside: I greatly respect the design team, and am not insulting their choices in any fashion.>

I guess I could write this up as a feature request for FileWrite() and FileWriteLine() ; an optional third parameter that would enable you to specify the pointer location.

The point of my post was twofold:

(1) to ensure that I had not missed something obvious in my forum search, and

(2) to request helpful commentary regarding methods to accomplish the task.

FileWrite($filehandle or "filename","text",[pointer location])
FileWriteLine($filehandle or "filename","line",[pointer location])

if Pointer Location is omitted or less than zero, use current behavior, else position pointer at byte offset location, 0 = beginning of file.

Won't break scripts and will give logfile functions and other apps that want to prepend file information or insert information into a file arbitrarily the means to do so w/o jumping through hoops.


Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

@randall,

Interesting stuff in the TailRW.au3 -- The file pointer is exposed , and I can arbitrarily overwrite any location in a file nearly instantaneously, I haven't gone over the API functions with a finetooth comb yet, but am hopeful that there is a flag that will turn "overwrite" into "insert" without having to manually keep up with the remainder of the file's contents.

Maybe I'm just being completely wrongheaded about this, but it seems that the filesystem (fat/ntfs/what have you) is responsible for keeping track of what data goes with what files, and in what order, such that the filesystem api's shouldn't have to...

I have seen this functionality referred to as "file seeking" and "rewinding" , moving the pointer to the desired insertion point.

Of course, I've never designed a filesystem from scratch, either.

Edited by flyingboz

Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites

and inserting text, regardless of what comes before or after where I choose to place the pointer. This is not rocket science; it is primarily a matter of getting the write api call exposing the proper functionality.

It's true that file I/O is no rocket science, but that doesn't mean you'll be handed everything on a silver platter either. There is no "insert" mode as far as Windows API is concerned. You SetFilePointer() where you want, and write starting from that point. Preserving what you overwrite is your business.

If you change data without changing filesize (replace), then it's pretty fast and simple regardless of filesize. Map view file, write new data at specified pointer, unmap file.

If you change filesize, then you have to read/rewrite everything from your new data to the end of the file (if you're inserting near the beginning of the file, that means pretty much whole file, and it will be relatively slow), and set the new EOF with SetEndOfFile API (if your insertion extends file, you can do this in the beginning of operation).


"be smart, drink your wine"

Share this post


Link to post
Share on other sites

There is no "insert" mode as far as Windows API is concerned. You SetFilePointer() where you want, and write starting from that point. Preserving what you overwrite is your business.

@Siao, thanks for the interesting and informative post.

If the API doesn't support the functionality directly, I guess we can see how much interest there would be in implementing the functionality by further wrapping the API at the language level as a FRQ.

FileWrite($filehandle,"stuff to write",[$pointer_position],[$insert_mode])

Because, of course, a silver platter is exactly what I'm looking for!


Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites

@Siao, thanks for the interesting and informative post.

If the API doesn't support the functionality directly, I guess we can see how much interest there would be in implementing the functionality by further wrapping the API at the language level as a FRQ.

FileWrite($filehandle,"stuff to write",[$pointer_position],[$insert_mode])oÝ÷ Ù«­¢+Ù}
¡É%¹ÍÉÑ   å  åÑ ÀÌØíÍ}¥±I°ÀÌØíÑáÑ%¹ÍÉÑ°ÀÌØí¥}MÑÉÑ  åÑ°ÀÌØí¥}IÁ±ôÄ°ÀÌØí¥}Q¥°ôÀ°ÀÌØí¥}   ÕôÔÀÀÀÀÀÀÀ¤

Thanks, Randall

Share this post


Link to post
Share on other sites

#11 ·  Posted (edited)

What is "FRQ"?;

Sorry, my software development / presales background rearing its head. FRQ is the TLA (Three Letter Acronym) for Feature ReQuest.

would you handle the insert by some other method than by

1. Reading rest of file at pointer to add after insert ?

Not positive. Presuming that Siao is correct, and there is no method of specifying *insert* in the Windows API (though I have no reason to doubt his veracity and accuracy) , then it or something similar would have to occur. The function calls and helper functions using the API may well be as efficient as we can get in windows. Nonetheless, I would argue that incorporating the functionality within the language enhances the language while minimizing the number of functions that need to be dealt with to handle file i/o.

Presumably (though not demonstrably yet) , the functionality being hidden from the user would be advantageous due to the reduction in the number of variables that would need to be created / populated / read, as well as the ability to utilize the standard filehandles being opened / closed by au3. Assuming the tailrw udf's are coded as efficiently as possible, integrating the operations would still likely be slightly faster. If there are any inefficiencies, the differences would be more dramatic.

I have created the FRQ (Bug Trac 135) and Valik did not reject it out of hand, though he didn't go running in joy over the idea either. While I guess I should be over the inefficiencies of Windows by now, it still is somewhat surprising to me that this isn't happening at the filesystem level. The automated defrag routine should take care of files being spread all over the disk later. A modern filesystem should not make the programmer have to worry about rewriting pieces of data that are already on disk.

edit: added bug trac ticket

Edited by flyingboz

Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0