Fossil Rock

Writing to the Beginning of a file.

18 posts in this topic

I'm not sure if this is possible but I want to write a short piece of code to the beginning of a file as opposed to appending it to the end (same concept opposite end).

It's safe to assume that the target file can be any size so file size may have an impact on performance.

Any suggestions ?


Agreement is not necessary - thinking for one's self is!

My-Colors.jpg

cuniform2.gif

Share this post


Link to post
Share on other sites



Write your code to file B, copy existing file A plus B to new file C, erase A and B, rename C to A.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

Thanks water. Considering this may be done 100's or even thousands of times (on the same file), is there no more direct method to just 'inject' the code to the beginning?

If I'm working with a file that could potentially be in the 100's of MB's (even GB's) then this method would take a very, very long time.


Agreement is not necessary - thinking for one's self is!

My-Colors.jpg

cuniform2.gif

Share this post


Link to post
Share on other sites

Can you please elaborate why you need to write lines at the top of such large files? Maybe there is a better solution than moving GB's of data :)
What do you mean by "code"?


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

It's just an experiment I'm testing, but if I can get it streamlined to be as fast as possible then it may have some useful purposes down the road.


Agreement is not necessary - thinking for one's self is!

My-Colors.jpg

cuniform2.gif

Share this post


Link to post
Share on other sites

Use fileread, filewrite: 

$sFilePath='output.txt' ;here fullpath of your file to be changed
$hFile=FileOpen($sFilePath,0)
$sOld=FileRead($hFile)
FileClose($hFile)
$hFile=FileOpen($sFilePath,2)
$sNew='New line 1'&@CRLF&$sOld
FileWrite($hFile,$sNew)
FileClose($hFile)

 

Share this post


Link to post
Share on other sites

@Fossil Rock,

theoretically it should be possible, although with considerable limitations. but you must be extremely proficient with file tables, streams and the rest of the gang. no-one, i dare say even most MS filesystem engineers, is not in that level. either go with the rewrite method, or rethink your goal entirely.

if it's of any consolation to you, even UNIX does not offer that feature.

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

That works @AutoBert, thanks. My initial test produced a speed of 80-90 seconds for 10,000 iterations (roughly 160KB).

The speed issue is with having to write the file each time. For each iteration that time gets longer and longer.

The $FO_APPEND method in AutoIt can do 10,000,000 iterations in 60 seconds (roughly 152MB).

Now if there were just a $FO_PREPEND that would be awesome.

Edited by Fossil Rock

Agreement is not necessary - thinking for one's self is!

My-Colors.jpg

cuniform2.gif

Share this post


Link to post
Share on other sites

_FileWriteLog can write at begin of a file. So i suggest copy this func and rename the copy. After that edit to your needs, maybe this func is faster than my solution.

Share this post


Link to post
Share on other sites

_FileWriteLog does exactly the same - read the source, then write it back.

if insisting on the rewrite approach, then the best optimization would be to skip all unnecessary pre/post-checks. in that sense, the solution suggested in post #6 by @AutoBert is probably the fastest possible.

Share this post


Link to post
Share on other sites

I don't know of a filesystem able to prepend data to a flat file.

If your use case is going to repeatedly prepend data to large files, then flat files aren't suitable. You'd be much, much happier with a database, like SQLite. We can assist you getting started with this.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

at this point this thread is no more than a Gedankenexperiment, as mentioned in post #5: "It's just an experiment I'm testing..."

if the OP wishes a more practical advice, he is probably aware of that other means and measures exist to serve whatever practical purpose he is after. if he chooses to share his purpose, he may get more concrete ideas.

 

Share this post


Link to post
Share on other sites

Unfortunately _FileWriteLog first reads the file into memory. That can lead to storage problems when dealting with GB files :( 


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

I think the OP doesn't provide enough information to provide a working AND fast solution:

Quote

Considering this may be done 100's or even thousands of times (on the same file)  ...

I do not see a use case where this would be sensible.

Quote

'inject' the code to the beginning?

Why would anyone inject code into a GB file?

He tells us that injecting lines of code at the top of a file is the solution. He wants to implement it using AutoIt.
But he tells us nothing about the problem that he thinks can be solved by injecting lines of code at the top of a file :huh:

We need to know more about the problem before we can suggest a proper solution.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

What about using FileSetPos ?

#Include <FileConstants.au3>

Local $h = FileOpen("file.txt", 1)
_FileWriteAtTheTop($h, "second line" & @CRLF)
_FileWriteAtTheTop($h, "first line" & @CRLF)
FileClose($h)


Func _FileWriteAtTheTop($hFile, $sText)
    FileSetPos($hFile, 0, $FILE_BEGIN)
    Local $sNewText = $sText & FileRead($hFile)
    FileSetPos($hFile, 0, $FILE_BEGIN)
    FileWrite($hFile, $sNewText)
    FileFlush($hFile)
EndFunc

 

Share this post


Link to post
Share on other sites

Has the same problem as orbs' solution. It reads the whole file into memory. Will take forever for a GB file :(


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

Not only the actual use case would be good to know, but writing to a file is only one side. There is no point doing that if the data is never processed. How it should be further processed (read) is the key point: there must be a reason why the OP inists on prepending data (I read "code" as data).

A common case where you'd like to have newest entries at top is with log files, where it's unpractical to have to read giant Gb or even Tb files to just search within a few last entries, which make flat files unsuitable in practice. Again, a well-designed DB is a much better choice, even if writing entries incurs a little time penalty.

Let's see if/when the OP tells us more.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

I think the way to perform this is divide files in chunks and create a new file to append all.

 

For Example

 

File A.data 1gb

File B.data 1gb

Create my new file C.data Create+Write+Append

Get file a.data 

DIvide in chunks 1gb/10

Loop and read 100Mbs (To avoid full file in memory) and append to C.data

Do the same with B.data File.

 

Saludos

 

Edited by Danyfirex

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now