Collect all text files into one large text file?

[newbiescripter]

Hi

Is it possible to combine all the txt files from one folder including the subfolders into one large txt file?

I was thinking about something where you choose which folder where you would like to combine all the txt files into one large and then when it's done you should choose your output folder

and if this is possible where do I start?

Regards

[Zedna]

Yes that's possible and it's quite easy.

List all the files (into arrary) then read (add) content of file into memory variable and finally write content of that variable into output file.

Apropriate AutoIt functions:

FileFindFirstFile
FileFindNextFile
FileRead
FileWrite

For recursive file/directory listing there are some UDF functions on the forum, I think one of them is _FileListToArrayEx() and second _FileSearch(). Just use Search engine.

[newbiescripter]

Thanks it works

But I have a little problem by changing the working directory.

$folder= FileSelectFolder("Find the folder including your text files to combine", "", 2 )
If @error = 1 Then Exit

$save = FileSaveDialog("Select were to save your file" , "" , "Text files (*.txt)" , 16 )
If @error = 1 Then Exit

$folder = @WorkingDir

$search = FileFindFirstFile("*.txt")

I've tryid the code above, but the working dir is changing to the folder where a try to save my file and not to the folder that I've selected even if I set $folder = @workingDir.

I've also tryid

FileChangeDir

but I can't make it work.

Can you please help me with that?

[weaponx]

expandcollapse popup

#include <array.au3> ;Only needed for array display

;Root folder to begin search
$folder = FileSelectFolder("Find the folder including your text files to combine", "", 2 )
If @error = 1 Then Exit

;Recursive search for txt (or TXT) files only
$Array = RecursiveFileSearch($folder, "(?i)\.(txt)\z")

;If any files were found
If $Array[0] > 0 Then
    ;_ArrayDisplay($Array)
    
    ;Output destination
    $save = FileSaveDialog("Select were to save your file" , "" , "Text files (*.txt)" , 16 )
    If @error = 1 Then Exit
    
    $output = FileOpen($save, 1)
    
    ;Loop through found files
    For $X = 1 to $Array[0]
        $file = FileRead($Array[$X])
        FileWrite($output, $file)
    Next
    
    FileClose($output)
EndIf
;RFSstartdir: Path to starting folder
;RFSpattern: RegEx pattern to match i.e. "(?i)\.(mp3)" find all mp3 files - case insensitive, "(?-i)\.(mp3|txt)" find all mp3 and txt files - case sensitive
;RFSrecurse: TRUE = Recursive, FALSE = Non-recursive
;RFSdepth: Internal use only
Func RecursiveFileSearch($RFSstartDir, $RFSpattern = ".", $RFSrecurse = true, $RFSdepth = 0)
   
    ;Ensure starting folder has a trailing slash
    If StringRight($RFSstartDir, 1) <> "\" Then $RFSstartDir &= "\"

    If $RFSdepth = 0 Then
        ;Get count of all files in subfolders for initial array definition
        $RFSfilecount = DirGetSize($RFSstartDir, 1)
        Global $RFSarray[$RFSfilecount[1] + 1]
    EndIf
   
    $RFSsearch = FileFindFirstFile($RFSstartDir & "*.*")
    If @error Then Return

    ;Search through all files and folders in directory
    While 1
        $RFSnext = FileFindNextFile($RFSsearch)
        If @error Then ExitLoop
       
        ;If folder, recurse
        If StringInStr(FileGetAttrib($RFSstartDir & $RFSnext), "D") Then
            If $RFSrecurse Then RecursiveFileSearch($RFSstartDir & $RFSnext, $RFSpattern, $RFSrecurse, $RFSdepth + 1)
        Else
            If StringRegExp($RFSnext, $RFSpattern, 0) Then
                ;Append filename to array
                $RFSarray[$RFSarray[0] + 1] = $RFSstartDir & $RFSnext
               
                ;Increment filecount
                $RFSarray[0] += 1
            EndIf
        EndIf
    WEnd
    FileClose($RFSsearch)

    If $RFSdepth = 0 Then
        Redim $RFSarray[$RFSarray[0] + 1]
        Return $RFSarray
    EndIf
EndFunc   ;==>RecursiveFileSearch

Edited January 30, 2008 by weaponx

[newbiescripter]

Thanks alot now you have created the hole script for me

I didn't no it was so complicated to change the @working dir. Then following code that I've made before were looking like this:

$folder= FileSelectFolder("Find the folder including your text files to combine", "", 2 )
If @error = 1 Then Exit

$save = FileSaveDialog("Select were to save your file" , "" , "Text files (*.txt)" , 16 )
If @error = 1 Then Exit

$folder = @WorkingDir

$search = FileFindFirstFile("*.txt")   

; Check if the search was successful
If $search = -1 Then
    MsgBox(0, "Error", "No files/directories matched the search pattern")
    Exit
EndIf

While 1
    $file = FileFindNextFile($search)
    If @error Then ExitLoop
    If $file = $save Then ExitLoop
    
    $chars = FileRead($file)
    If @error = -1 Then ExitLoop
    
    FileWrite($save , $chars & @CRLF & @CRLF)
    
WEnd

; Close the search handle
FileClose($search)

But once again thanks alot.. because your script does also takes the subfolder with when searching

Regards

[newbiescripter]

It's a great script you have made, but there is a little bug that I hope you maybe could fix

I've changed the filewrite so you can see what I mean so here is the code:

expandcollapse popup

#include <array.au3>;Only needed for array display

;Root folder to begin search
$folder = FileSelectFolder("Find the folder including your text files to combine", "", 2 )
If @error = 1 Then Exit

;Recursive search for txt (or TXT) files only
$Array = RecursiveFileSearch($folder, "(?i)\.(txt)\z")

;If any files were found
If $Array[0] > 0 Then
   ;_ArrayDisplay($Array)
    
   ;Output destination
    $save = FileSaveDialog("Select were to save your file" , "" , "Text files (*.txt)" , 16 )
    If @error = 1 Then Exit
    
    $output = FileOpen($save, 1)
    
    $seperator = @CRLF & @CRLF & "----------------New TXT file----------------" & @CRLF & @CRLF
    
   ;Loop through found files
    For $X = 1 to $Array[0]
        $file = FileRead($Array[$X])
        FileWrite($output, $file & $seperator)
    Next
EndIf
;RFSstartdir: Path to starting folder
;RFSpattern: RegEx pattern to match i.e. "(?i)\.(mp3)" find all mp3 files - case insensitive, "(?-i)\.(mp3|txt)" find all mp3 and txt files - case sensitive
;RFSrecurse: TRUE = Recursive, FALSE = Non-recursive
;RFSdepth: Internal use only
Func RecursiveFileSearch($RFSstartDir, $RFSpattern = ".", $RFSrecurse = true, $RFSdepth = 0)
   
   ;Ensure starting folder has a trailing slash
    If StringRight($RFSstartDir, 1) <> "\" Then $RFSstartDir &= "\"

    If $RFSdepth = 0 Then
       ;Get count of all files in subfolders for initial array definition
        $RFSfilecount = DirGetSize($RFSstartDir, 1)
        Global $RFSarray[$RFSfilecount[1] + 1]
    EndIf
   
    $RFSsearch = FileFindFirstFile($RFSstartDir & "*.*")
    If @error Then Return

   ;Search through all files and folders in directory
    While 1
        $RFSnext = FileFindNextFile($RFSsearch)
        If @error Then ExitLoop
       
       ;If folder, recurse
        If StringInStr(FileGetAttrib($RFSstartDir & $RFSnext), "D") Then
            If $RFSrecurse Then RecursiveFileSearch($RFSstartDir & $RFSnext, $RFSpattern, $RFSrecurse, $RFSdepth + 1)
        Else
            If StringRegExp($RFSnext, $RFSpattern, 0) Then
               ;Append filename to array
                $RFSarray[$RFSarray[0] + 1] = $RFSstartDir & $RFSnext
               
               ;Increment filecount
                $RFSarray[0] += 1
            EndIf
        EndIf
    WEnd
    FileClose($RFSsearch)

    If $RFSdepth = 0 Then
        Redim $RFSarray[$RFSarray[0] + 1]
        Return $RFSarray
    EndIf
EndFunc  ;==>RecursiveFileSearch

If you try run the script and then combining the 3 txt files that I've uploaded into one then you can see what I mean. It does only takes a little part of the texts :/

Hope you could see what i mean and maybe you could fix it or tell me how to do

Regards

Table__126066_1304688.txt

Table__126808_1305531.txt

Table__126962_1305837.txt

[newbiescripter]

Is it possible to make it read the hole first txt file and then the hole second txt and at last the hole third txt file and then combine them all into one large so that they are in order so the first is in the top the second after and the third is last..

Hope you understand what i mean

Thanks for your help weaponx!

[weaponx]

Your files have a hidden NULL character before each line like this:

NULLGame #6657555953 starts.

You can see it by opening the files in SciTe...it is causing a break in FileRead().

[newbiescripter]

Your files have a hidden NULL character before each line like this:

NULLGame #6657555953 starts.

You can see it by opening the files in SciTe...it is causing a break in FileRead().

Is it possible to make it ignore that?

[randallc]

Is it possible to make it ignore that?

Hi,

Helpfile for fileread; binary option.

then strip off the first char, binarymid ,2 etc, before saving again?

best, Randall

Edited January 30, 2008 by randallc

[seandisanti]

Is it possible to make it ignore that?

here's the way i'd go...

$tosearch = FileSelectFolder("Select Folder to combine files from", "c:\");this is the starting directory
$output = FileOpen("c:\output.txt", 2);this is where the list is saved
RecFileSearch($tosearch);this calls the function, passing the starting directory
FileClose($output);once the function is done, this closes the output list
Func RecFileSearch($current);the function takes a string containing the folder currently being searched, on the first call
    Local $search = FileFindFirstFile($current & "\*.*");creates a handle to access all files in the directory being searched
    While 1;infinite loop
        Dim $file = FileFindNextFile($search);this grabs the next file in the folder
        If @error Or StringLen($file) < 1 Then ExitLoop;if an empty string is returned, or if the error is set because there are no more files
        ;the function exits
        If StringLower(StringRight($file, 4)) = ".txt" Then FileWrite($output, FileRead($current & "\" & $file));this writes the contents of the text file to the output
        If StringInStr(FileGetAttrib($current & "\" & $file), "D") And ($file <> "." Or $file <> "..") Then RecFileSearch($current & "\" & $file)
        ;if the file is marked as a directory, but not the . or .. directory, then the function is called for that directory
    WEnd;once the call from the last line (if the function called itself) returns, this loops for the next file/folder
    FileClose($search);once all of the folders and files have been combined, the $search handle is closed
EndFunc   ;==>RecFileSearch

[newbiescripter]

here's the way i'd go...

$tosearch = FileSelectFolder("Select Folder to combine files from", "c:\");this is the starting directory
$output = FileOpen("c:\output.txt", 2);this is where the list is saved
RecFileSearch($tosearch);this calls the function, passing the starting directory
FileClose($output);once the function is done, this closes the output list
Func RecFileSearch($current);the function takes a string containing the folder currently being searched, on the first call
    Local $search = FileFindFirstFile($current & "\*.*");creates a handle to access all files in the directory being searched
    While 1;infinite loop
        Dim $file = FileFindNextFile($search);this grabs the next file in the folder
        If @error Or StringLen($file) < 1 Then ExitLoop;if an empty string is returned, or if the error is set because there are no more files
        ;the function exits
        If StringLower(StringRight($file, 4)) = ".txt" Then FileWrite($output, FileRead($current & "\" & $file));this writes the contents of the text file to the output
        If StringInStr(FileGetAttrib($current & "\" & $file), "D") And ($file <> "." Or $file <> "..") Then RecFileSearch($current & "\" & $file)
        ;if the file is marked as a directory, but not the . or .. directory, then the function is called for that directory
    WEnd;once the call from the last line (if the function called itself) returns, this loops for the next file/folder
    FileClose($search);once all of the folders and files have been combined, the $search handle is closed
EndFunc   ;==>RecFileSearch

The code works fine but doesn't ignore the ***** as NULL :/

Is there any way to ignore the NULL in autoit ?

[weaponx]

You can manually remove the NULL characters in a text editor if you aren't using them in realtime.

Edited January 31, 2008 by weaponx

[randallc]

The code works fine but doesn't ignore the ***** as NULL :/

Is there any way to ignore the NULL in autoit ?

Hi,

Did you see post # 10?

Hi,

Helpfile for fileread; binary option.

then strip off the first char, binarymid ,2 etc, before saving again?

best, Randall

best, Randall

[newbiescripter]

Hi,

Helpfile for fileread; binary option.

then strip off the first char, binarymid ,2 etc, before saving again?

best, Randall

Hi thanks, but I haven't work so much with that :/

Can't really see how I should use it. Is it possible for you to make a little expample?

If I understand you right, then you mean that I should run the binarymid first and then afterwoods run the whole script when one of the * is gone?

Regards

[weaponx]

Hi thanks, but I haven't work so much with that :/

Can't really see how I should use it. Is it possible for you to make a little expample?

If I understand you right, then you mean that I should run the binarymid first and then afterwoods run the whole script when one of the * is gone?

Regards

I'm not sure what he means either. Maybe he thinks the NULL character is at the beginning of the files, when it actually is in multiple places in the each file.

[newbiescripter]

I´'ve looked in the help file and there I can see that it's some sort of fileread where it's instead of reading the characters is reading the 'bytes' and then rewrite the file. and it could might work just as well as fileread, but don't you think that the NULL will break it just like with fileread?

[randallc]

I´'ve looked in the help file and there I can see that it's some sort of fileread where it's instead of reading the characters is reading the 'bytes' and then rewrite the file. and it could might work just as well as fileread, but don't you think that the NULL will break it just like with fileread?

Hi,

Sorry, I was wrong; if there are more than 1 "null" chr(0) at beginning, I think you can just use fileread and stringreplace;

If StringLower(StringRight($file, 4)) = ".txt" Then FileWrite($output, StringReplace(FileRead($current & "\" & $file),chr(0),""))oÝ÷ Ù«­¢+Øì9Õ±±àȹÔÌ)±½°ÀÌØí͹µÄôÅÕ½ÐíQ±}|ÄÈØÀØÙ|ÄÌÀÐØàà¹ÑáÐÅÕ½Ðì°ÀÌØí͹µÈôÅÕ½ÐíQ±}|ȹÑáÐÅÕ½Ðì(ìôôôôôôôôôôôôôôôôôôôôôôôôôôôI¹Ü¥±íI¥ÉÍÐ¥±)±½°ÀÌØíÍ¥±Iõ¥±I1¥¹ ÀÌØí͹µÄ°Ä¤)
½¹Í½±]É¥Ñ ÅÕ½ÐíI¥±±¥¹¥ÉÍÐìÅÍб¥¹ôÅÕ½ÐìµÀìÀÌØíÍ¥±IµÀí1¤)±½°ÀÌØíÍ¥±Iõ¥±I ÀÌØí͹µÄ¤)
½¹Í½±]É¥Ñ ÅÕ½Ðí1¹Ñ ½¥±ÌMÑÉ¥¹IíMÑÉ¥¹1¸ ÀÌØíÍ¥±I¤ôÅÕ½ÐìµÀíMÑÉ¥¹1¸ ÀÌØíÍ¥±I¤µÀí1¤((ìôôôôôôôôôôôôôôôôôôôôôôôôôôô=Á¸°ÉÁ±¹Õ±±Ììm9=Pµ¥¹Éäýt)±½°ÀÌØíÍ¥±IÈõ¥±I ÀÌØí͹µÄ¤(ÀÌØíÍ¥±IÈõMÑÉ¥¹IÁ± ÀÌØíÍ¥±Iȱ¡È À¤°ÅÕ½ÐìÅÕ½Ðì¤)
½¹Í½±]É¥Ñ ÅÕ½Ðí1¹Ñ ½¥±ÌÍÑÉ¥¹ÑÈMÑÉ¥¹¹MÑÉ¥¹IÁ±9Õ±°¡¥¹½Ð½¹¤ôÅÕ½ÐìµÀíMÑÉ¥¹1¸ ÀÌØíÍ¥±IȤµÀí1¤)¥±±Ñ ÀÌØí͹µÈ¤)¥±]É¥Ñ ÀÌØí͹µÈ°ÀÌØíÍ¥±IȤì©ÕÍи½É¥¹ÉäÝɥѹ½Ü½ÍÑÉ¥¹Ù±Õ(ìôôôôôôôôôôôôôôôôôôôôôôôôôôôI¹Ü¥±ì)±½°ÀÌØíÍ¥±I1¥¹Èõ¥±I1¥¹ ÀÌØí͹µÈ°Ä¤)
½¹Í½±]É¥Ñ ÅÕ½ÐíI¥¸ÑÈÑÈIÁ±¹Õ±±ÌìÅÍб¥¹ôÅÕ½ÐìµÀìÀÌØíÍ¥±I1¥¹ÈµÀí1¤)±½°ÀÌØíÍ¥±IÈõ¥±I ÀÌØí͹µÈ¤)
½¹Í½±]É¥Ñ ÅÕ½ÐíI¥¸ÑÈÑÈIÁ±¹Õ±±Ìì1¹Ñ ½¥±ôÅÕ½ÐìµÀíMÑÉ¥¹1¸ ÀÌØíÍ¥±IȤµÀí1

Best, randall

[newbiescripter]

Where in the code should I place it ?

you can place it in weaponx's or cemeronsdad's Script I think they both a great.

It's because I can't really see where to place it, sorry!

All thanks for helping!

[randallc]

Where in the code should I place it ?

you can place it in weaponx's or cemeronsdad's Script I think they both a great.

It's because I can't really see where to place it, sorry!

All thanks for helping!

If StringLower(StringRight($file, 4)) = ".txt" Then FileWrite($output, StringReplace(FileRead($current & "\" & $file),chr(0),""))

I had intended to replace that line in @weaponX script, but where you will;

the other script is just a demo for you to see it working on one file..

best, randall