_FileWriteToLine is too slow

[Bert]

I'm attempting to reformat a text file. Here is a small snippet of what I'm dealing with:

Actualtests.com - The Power of Knowing

QUESTION 1:

The Certkiller .com network is based on the star topology. What will happen when a

SINGLE connection to one of the network hosts fails?

A. The whole network will be down and none of the hosts will have network

connectivity.

B. The failed connection will be down and only its hosts will have no network

connectivity.

C. All hosts will still be connected to the network and only failed connection will be

down.

D. The failed connection will be down and only the two hosts between the failed

connection will have no network connectivity.

Answer: B

Explanation:

In the star topology each computer is connected to a central point by a separate cable or

wireless connection. Thus each computer has a dedicated link to the network central

device and a break in the link between a particular computer and the central network

device will affect only that computer.

Incorrect Answers:

A: In a bus or a ring topology there is no fault tolerance and a single fault in the ring or

bus will bring down the whole network.

C: In a full-mesh network there are multiple connections to each host. Thus if one

connection fails the other connections will ensure that the network remain fully

operational.

D: In the star topology hosts are connected to a central hub or switch and are not directly

connected to each other.

References:

David Groth and Toby Skandier, Network+ Study Guide (4th Edition), Sybex, Alameda

CA, 2005, pp. 10-16.

QUESTION 2:

What will happen when a SINGLE connection to one of the network hosts in a full

mesh network fails?

A. The whole network will be down and none of the hosts will have network

connectivity.

B. The failed connection will be down and only its hosts will have no network

connectivity.

C. The failed connection will be down but all hosts will still be connected to the network.

D. The failed connection will be down and only the two hosts between the failed

connection will have no network connectivity.

Actualtests.com - The Power of Knowing

Answer: C

Explanation:

In a full mesh network, each node has a connection to at least two other nodes. Thus,

should one connection fail, it will have no effect on communication as all nodes will be

connected to at least one other node.

Incorrect Answers:

A: In a bus or a ring topology there is no fault tolerance and a single fault in the ring or

bus will bring down the whole network.

B: In the star topology each computer is connected to a central point by a separate cable

or wireless connection. Thus each computer has a dedicated link to the network central

device and a break in the link between a particular computer and the central network

device will affect only that computer.

D: In the star topology hosts are connected to a central hub or switch and are not directly

connected to each other.

References:

David Groth and Toby Skandier, Network+ Study Guide (4th Edition), Sybex, Alameda

CA, 2005, pp. 14-15.

QUESTION 3:

A single connection in a token bus network fails. What will happen?

A. The whole network will be down and none of the hosts will have network

connectivity.

B. The failed connection will be down and only its hosts will have no network

connectivity.

C. All hosts will still be connected to the network and only failed connection will be

down.

D. The failed connection will be down and only the two hosts between the failed

connection will have no network connectivity.

Answer: A

Explanation:

In a bus topology there is no fault tolerance and a single fault in the bus will bring down

the whole network

Incorrect Answers:

B: In the star topology each computer is connected to a central point by a separate cable

or wireless connection. Thus each computer has a dedicated link to the network central

device and a break in the link between a particular computer and the central network

device will affect only that computer.

C: In the star topology each computer is connected to a central point by a separate cable

or wireless connection. Thus each computer has a dedicated link to the network central

device and a break in the link between a particular computer and the central network

Actualtests.com - The Power of Knowing

device will affect only that computer.

D: In the star topology hosts are connected to a central hub or switch and are not directly

connected to each other.

References:

David Groth and Toby Skandier, Network+ Study Guide (4th Edition), Sybex, Alameda

CA, 2005, pp. 14-15.

Now, I need to reformat the information and remove lines here or there to make everything work the way I need it to. For example, I need to remove every reference of "Actualtests.com - The Power of Knowing". Here is the script I'm using to just remove this

#include <file.au3>
$file = FileOpen("N10-003.txt", 0)
$CountLines = _FileCountLines("N10-003.txt")
; Check if file opened for reading OK
TraySetToolTip("0 of "&$CountLines&" lines checked")
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
$x=1
While 1
    $line = FileReadLine($file, $x)
    If @error = -1 Then ExitLoop
    $string = StringInStr($line, "Actualtests.com")
    if $line = "Actualtests.com - The Power of Knowing" then    
        _FileWritetoLine(@ScriptDir&"/N10-003.txt", $x,"", 1)
    endif
    TraySetToolTip($x&" of "&$CountLines&" lines checked.")
    $x= $x+1
Wend

FileClose($file)

The file has about 13000 lines. It takes about 20 minutes to run through the thing. Any way to speed it up?

[MikeP]

I don't know if i'm allowed to advise for another soft but i'm doing such text management with UltraEdit which is a highly powerful text editor and this would be done in a couple of seconds .. you gotta learn UE Macro writing though ..and UE regexp but it's not that hard.. there's not hundreds of functions.

[Jos]

It should be a lot faster when you first use a array to "write the records to and then when processed the whole file you writet he array to the file.

Jos

[MikeP]

_FileReadToArray() is handy yup

[monoceres]

It seems like your over complicating things, this parse my 13k lines text file in 300 ms :

$out=FileOpen("test2.txt",1)
$in=FileOpen("test.txt",0)
$timer=TimerInit()
while True
    $temp=FileReadLine($in)
    If $temp="" Then ExitLoop
    If Not StringInStr($temp,"teststring") Then FileWriteLine($out,$temp)
WEnd
ConsoleWrite(TimerDiff($timer)/1000&@CRLF)

Tested by copying your example over and over....

Or have I missed something?

Edit: of course you can make all sorts of comparisons in the loop

Edited May 7, 2008 by monoceres

[Siao]

1. Reading entire file into variable, StringRegExpReplace whatever you need, and writing to file, would be the fastest. There are many topics with RegExps, including about replacing lines, one for example as far ago as yesterday: http://www.autoitscript.com/forum/index.ph...mp;#entry517605

2. What Jos said would look like:

$aLines = StringSplit(FileRead(@DesktopDir&"\test.txt"), @CRLF, 1)
For $i = 1 To $aLines[0]
    If StringInStr($aLines[$i], "Actualtests.com - The Power of Knowing") Then $aLines[$i] = ""
Next
$sOut = $aLines[1]
For $i = 2 To $aLines[0]
    $sOut &= @CRLF & $aLines[$i]
Next
FileWrite(@DesktopDir&"\test_mod.txt", $sOut)

And there's certainly no need to use any of those newb-ass File.au3 functions, when the above code basically does the same but without overhead.

Edited May 7, 2008 by Siao

[Bert]

I'm like all the suggestions. The test that you see have to be reformatted so they will work in the transdumper test engine. This test has about 600+ questions, so formatting the thing to make it work is a real bitch. I want to automate as much as possible to save time. From the suggestions so far, it should be pretty simple to fix most of the formatting issues. Something tells me I will still have to go through the thing and make minor fixes.

I will get to test the results of everyone's suggestions tomorrow.

[SmOke_N]

1. Reading entire file into variable, StringRegExpReplace whatever you need, and writing to file, would be the fastest. There are many topics with RegExps, including about replacing lines, one for example as far ago as yesterday: http://www.autoitscript.com/forum/index.ph...mp;#entry517605

2. What Jos said would look like:

$aLines = StringSplit(FileRead(@DesktopDir&"\test.txt"), @CRLF, 1)
For $i = 1 To $aLines[0]
    If StringInStr($aLines[$i], "Actualtests.com - The Power of Knowing") Then $aLines[$i] = ""
Next
$sOut = $aLines[1]
For $i = 2 To $aLines[0]
    $sOut &= @CRLF & $aLines[$i]
Next
FileWrite(@DesktopDir&"\test_mod.txt", $sOut)

And there's certainly no need to use any of those newb-ass File.au3 functions, when the above code basically does the same but without overhead.

Question: Any specific reason you are looping through it twice rather than just making a conditional statement in the first loop?

pseudo:

If StringInStr(blah) Then
    $sOut &= $var[$i] & @CRLF
Else
    $sOut &= @CRLF
EndIf

[Bert]

Ok, so far I got it so I can fix most of the formatting pretty quick. I can do the 13K file in about 7 seconds

expandcollapse popup

#include <String.au3>
dim $temp
dim $name = "N10-003"
$out=FileOpen("N10-003_1.txt",1)
$in=FileOpen("N10-003.txt",0)
$timer=TimerInit()
$r = 1
$j = 20000; for testing
$s = 1
 
while True
    _FRL()
    If $temp="" Then 
        $r = $r+1
        if $r = 10 then ExitLoop
    else

        Select
            case StringInStr($temp,"QUESTION")
                $STL = StringTrimLeft($temp, 9)
                $STR = StringTrimRight($STL, 1)
                if $STR = "399" then 
                    FileClose($out)
                    $out=FileOpen("N10-003_2.txt",1)
                    Dim $out
                endif   
                _FRL()
                if $temp = "" then _FRL()
                if $temp ="N10-003" then _FRL()
                if $temp ="Actualtests.com - The Power of Knowing" then _FRL()
                if $temp =$name then _FRL() 
                $replace_1 = StringReplace($temp, "Certkiller .com", "AutoIt.com")
                FileWriteLine($out,@CRLF&$STR&". "&$replace_1);each question start              
            case StringInStr($temp,"Answer:")
                FileWriteLine($out,@CRLF&$temp);each answer start       
            case StringInStr($temp,"Explanation")
                FileWriteLine($out,@CRLF&"<B>"&$temp&"</B><br>");each Explanation start
            case StringInStr($temp,"Incorrect Answers:")
                FileWriteLine($out,@CRLF&"<B>"&$temp&"</B><br>");each Incorrect Answers: start  
            case StringInStr($temp,"Incorrect Answers")
                FileWriteLine($out,@CRLF&"<B>"&$temp&"</B><br>");each Incorrect Answers: start                  
            case StringInStr($temp,"References:")
                FileWriteLine($out,@CRLF&"<B>"&$temp&"</B><br>");each References Answers: start 
            case StringInStr($temp,"Reference:")
                FileWriteLine($out,@CRLF&"<B>"&$temp&"</B><br>");each References Answers: start             
            case StringInStr($temp,"Certkiller .com")
                $replace_1 = StringReplace($temp, "Certkiller .com", "AutoIt.com")
                FileWriteLine($out,$replace_1)
            case StringInStr($temp,"A:")
                $STL = StringTrimLeft($temp, 2)
                FileWriteLine($out,"<B>A -</B>"&$STL)
            case StringInStr($temp,"B:")
                $STL = StringTrimLeft($temp, 2)
                FileWriteLine($out,"<B>B -</B>"&$STL)
            case StringInStr($temp,"C:")
                $STL = StringTrimLeft($temp, 2)
                FileWriteLine($out,"<B>C -</B>"&$STL)   
            case StringInStr($temp,"D:")
                $STL = StringTrimLeft($temp, 2)
                FileWriteLine($out,"<B>D -</B>"&$STL)   
            case StringInStr($temp,"E:")
                $STL = StringTrimLeft($temp, 2)
                FileWriteLine($out,"<B>E -</B>"&$STL)   
            case StringInStr($temp,"F:")
                $STL = StringTrimLeft($temp, 2)
                FileWriteLine($out,"<B>F -</B>"&$STL)               
            case StringInStr($temp,$name)
            ;;;;
            case StringInStr($temp,"Actualtests.com - The Power of Knowing")
            ;;;;
            case StringInStr($temp,"?")
                FileWriteLine($out,$temp&@CRLF&""&@CRLF)
            case Else 
                FileWriteLine($out,$temp)
                $r = 1
                $s = $s+1
                if $s= $j then 
                    exitloop
                endif               
        EndSelect               
    endif
WEnd
FileClose($out)
FileClose($in)

func _FRL()
    $temp=FileReadLine($in)
    Return($temp)
EndFunc

[monoceres]

That's great, from 20 minutes to 7 seconds. That's a improvement by around 17000 % (If I got my maths straight).

A great example why you shouldn't rely on the UDF's for speed.

[Bert]

I'm still having some format issues I would like to fix. Here is the first question from the test:

Exam : 
Title : Network+ (2007)
Ver : 05.07.08

Actualtests.com - The Power of Knowing
QUESTION 1:
The Certkiller .com network is based on the star topology. What will happen when a
SINGLE connection to one of the network hosts fails?
A. The whole network will be down and none of the hosts will have network
connectivity.
B. The failed connection will be down and only its hosts will have no network
connectivity.
C. All hosts will still be connected to the network and only failed connection will be
down.
D. The failed connection will be down and only the two hosts between the failed
connection will have no network connectivity.
Answer: B
Explanation:
In the star topology each computer is connected to a central point by a separate cable or
wireless connection. Thus each computer has a dedicated link to the network central
device and a break in the link between a particular computer and the central network
device will affect only that computer.
Incorrect Answers:
A: In a bus or a ring topology there is no fault tolerance and a single fault in the ring or
bus will bring down the whole network.
C: In a full-mesh network there are multiple connections to each host. Thus if one
connection fails the other connections will ensure that the network remain fully
operational.
D: In the star topology hosts are connected to a central hub or switch and are not directly
connected to each other.
References:
David Groth and Toby Skandier, Network+ Study Guide (4th Edition), Sybex, Alameda
CA, 2005, pp. 10-16.

Now, with the code I have written, I can get it to this:

Exam : 
Title : Network+ (2007)
Ver : 05.07.08

1. The AutoIt.com network is based on the star topology. What will happen when a
SINGLE connection to one of the network hosts fails?

A. The whole network will be down and none of the hosts will have network
connectivity.
B. The failed connection will be down and only its hosts will have no network
connectivity.
C. All hosts will still be connected to the network and only failed connection will be
down.
D. The failed connection will be down and only the two hosts between the failed
connection will have no network connectivity.

Answer: B

<B>Explanation:</B><br>
In the star topology each computer is connected to a central point by a separate cable or
wireless connection. Thus each computer has a dedicated link to the network central
device and a break in the link between a particular computer and the central network
device will affect only that computer.

<B>Incorrect Answers:</B><br>
<B>A -</B> In a bus or a ring topology there is no fault tolerance and a single fault in the ring or
bus will bring down the whole network.
<B>C -</B> In a full-mesh network there are multiple connections to each host. Thus if one
connection fails the other connections will ensure that the network remain fully
operational.
<B>D -</B> In the star topology hosts are connected to a central hub or switch and are not directly
connected to each other.

<B>References:</B><br>
David Groth and Toby Skandier, Network+ Study Guide (4th Edition), Sybex, Alameda
CA, 2005, pp. 10-16.

What I need to fix are things like in my answer choices that show a carriage return causing the information to be displayed on 2 lines, I need to fix the answer so it goes from this:

A. The whole network will be down and none of the hosts will have network
connectivity.

To this:

A. The whole network will be down and none of the hosts will have network connectivity.

The differences is the whole string is on one line, without the carriage return that notepad shows in it's formatting. What would be the best method of handling _stringinsert() so that I can combine each FileReadLine command on the answers? I would use the same formula in other parts of the test so as to address the notepad carriage formatting problems I'm experiencing.

[SmOke_N]

StringReplace($sString, @CRLF, "")?

[Bert]

The problem with that is there is no @crlf at the end of the string. What you see in my example is the string I'm trying to fix. There are 2 strings I need to make as one string.

[SmOke_N]

The problem with that is there is no @crlf at the end of the string. What you see in my example is the string I'm trying to fix. There are 2 strings I need to make as one string.

You're too confusing lol. The test string you provided does in fact have a carriage return/Line feed in it.

ConsoleWrite(StringReplace(ClipGet(), @CRLF, " ") & @CRLF)

A. The whole network will be down and none of the hosts will have network

connectivity.

I copied the above from your example and ran the consolewrite test to show.

Otherwise, you may look at StringRegExpReplace to do it with ALOT of tweaking.

[Siao]

Since you apparently did your best to thoroughly ignore all the good advice and take the ugliest route, I see no reason for me to continue posting in this thread any more, but a

StringRegExpReplace($string, "(\n[[:alnum:]]\.\V+)\v+(\V+\r(?=\n([[:alnum:]]\.)|\v))", "\1 \2")

would change

1. The AutoIt.com network is based on the star topology. What will happen when a
SINGLE connection to one of the network hosts fails?

A. The whole network will be down and none of the hosts will have network
connectivity.
B. The failed connection will be down and only its hosts will have no network
connectivity.
C. All hosts will still be connected to the network and only failed connection will be
down.
D. The failed connection will be down and only the two hosts between the failed
connection will have no network connectivity.

Answer: B

to

1. The AutoIt.com network is based on the star topology. What will happen when a SINGLE connection to one of the network hosts fails?

A. The whole network will be down and none of the hosts will have network connectivity.
B. The failed connection will be down and only its hosts will have no network connectivity.
C. All hosts will still be connected to the network and only failed connection will be down.
D. The failed connection will be down and only the two hosts between the failed connection will have no network connectivity.

Answer: B

Whatever floats your boat though...

[Siao]

You're too confusing lol. The test string you provided does in fact have a carriage return/Line feed in it.

Yes, but he reads the file line by line so that's two separate lines to him.

[SmOke_N]

Yes, but he reads the file line by line so that's two separate lines to him.

Your example shows what I was thinking of anyway. I probably would have used RegExp for the whole thing.

[Bert]

Since you apparently did your best to thoroughly ignore all the good advice and take the ugliest route, I see no reason for me to continue posting in this thread any more, but a

StringRegExpReplace($string, "(\n[[:alnum:]]\.\V+)\v+(\V+\r(?=\n([[:alnum:]]\.)|\v))", "\1 \2")

would change

1. The AutoIt.com network is based on the star topology. What will happen when a
SINGLE connection to one of the network hosts fails?

A. The whole network will be down and none of the hosts will have network
connectivity.
B. The failed connection will be down and only its hosts will have no network
connectivity.
C. All hosts will still be connected to the network and only failed connection will be
down.
D. The failed connection will be down and only the two hosts between the failed
connection will have no network connectivity.

Answer: B

to

1. The AutoIt.com network is based on the star topology. What will happen when a SINGLE connection to one of the network hosts fails?

A. The whole network will be down and none of the hosts will have network connectivity.
B. The failed connection will be down and only its hosts will have no network connectivity.
C. All hosts will still be connected to the network and only failed connection will be down.
D. The failed connection will be down and only the two hosts between the failed connection will have no network connectivity.

Answer: B

Whatever floats your boat though...

I didn't ignore you advice, I didn't quite understand your approach. The solution I did come up with does work.

I tried this using what you showed me, and it didn't work as I needed

$in=FileOpen("N10-003.txt",0)
$String = FileRead($in)
$STR = StringRegExpReplace($string, "(\n[[:alnum:]]\.\V+)\v+(\V+\r(?=\n([[:alnum:]]\.)|\v))", "\1 \2")
$FW = FileWrite("N10-003_3.txt", $STR)

My output was this:

Exam : 
Title : Network+ (2007)
Ver : 05.07.08

Actualtests.com - The Power of Knowing
QUESTION 1:
The Certkiller .com network is based on the star topology. What will happen when a
SINGLE connection to one of the network hosts fails?
A. The whole network will be down and none of the hosts will have network connectivity.
B. The failed connection will be down and only its hosts will have no network connectivity.
C. All hosts will still be connected to the network and only failed connection will be down.
D. The failed connection will be down and only the two hosts between the failed connection will have no network connectivity.
Answer: B
Explanation:
In the star topology each computer is connected to a central point by a separate cable or
wireless connection. Thus each computer has a dedicated link to the network central
device and a break in the link between a particular computer and the central network
device will affect only that computer.
Incorrect Answers:
A: In a bus or a ring topology there is no fault tolerance and a single fault in the ring or
bus will bring down the whole network.
C: In a full-mesh network there are multiple connections to each host. Thus if one
connection fails the other connections will ensure that the network remain fully
operational.
D: In the star topology hosts are connected to a central hub or switch and are not directly
connected to each other.
References:
David Groth and Toby Skandier, Network+ Study Guide (4th Edition), Sybex, Alameda
CA, 2005, pp. 10-16.

The formatting is still not right, and seeing how the questions can vary in format, and the fact I have 623 of them, it can get quite complex. If you think your approach is better, and I do believe it is (Though I do not understand it nearly as well as you do) if you want, I can send you the entire txt file for you to figure out. No offence, I'm not trying to be thick headed or a jerk here. I like to learn.

[Bert]

Here is the txt file I'm attempting to fix. For those who want to take a stab at it....http://www.autoitscript.com/fileman/index.php?act=list&op=get&target=N10-003.txt

[SmOke_N]

I didn't ignore you advice, I didn't quite understand your approach. The solution I did come up with does work.

I tried this using what you showed me, and it didn't work as I needed

$in=FileOpen("N10-003.txt",0)
$String = FileRead($in)
$STR = StringRegExpReplace($string, "(\n[[:alnum:]]\.\V+)\v+(\V+\r(?=\n([[:alnum:]]\.)|\v))", "\1 \2")
$FW = FileWrite("N10-003_3.txt", $STR)

My output was this:

Exam : 
Title : Network+ (2007)
Ver : 05.07.08

Actualtests.com - The Power of Knowing
QUESTION 1:
The Certkiller .com network is based on the star topology. What will happen when a
SINGLE connection to one of the network hosts fails?
A. The whole network will be down and none of the hosts will have network connectivity.
B. The failed connection will be down and only its hosts will have no network connectivity.
C. All hosts will still be connected to the network and only failed connection will be down.
D. The failed connection will be down and only the two hosts between the failed connection will have no network connectivity.
Answer: B
Explanation:
In the star topology each computer is connected to a central point by a separate cable or
wireless connection. Thus each computer has a dedicated link to the network central
device and a break in the link between a particular computer and the central network
device will affect only that computer.
Incorrect Answers:
A: In a bus or a ring topology there is no fault tolerance and a single fault in the ring or
bus will bring down the whole network.
C: In a full-mesh network there are multiple connections to each host. Thus if one
connection fails the other connections will ensure that the network remain fully
operational.
D: In the star topology hosts are connected to a central hub or switch and are not directly
connected to each other.
References:
David Groth and Toby Skandier, Network+ Study Guide (4th Edition), Sybex, Alameda
CA, 2005, pp. 10-16.

The formatting is still not right, and seeing how the questions can vary in format, and the fact I have 623 of them, it can get quite complex. If you think your approach is better, and I do believe it is (Though I do not understand it nearly as well as you do) if you want, I can send you the entire txt file for you to figure out. No offence, I'm not trying to be thick headed or a jerk here. I like to learn.

You can send it to me... I'll have a look, I just need a detailed outline of one of them to see what the exact output you hope to achieve would be.