Sign in to follow this  
Followers 0
hogg

lots of data

4 posts in this topic

I have wrote this script that goes through many diffrent files and places the selected data into a text file. my problem is that after going through about 510 files it stops posting this data to the text file. this needs to work on about 30,000 files like this. is there any limitations that I may not be aware of.

sample code atached

$x = 1400000

$z=0

$n=1

Do

$file = FileOpen("C:\vote\" & $x & ".html", 0)

; Check if file opened for reading OK

#cs

If $file = -1 Then

MsgBox(0, "Error", "Unable to open file.")

Exit

EndIf

#ce

If $file = -1 Then

Else

$id = FileReadLine($file, 85)

$regdate = FileReadLine($file, 89)

$lastvote = FileReadLine($file, 87)

$name = FileReadLine($file, 106)

$dob = FileReadLine($file, 107)

$street = FileReadLine($file, 131)

$city = FileReadLine($file, 133)

$poll = FileReadLine($file, 148)

$polladd = FileReadLine($file, 149)

$sd = FileReadLine($file, 168)

$citylim = FileReadLine($file, 170)

$cityward = FileReadLine($file, 171)

$status = FileReadLine($file, 111)

$Precinct = FileReadLine($file, 134)

$schoolzone = FileReadLine($file, 169)

$cd = FileReadLine($file, 172)

$jd = FileReadLine($file, 173)

$dc = FileReadLine($file, 174)

$sr = FileReadLine($file, 190)

$srnum = FileReadLine($file, 189)

$ss = FileReadLine($file, 192)

$ssnum = FileReadLine($file, 191)

$jp = FileReadLine($file, 194)

$jpnum = FileReadLine($file, 193)

$precsplit = FileReadLine($file, 208)

$party = FileReadLine($file, 209)

$id2 = StringTrimRight($id, 5)

$id3 = StringTrimLeft($id2, 26)

$lastvote2 = StringTrimRight($lastvote, 5)

$lastvote3 = StringTrimLeft($lastvote2, 26)

$name2 = StringTrimRight($name, 5)

$name3 = StringTrimLeft($name2, 38)

$dob2 = StringTrimRight($dob, 5)

$dob3 = StringTrimLeft($dob2, 38)

$street2 = StringTrimRight($street, 5)

$street3 = StringTrimLeft($street2, 38)

$city2 = StringTrimRight($city, 5)

$city3 = StringTrimLeft($city2, 38)

$poll2 = StringTrimRight($poll, 5)

$poll3 = StringTrimLeft($poll2, 38)

$polladd2 = StringTrimRight($polladd, 5)

$polladd3 = StringTrimLeft($polladd2, 38)

$sd2 = StringTrimRight($sd, 5)

$sd3 = StringTrimLeft($sd2, 38)

$citylim2 = StringTrimRight($citylim, 5)

$citylim3 = StringTrimLeft($citylim2, 38)

$cityward2 = StringTrimRight($cityward, 5)

$cityward3 = StringTrimLeft($cityward2, 38)

$Precinct2 = StringTrimRight($Precinct, 5)

$Precinct3 = StringTrimLeft($Precinct2, 37)

$schoolzone2 = StringTrimRight($schoolzone, 5)

$schoolzone3 = StringTrimLeft($schoolzone2, 37)

$cd2 = StringTrimRight($cd, 5)

$cd3 = StringTrimLeft($cd2, 38)

$jd2 = StringTrimRight($jd, 5)

$jd3 = StringTrimLeft($jd2, 37)

$dc2 = StringTrimRight($dc, 5)

$dc3 = StringTrimLeft($dc2, 38)

$ss2 = StringTrimRight($ss, 5)

$ss3 = StringTrimLeft($ss2, 38)

$ssnum2 = StringTrimRight($ssnum, 5)

$ssnum3 = StringTrimLeft($ssnum2, 37)

$jp2 = StringTrimRight($jp, 5)

$jp3 = StringTrimLeft($jp2, 38)

$jpnum2 = StringTrimRight($jpnum, 5)

$jpnum3 = StringTrimLeft($jpnum2, 37)

$precsplit2 = StringTrimRight($precsplit, 5)

$precsplit3 = StringTrimLeft($precsplit2, 26)

$party2 = StringTrimRight($party, 5)

$party3 = StringTrimLeft($party2, 26)

$sr2 = StringTrimRight($sr, 5)

$sr3 = StringTrimLeft($sr2, 38)

$srnum2 = StringTrimRight($srnum, 5)

$srnum3 = StringTrimLeft($srnum2, 37)

$regdate2 = StringTrimRight($regdate, 5)

$regdate3 = StringTrimLeft($regdate2, 26)

;MsgBox(0, "Line read:", $id3 & " "& $lastvote3 &" "& $name3 & " " & $regdate3 &" "& $dob3 & " "& $street3 & " "& $city3 & " "& $poll3 & " "& $polladd3 & " "& $sd3 & " "& $citylim3 & " "& $cityward3& " "& $status & " "& $Precinct3 & " "& $schoolzone3& " "& $cd3& " "& $jd3& " "& $dc3& " "& $ss3& " "& $ssnum3 & " "& $jp3& " "& $jpnum3& " "& $precsplit3& " "& $party3& " "& $sr3& " "& $srnum3 )

$file2 = "c:\voter"& $n &".txt"

FileOpen($file2, 1)

FileWrite($file2, $id3 & "," & $lastvote3 & "," & $regdate3 & "," & $name3 & "," & $dob3 & "," & $street3 & ", " & $city3 & "," & $poll3 & "," & $polladd3 & "," & $sd3 & "," & $citylim3 & "," & $cityward3 & "," & $status & "," & $Precinct3 & "," & $schoolzone3 & "," & $cd3 & "," & $jd3 & "," & $dc3 & "," & $ss3 & "," & $ssnum3 & "," & $jp3 & "," & $jpnum3 & "," & $precsplit3 & "," & $party3 & "," & $sr3 & "," & $srnum3 & @CRLF)

FileClose($file)

FileClose($file2)

$z=$z+1

EndIf

if $z = 500 then

$n=$n+1

$z=0

EndIf

$x = $x + 1

Until $x = 1500000

data.au3

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

You have an issue with $File2 not using the file handle to close the file.

Try:

$x = 1400000
$z = 0
$n = 1
Do
    $h_file = FileOpen("C:\vote\" & $x & ".html", 0)

; Check if file opened for reading OK
    #cs
        If $h_file = -1 Then
        MsgBox(0, "Error", "Unable to open file.")
        Exit
        EndIf
    #ce
    If Not ($h_file = -1) Then
        $id = FileReadLine($h_file, 85)
        $regdate = FileReadLine($h_file, 89)
        $lastvote = FileReadLine($h_file, 87)
        $name = FileReadLine($h_file, 106)
        $dob = FileReadLine($h_file, 107)
        $street = FileReadLine($h_file, 131)
        $city = FileReadLine($h_file, 133)
        $poll = FileReadLine($h_file, 148)
        $polladd = FileReadLine($h_file, 149)
        $sd = FileReadLine($h_file, 168)
        $citylim = FileReadLine($h_file, 170)
        $cityward = FileReadLine($h_file, 171)
        $status = FileReadLine($h_file, 111)
        $Precinct = FileReadLine($h_file, 134)
        $schoolzone = FileReadLine($h_file, 169)
        $cd = FileReadLine($h_file, 172)
        $jd = FileReadLine($h_file, 173)
        $dc = FileReadLine($h_file, 174)
        $sr = FileReadLine($h_file, 190)
        $srnum = FileReadLine($h_file, 189)
        $ss = FileReadLine($h_file, 192)
        $ssnum = FileReadLine($h_file, 191)
        $jp = FileReadLine($h_file, 194)
        $jpnum = FileReadLine($h_file, 193)
        $precsplit = FileReadLine($h_file, 208)
        $party = FileReadLine($h_file, 209)

        $id2 = StringTrimRight($id, 5)
        $id3 = StringTrimLeft($id2, 26)

        $lastvote2 = StringTrimRight($lastvote, 5)
        $lastvote3 = StringTrimLeft($lastvote2, 26)

        $name2 = StringTrimRight($name, 5)
        $name3 = StringTrimLeft($name2, 38)

        $dob2 = StringTrimRight($dob, 5)
        $dob3 = StringTrimLeft($dob2, 38)

        $street2 = StringTrimRight($street, 5)
        $street3 = StringTrimLeft($street2, 38)

        $city2 = StringTrimRight($city, 5)
        $city3 = StringTrimLeft($city2, 38)

        $poll2 = StringTrimRight($poll, 5)
        $poll3 = StringTrimLeft($poll2, 38)

        $polladd2 = StringTrimRight($polladd, 5)
        $polladd3 = StringTrimLeft($polladd2, 38)

        $sd2 = StringTrimRight($sd, 5)
        $sd3 = StringTrimLeft($sd2, 38)

        $citylim2 = StringTrimRight($citylim, 5)
        $citylim3 = StringTrimLeft($citylim2, 38)

        $cityward2 = StringTrimRight($cityward, 5)
        $cityward3 = StringTrimLeft($cityward2, 38)


        $Precinct2 = StringTrimRight($Precinct, 5)
        $Precinct3 = StringTrimLeft($Precinct2, 37)

        $schoolzone2 = StringTrimRight($schoolzone, 5)
        $schoolzone3 = StringTrimLeft($schoolzone2, 37)

        $cd2 = StringTrimRight($cd, 5)
        $cd3 = StringTrimLeft($cd2, 38)


        $jd2 = StringTrimRight($jd, 5)
        $jd3 = StringTrimLeft($jd2, 37)

        $dc2 = StringTrimRight($dc, 5)
        $dc3 = StringTrimLeft($dc2, 38)

        $ss2 = StringTrimRight($ss, 5)
        $ss3 = StringTrimLeft($ss2, 38)


        $ssnum2 = StringTrimRight($ssnum, 5)
        $ssnum3 = StringTrimLeft($ssnum2, 37)

        $jp2 = StringTrimRight($jp, 5)
        $jp3 = StringTrimLeft($jp2, 38)

        $jpnum2 = StringTrimRight($jpnum, 5)
        $jpnum3 = StringTrimLeft($jpnum2, 37)


        $precsplit2 = StringTrimRight($precsplit, 5)
        $precsplit3 = StringTrimLeft($precsplit2, 26)

        $party2 = StringTrimRight($party, 5)
        $party3 = StringTrimLeft($party2, 26)

        $sr2 = StringTrimRight($sr, 5)
        $sr3 = StringTrimLeft($sr2, 38)

        $srnum2 = StringTrimRight($srnum, 5)
        $srnum3 = StringTrimLeft($srnum2, 37)

        $regdate2 = StringTrimRight($regdate, 5)
        $regdate3 = StringTrimLeft($regdate2, 26)

    ;MsgBox(0, "Line read:", $id3 & " "& $lastvote3 &" "& $name3 & " " & $regdate3 &" "& $dob3 & " "& $street3 & " "& $city3 & " "& $poll3 & " "& $polladd3 & " "& $sd3 & " "& $citylim3 & " "& $cityward3& " "& $status & " "& $Precinct3 & " "& $schoolzone3& " "& $cd3& " "& $jd3& " "& $dc3& " "& $ss3& " "& $ssnum3 & " "& $jp3& " "& $jpnum3& " "& $precsplit3& " "& $party3& " "& $sr3& " "& $srnum3 )
        $h_file2 = "c:\voter" & $n & ".txt"
        $h_file2 = FileOpen($h_file2, 1)
        FileWrite($h_file2, $id3 & "," & $lastvote3 & "," & $regdate3 & "," & $name3 & "," & $dob3 & "," & $street3 & ",  " & $city3 & "," & $poll3 & "," & $polladd3 & "," & $sd3 & "," & $citylim3 & "," & $cityward3 & "," & $status & "," & $Precinct3 & "," & $schoolzone3 & "," & $cd3 & "," & $jd3 & "," & $dc3 & "," & $ss3 & "," & $ssnum3 & "," & $jp3 & "," & $jpnum3 & "," & $precsplit3 & "," & $party3 & "," & $sr3 & "," & $srnum3 & @CRLF)
        FileClose($h_file)
        FileClose($h_file2)
        $z = $z + 1
    EndIf
    If $z = 500 Then
        $n = $n + 1
        $z = 0
    EndIf
    $x = $x + 1
Until $x = 1500000
Edited by Jos
Fixed the code as indicated by Bowmore

Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

You have an issue with $File2 not using the file handle to close the file.

Try:

$x = 1400000
$z = 0
$n = 1
Do
    $h_file = FileOpen("C:\vote\" & $x & ".html", 0)


.
.
.
.
.
.
        $file2 = "c:\voter" & $n & ".txt"
        $h_File2 = FileOpen($file2, 1)
        FileWrite($file2, $id3 & "," & $lastvote3 & "," & $regdate3 & "," & $name3 & "," & $dob3 & "," & $street3 & ",  " & $city3 & "," & $poll3 & "," & $polladd3 & "," & $sd3 & "," & $citylim3 & "," & $cityward3 & "," & $status & "," & $Precinct3 & "," & $schoolzone3 & "," & $cd3 & "," & $jd3 & "," & $dc3 & "," & $ss3 & "," & $ssnum3 & "," & $jp3 & "," & $jpnum3 & "," & $precsplit3 & "," & $party3 & "," & $sr3 & "," & $srnum3 & @CRLF)
        FileClose($h_file)
        FileClose($h_File2)
        $z = $z + 1
    EndIf
    If $z = 500 Then
        $n = $n + 1
        $z = 0
    EndIf
    $x = $x + 1
Until $x = 1500000

You will need to change all instances of

FileReadLine($file,

to

FileReadLine($h_file,

to match Jos's change to the first file open.


"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites

yep... forgot changing that one but tried to show to use a "standard" variable for handles starting with $h_ ;)


Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0