Finding combinations and remove line

UEZ · January 22, 2012

Can you compress the 200mb file e.g. with 7Zip and upload it to a file hoster to make real tests?

Br,

UEZ

Read · January 22, 2012

water,

Some error, memory....

UEZ:

http://www.upload.ee/files/2008094/data.zip.html

water · January 22, 2012

Read and I did some tests. FileRead crashes with "Error allocating memory" when reading about 200000000 Bytes. Here are the results.

Reading 100.000 bytes is OK

Reading 1.000.000 Bytes isk OK too!

Reading 10.000.000 Bytes is OK!

Reading 100.000.000 Bytes is OK!

Reading 125.000.000 Bytes is OK!

Reading 127.000.000 Bytes is OK!

Reading 157.000.000 Bytes is OK!

Reading 177.000.000 Bytes is OK!

Reading 197.000.000 Bytes is OK!

Reading 199.000.000 Bytes is OK!

Reading 199.500.000 Bytes is OK!

Reading 199.750.000 Bytes is NOT OK!

Reading 199.900.000 Bytes is NOT OK!

I have absolutely no idea why there is a limit between 199.500.000 and 199.750.000 Bytes.

I would suspect it to be a number like 2**31 or 2**32.

Can anyone imagine what happens here?

kylomas · January 22, 2012

Read,

The following code will read through your "large" file once finding matches from a transaction file.

The number strings in the transaction file can be any length E.G "01 99" will match any string containing these two numbers at any position if they are in the order specified.

Finding 10 strings in a 1,000,000 line file takes about 110 seconds. You really need to solve your memory problem and use the solution that water has provided.

This code is functionally the same as code offered previously by Malkey except that I am using regexp for matching.

;
;
;
;

#include <array.au3>
#include <file.au3>

; Change the following to your file names

Local $trans_fl      = "c:tmptrans_fl.txt"                  ; file of number strings to delete from the master file
Local $mst_fl        = "c:tmpmst_fl.txt"                        ; file of all number strings (master file)
Local $mst_fl_new    = "c:tmpmst_fl_new.txt"                    ; new file of all number strings (master file)

;_gen_test_data()                                               ; generate test data for each file (leave this alone - was for testing)
_del_strings()                                                  ; delete number strings

Func _del_strings()

    Local $h_trans_fl                                           ; file handle for number strings to delete file (transaction file)
    Local $h_mst_fl                                             ; file handle for master file
    Local $h_mst_fl_new                                         ; file handle for new master file

    $h_trans_fl = FileOpen($trans_fl,0)                         ; open transaction file for read
    If $h_trans_fl = -1 Then _                                  ; if open error then
        MsgBox(0,'File Open Error','File = ' & $h_trans_fl)     ;    display message

    $h_mst_fl = FileOpen($mst_fl,0)                             ; open master file for read
    If $h_mst_fl = -1 Then _                                    ; if open error then
        MsgBox(0,'File Open Error','File = ' & $h_mst_fl)       ;    display message

    $h_mst_fl_new = FileOpen($mst_fl_new,2)                     ; open new master file for output
    If $h_mst_fl_new = -1 Then _                                ; if open error then
        MsgBox(0,'File Open Error','File = ' & $h_mst_fl_new)   ;    display message

    ; assumption - you have enough memory to read the entire transaction file and create an array from it

    Local $a_trans = StringSplit(FileRead($h_trans_fl),@crlf,3) ; create a zero based array from transaction records - do not store ele count at offset 0

    ; transform array entries into regex arguments

    For $i = 0 To UBound($a_trans) - 1
        If StringLen($a_trans[$i]) = 0 Then exitloop
        Local $a_tmp = StringSplit($a_trans[$i],' ',3)
        Local $s_tmp = ''
        For $j = 0 To UBound($a_tmp) - 1
            $s_tmp &= '(' & $a_tmp[$j] & ').+'
        Next
        $s_tmp = StringTrimRight($s_tmp,2)
        $a_trans[$i] = $s_tmp
    next

    ;_arraydisplay($a_trans)

    ; read through the master file once and delete records that match the transaction file

    Local $timer = TimerInit()

    While 1

        Local $s_mst_fl = FileReadLine($h_mst_fl)
        If @error = -1 Then exitloop

        Local $hit = 'n'

        For $i = 0 To UBound($a_trans) - 1

            If StringLen($a_trans[$i]) = 0 Then ContinueLoop

            If StringRegExp($s_mst_fl,$a_trans[$i]) = 1 Then
                ConsoleWrite('pattern = ' & $a_trans[$i] & ' matched record = ' & $s_mst_fl & @lf)
                $hit = 'y'
                exitloop
            endif

        next

        If $hit = 'n' Then FileWrite($h_mst_fl_new,$s_mst_fl & @crlf)

    WEnd

    ConsoleWrite(Round(TimerDiff($timer)/1000,2) & @lf)

    FileClose($h_trans_fl)
    FileClose($h_mst_fl_new)
    FileClose($h_mst_fl)
    Exit

endfunc

Func _gen_test_data()

    ; generate master file

    Local $a_10[1000001]
    For $i = 0 To UBound($a_10) - 1
        For $j = 1 To 6
            $a_10[$i] &= StringFormat("%02s",Random(1,99,1)) & ' '
        Next
    next
    _FileWriteFromArray($mst_fl,$a_10)

    ; generate transaction file

;~  Local $a_10[11]
;~  $timer = TimerInit()
;~  For $i = 0 To UBound($a_10) - 1
;~      For $j = 1 To 6
;~          $a_10[$i] &= StringFormat("%02s",Random(1,99,1)) & ' '
;~      Next
;~  next
;~  $timer = Round(TimerDiff($timer)/1000,2) & ' Seconds'
;~  _FileWriteFromArray($trans_fl,$a_10)

endfunc

Good Luck,

kylomas

water · January 22, 2012

@kylomas

I hope this definitely solves the problem!

kylomas · January 22, 2012

@water - can I have the code you used for the memory test? I would like to run it on my machine.

Win 7 Home Premium 64 bit

6 GB memory

Athlon II x4 620 processor

kylomas

kylomas · January 22, 2012

@water - yes, interesting problem and I learned a couple things. If not for that I would not have written this without 1 line of code from Read...

kylomas

water · January 22, 2012

@water - can I have the code you used for the memory test? I would like to run it on my machine

This is the code I've used to create the 200MB input file:

$hFileOut = FileOpen("C:\temp\FileOut.txt", 2)
Global $iStart = 1
Global $sString
For $j = 1 To 12299150
    $sString = ""
    For $i = 1 To 6
        $iNumber = Random($iStart, 93 + $i, 1)
        $iStart = $iNumber + 1
        $sString = $sString & StringRight("0" & $iNumber, 2) & " "
    Next
    FileWrite($hFileOut, StringLeft($sString, 17) & @LF)
Next
FileClose($hFileOut)

And here is the code I used to test the memory problem:

$hFileIn = FileOpen("C:\temp\FileOut.txt")
$sDataFile = FileRead($hFileIn, x)
MsgBox(16, "", @error)
FileClose($hFileIn)
Exit

Replace x with the number of bytes to read. We tested the numbers I posted in #63. The separators (".") are just to enhance readability.

Spiff59 · January 23, 2012

I, too, was unable to read your entire file in one swoop due to StringSplit (whether called directly, or from within _FileReadToArray) kicking out a memory allocation error. So, I was forced to use the line-by-line method.

Either way, I'd think building an array of deletion masks up front, then simply referencing the variables would be quicker than either a third nested loop, or a compound StringRegExp statement.

#include <file.au3>

Global $aDeletePatterns
_FileReadToArray("DeletePatterns.txt", $aDeletePatterns)
Global $aMasks[$aDeletePatterns[0] + 1][100]
For $i = 1 to $aDeletePatterns[0]
  $aTemp = StringSplit($aDeletePatterns[$i], " ")
  For $j = 1 to $aTemp[0]
    $aMasks[$i][$aTemp[$j]] = 1
  Next
Next

$hDataFile = FileOpen("DataFile.txt")
$hFileOut = FileOpen("DataFileOut.txt", 2)
While 1
  $sDataLine = FileReadLine($hDataFile)
  If @error = -1 Then ExitLoop
  $aTemp = StringSplit($sDataLine, " ")
  For $i = 1 To $aDeletePatterns[0]
    If $aMasks[$i][$aTemp[1]] + $aMasks[$i][$aTemp[2]] + $aMasks[$i][$aTemp[3]] + $aMasks[$i][$aTemp[4]] + $aMasks[$i][$aTemp[5]]  + $aMasks[$i][$aTemp[6]] = 5 Then
        $sDataLine = ""
        ExitLoop
    Endif
  Next
  If $sDataLine <> "" Then FileWriteLine($hFileOut, $sDataLine)
WEnd

FileClose($hDataFile)
FileClose($hFileOut)

typo

Edited January 24, 2012 by Spiff59

water · January 23, 2012

I, too, was unable to read your entire file in one swoop due to StringSplit (whether called directly, or from within _FileReadToArray) kicking out a memory allocation error

On the OPs PC already the FileRead kicked out the memory error when reading more than 199.500.000 Bytes.

Seems to be some kind of OS related limitation.

kylomas · January 23, 2012

Water,

It is autoIT that is issueing the error. For what reason, don't know...

I found doc in the help file under the variable types section indicating that the max string size = 2 GB. When I run the following code I get mem error at about 500 MB

#include <string.au3>

Local $block

For $i = 0 To 4095
    $block &= 0
Next
ConsoleWrite(StringLen($block) & @lf)

For $i = 0 To 255
    For $j = 0 To 255
        For $k = 0 To 255
            $block &= $block
            ConsoleWrite(StringLen($block) & @lf)
        Next
    Next
next

Seems to be an incongruity here.

kylomas

Spiff59 · January 23, 2012

When I run the following code I get mem error at about 500 MB

I don't even get that far...

The last report I get out of this is "128MB":

Local $block = "x", $x, $bytes, $bytes_suffix[5] = [" B"," KB"," MB"," GB"," TB"]
While 1
    $block &= $block
    $bytes = StringLen($block)
    $x = 0
    While $bytes > 1023
        $x += 1
        $bytes /= 1024
    WEnd
    ToolTip(Round($bytes, 2) & $bytes_suffix[$x])
    Sleep(100)
WEnd

So I guess it's blowing when trying to allocate a 256MB string variable.

I'm XP Pro SP3 with 2GB RAM. Task manager shows 1.55GB free at runtime.

Anyone have a pre-3.8 version of Autoit? Does this behavior exhibit itself on older versions?

Maybe a Dev ought to look at this...

water · January 23, 2012

I get the following results:

AutoIt Version: 3.3.8.0, Windows 7 64 Bit, 4 GB RAM: 512 MB
AutoIt Version: 3.3.6.0, Windows XP 32 Bit, 2 GB RAM: 128 MB

Spiff59 · January 23, 2012

I get the following results:

AutoIt Version: 3.3.8.0, Windows 7 64 Bit, 4 GB RAM: 512 MB
AutoIt Version: 3.3.6.0, Windows XP 32 Bit, 2 GB RAM: 128 MB

In 3.3.6.0 also!

Interesting it never came up.

Either:

1. The statements "Maximum string length: 2,147,483,647 characters" in the FAQ and "Strings can contain up to 2147483647 characters." in the helpfile are wrong.

2. Something has been broken at least since version 3.3.6.0

3. Or, a certain Dev will tell us we're doing something stupid

Edited January 23, 2012 by Spiff59

kylomas · January 23, 2012

spiff59,

Ran your code and barf at 512 MB...

Version 3.8 running 32 bit version of autoIT on 64 bit win 7 OS.

kylomas

water · January 23, 2012

I'm going to start a new thread asking DEVs for more insight information and pointing them to this thread.

Maybe they can shed some light on this subject

PsaltyDS mentioned in this that the problem we encounter might be caused by Windows memory management.

Edited January 23, 2012 by water

water · January 24, 2012

Yashied said:

"The maximum string size equal to the maximum contiguous memory block that is available for your program (by default 2 GB). Since the memory can be defragmented, the maximum size of the memory block is smaller than 2 GB."

So you have to read the file in chunks.

Spiff59 · January 24, 2012

Just for grins, I downloaded your full data files and ran them through the code in post #69.

Result: 599 seconds... 10 minutes, on my slow machine here at work.

kylomas · January 24, 2012

spiff59,

Ditto- results form code in #64 using the diownloaded files

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:DAutoitSourcenumber array delete element without gui.au3" /autoit3dir "C:Program Files (x86)AutoIt3" /UserParams  
+>15:47:20 Starting AutoIt3Wrapper v.2.0.3.0    Environment(Language:0409  Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64)
>Running AU3Check (1.54.22.0)  from:C:Program Files (x86)AutoIt3
+>15:47:20 AU3Check ended.rc:0
>Running:(3.3.8.0):C:Program Files (x86)AutoIt3autoit3.exe "C:DAutoitSourcenumber array delete element without gui.au3"  
pattern = (10).+(25).+(30).+(33).+(44) matched record = 01 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 02 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 03 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 04 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 05 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 06 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 07 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 08 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 09 10 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 11 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 12 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 13 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 14 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 15 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 16 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 17 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 18 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 19 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 20 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 21 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 22 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 23 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 24 25 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 26 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 27 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 28 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 29 30 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 31 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 32 33 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 34 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 35 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 36 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 37 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 38 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 39 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 40 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 41 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 42 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 43 44
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 44 45
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 44 46
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 44 47
pattern = (10).+(25).+(30).+(33).+(44) matched record = 10 25 30 33 44 48
375.25
+>15:53:36 AutoIT3.exe ended.rc:0
>Exit code: 0   Time: 376.743

kylomas

Edited January 24, 2012 by kylomas

kylomas · January 24, 2012

spiff59, water, et al,

I loaded the test file to a string variable w/o memory problems and run this code

#include <array.au3>

Local $timer    = TimerInit()
Local $s10      = FileRead("c:tmpdatafile.txt")
ConsoleWrite('Time to read file to string variable = ' & Round(TimerDiff($timer)/1000,2) & ' seconds' & @lf)

Local $a_trans = StringSplit(FileRead("c:tmpdeletepatterns.txt"),@crlf,3)

; transform array entries into regex arguments

For $i = 0 To UBound($a_trans) - 1
    If StringLen($a_trans[$i]) = 0 Then exitloop
    Local $a_tmp = StringSplit($a_trans[$i],' ',3)
    Local $s_tmp = ''
    For $j = 0 To UBound($a_tmp) - 1
        If StringLen($a_tmp[$j]) = 0 Then exitloop
        $s_tmp &= '(' & $a_tmp[$j] & ').+'
    Next
    $s_tmp = StringTrimRight($s_tmp,2)
    $a_trans[$i] = $s_tmp
next

Local $timer = TimerInit()

For $i = 0 To UBound($a_trans) - 1

    If StringLen($a_trans[$i]) = 0 Then exitloop

    Local $a_10 = StringRegExpReplace($s10,$a_trans[$i],"")
    If @error = 2 Then
        ConsoleWrite('error in expression at offset = ' & @extended & @lf)
    else
        ConsoleWrite('Number of strings replaced = ' & @extended & @lf)
    endif

next

ConsoleWrite('Time to perform replacements = ' & Round(TimerDiff($timer)/1000,2) & ' seconds' & @lf)

$timer = TimerInit()

FileWrite("c:tmpdatafile_new.txt",$s10)

ConsoleWrite('Time to write new file       = ' & Round(TimerDiff($timer)/1000,2) & ' seconds' & @lf)

which produced these results

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:UsersDadDesktopmemtst.au3" /autoit3dir "C:Program Files (x86)AutoIt3" /UserParams    
+>16:28:02 Starting AutoIt3Wrapper v.2.0.3.0    Environment(Language:0409  Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64)
>Running AU3Check (1.54.22.0)  from:C:Program Files (x86)AutoIt3
+>16:28:02 AU3Check ended.rc:0
>Running:(3.3.8.0):C:Program Files (x86)AutoIt3autoit3.exe "C:UsersDadDesktopmemtst.au3"    
Time to read file to string variable = 9.05 seconds
Number of strings replaced = 43
Time to perform replacements = 6.88 seconds
Time to write new file       = 2.65 seconds
+>16:28:21 AutoIT3.exe ended.rc:0
>Exit code: 0    Time: 20.222

I thought that the simplicity of this might amuse you guys..

kylomas

Finding combinations and remove line

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members