Sign in to follow this  
Followers 0
Phaser

Storing data

14 posts in this topic

Hi everyone

I have 8500 sets of data in this format 01,99,83,27,12,30,114,299, and so on, I need to search this data so I thought of storing it in an array, $data[8500]=[01,99,83,27,12,30,114,299,....................] looks like an array wont hold that much data, any suggestions?

I will be searching it later with current data in chunks like this

$getdatamatch = "99,83"

Any pointers as to how I can proceed?

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Hi,

it's not really clear for me, what you have.

Have you 8500 numbers: 01,99,83,27,12,30........ -> amount 8500 -> This isn't a problem, arrays can store 2^24 (16 777 216).

or

have you 8500 datasets like:

1. dataset: 01,99,83,27,12,30........

2. dataset: 02,77,23,09,88 ....

.

.

.

8500 dataset: .....

Next question is, how you get your data / sets? From file, from database......

;-))

Stefan

Edited by 99ojo

Share this post


Link to post
Share on other sites

Hi Stefan

The data is currently in a .txt file so it is one long line of data seperated by a comma , I tried to put it all as an array

$data[8500]=[01,99,83,27,12,30,02,77,23,09,88........] ---->>>> 8500 total indices

but it gave an error in the array pointing to somewhere around 1500 indices into the array, which made me think the data was too long

Missing right bracket ')' in expression.:

Am I doing it correctly?

Share this post


Link to post
Share on other sites

Hi

I dont get anything out from that??

MsgBox(0,"box name", $aData)

I wasn't trying to read the txt file, as I had used all those numbers in the array that gave the error, wont the array be the best way so I can search it later?

Share this post


Link to post
Share on other sites

Phaser,

I would think that you are running into the AutoIt line limit of 4095 characters when you declare your array. At 3 chars per element, that would give you an error about 1500 elements in. :)

Try splitting the declaration like this - although you can make each "line" much longer:

$data[8500]=[01,99,83,27,12, _
    30,02,77,23,09,55,35,27, _
    88........]

Just make sure that each "line" is less that 4095 characters long. :idea:

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Hi,

add #include <array.au3> at the beginning of your script.

Change MsgBox(0,"box name", $aData) to _ArrayDispplay ($aData), because StringSplit returns an array.

Jumping on Tvern's code:

#include <array.au3>
$hFile = FileOpen("file.txt",)
$sData = FileRead($hFile)
FileClose($hFile)
$aData = StringSplit($sData,",")
_ArrayDispplay ($aData)

;-))

Stefan

Share this post


Link to post
Share on other sites

Hi, searching in arrays is slower than in strings.

You could "transform" the string in the file into an array with stringsplit()

$data = FileRead("test.txt")   ;file with CSV
$array=stringsplit($data,",",3) ;splits string into an array

$search = "02,77,23"  ;searchstring

;search in text
$pos = StringInStr($data, $search, 1, 1)
If $pos Then  ;string is found
    MsgBox(0, "StringInstr()", "Pattern " & $search & " found at position " & $pos)
Else
    MsgBox(0, "StringInstr()", "Pattern " & $search & " not found")
EndIf

;search
$searcharray=stringsplit($search,",",3)
$foundflag=0
for $i=0 to ubound($array)-1-ubound($searcharray)  ;all numbers in the array
    if $searcharray[0]=$array[$i] then ;first number match
for $x=1 to ubound($searcharray)-1   ;compare the numbers in the array with the numbers in the searchstring
if $searcharray[$x]<>$array[$i+$x] then ContinueLoop 2 ;if one of the following numbers doesnt match, then next number in array
        Next
        msgbox(0,"Data","Pattern " & $search & " found at array[" & $i&"] and following")
        $foundflag=1 ;flag that pattern has matched
        ExitLoop
    EndIf
next
if $foundflag=0 then msgbox(0,"Data","Pattern " & $search & " not found in array")
 

Share this post


Link to post
Share on other sites

Thank you very much AndyG, I think I can work with what you have posted

@ other members, thanks for helping out, AndyG's code may do everything I was after.

Share this post


Link to post
Share on other sites

Here is an array free example.

Local $sStr = ""
Local $sSearchFor = "34,35"

; $Data = FileRead("test.txt")
; or
For $i = 1 To 8500
    $sStr &= StringRight("0" & Random(1, 99, 1), 2) & ","
Next
$Data = StringTrimRight($sStr, 1) ; Remove last comma
ConsoleWrite($Data & @CRLF)


StringReplace($Data, $sSearchFor, $sSearchFor)
Local $iNum = @extended
Local $iFirstLocation = (StringInStr($Data, $sSearchFor) - 1) / 3

ConsoleWrite('"' & $sSearchFor & '" number of occurrences = ' & $iNum & @CRLF)
ConsoleWrite("First found position = " & $iFirstLocation & @CRLF)

MsgBox(0, "Results", '"' & $sSearchFor & '" number of occurrences = ' & $iNum & @CRLF & _
        "First found position = " & $iFirstLocation)

;or

If StringInStr($Data, $sSearchFor) Then
    MsgBox(0, "Results", '"' & $sSearchFor & '"' & ' found')
Else
    MsgBox(0, "Results", '"' & $sSearchFor & '"' & ' not found')
EndIf

Share this post


Link to post
Share on other sites

Hi Guys

Just had time to revisit this thread as I am not getting exaclty what I wanted, although I didn't actually mention it first time, AndyGs' method is what I am using currently and it gets the first found result, malkeys' finds how many times it was in the string, what I really need is to find the 2 number sequence and grab the next number, for example

string/array 12,86,44,21,79,33

search for 44,21

find it and grab 79 to use in my next calculation

The above is what I am using BUT if the next scenario is

string/array 12,86,44,21,79,33,01,51,44,21,28,93

search for 44,21

find it and grab 79 AND 28 to use in my next calculation

I am guessing an array is the only way to do this as I can grab the next key

$data[$i+2]; plus 2 as it works from the first number of the sequence

But for the life of me cannot figure out how to change AndyGs' code, any ideas?

Share this post


Link to post
Share on other sites

Phaser,

This will get the numbers following the first 2 matches with the pattern you define:

Global $aFound[2]

$sData = "12,86,44,21,79,33,01,51,44,21,28,93"

$aArray = StringSplit($sData, ",", 3) ;splits string into an array

$sSearch = "44,21" ;searchstring

; We will work on this copy of the data
$sSection = $sData

For $i = 0 To 1

    ;search in text
    $iPos = StringInStr($sSection, $sSearch, 1, 1)

    If $iPos Then ;string is found
        ; Determine start position of next number
        $iNext = $iPos + StringLen($sSearch) + 1
        $sNext = ""
        ; Add digits until next , or eol
        For $j = $iNext To $iNext + 5
            $sTemp = StringMid($sSection, $j, 1)
            If $sTemp = "," Or $sTemp = "" Then
                ; If we reach , or eol
                ExitLoop
            Else
                ; Add digit to number
                $sNext &= $sTemp
            EndIf
        Next
        $aFound[$i] = $sNext
        ; Strip the used numbers form the front of the string for teh next search
        $sSection = StringMid($sSection, $iNext)
    EndIf
Next
; Check if we got both values
If $aFound[1] <> "" Then
    MsgBox(0, "Next numbers", $aFound[0] & " - " & $aFound[1])
Else
    MsgBox(0, "StringInstr()", "Pattern " & $sSearch & " not found twice")
EndIf

Global $aFound[2]

;search
$sSearcharray = StringSplit($sSearch, ",", 3)
$iIndex = 0
For $i = 0 To UBound($aArray) - 1 - UBound($sSearcharray) ;all numbers in the array
    If $sSearcharray[0] = $aArray[$i] Then ;first number match
        For $x = 1 To UBound($sSearcharray) - 1 ;compare the numbers in the array with the numbers in the searchstring
            If $sSearcharray[$x] <> $aArray[$i + $x] Then ContinueLoop 2 ;if one of the following numbers doesnt match, then next number in array
        Next
        ; Add next element to array
        $iNext = $i + UBound($sSearcharray)
        $aFound[$iIndex] = $aArray[$iNext]
        $iIndex += 1
    EndIf
Next
; Check if we got both values
If $aFound[1] <> "" Then
    MsgBox(0, "Next numbers", $aFound[0] & " - " & $aFound[1])
Else
    MsgBox(0, "StringInstr()", "Pattern " & $sSearch & " not found twice")
EndIf

Does it do what you want? :mellow:

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

M23 as usual, perfect, however small nigle I think

C:\Program Files\AutoIt3\SciTE\phaser\template\datacapture.au3 (91) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:

$aFound[$iIndex] = $aArray[$iNext]

^ ERROR

The sequence of numbers can appear 1 or many times, on average it's about 5 to 7 times, I want the 5 to 7 following numbers and will probably do a looping bit of work on them, maybe push them to an array the walk through them, I just need to get the numbers initially, thanks again

MMM are there 2 seperate methods posted above? I am using the forst one, and have adjusted it to array size 5 and loop count 0 to 4 now it catches up to 5, can it be dynamic?

Edited by Phaser

Share this post


Link to post
Share on other sites

Phaser,

Declare the $aFound array to be bigger than the largest number of matches you may find - that will prevent the "subscript out of bounds" error. You will have to then walk through the array until you find an empty element to find the total number of matches - or use $iIndex in the second case.

And you will also need to change the If statement - perhaps checking that you have at least the minimum number of matches you require. :P

Let me know if you have any problems getting it to work. :mellow:

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0