BIG String to Array

[Reizvoller]

Greetings,

I am getting the "RawData" return from FastFind's "FFGetRawData" function and then attempting to make it into an array.

What I have here does the job as it should but it is a slow process. I feel I may be overlooking a more effecient way to handle this matter. In the end ( in an ideal situation ) I'd like to end up with a 2D array with 580 col and 375 rows. Currently it's just a 1D array as I am still trying to get my head around how to make it into a 2D. BUT... My question is this ...

Is there a better way to handle what I am trying to do? 

Func trimmer ()
    Local $i = 0 ;The "Counter"
    Local $WorkAreaLength = StringLen ($WorkAreaData) ;Get the length of the string
    ConsoleWrite ("WorkAreaLegnth="&$WorkAreaLength&@CR)
    Local $iWorkAreaLengthCount = $WorkAreaLength / 8 ;How many segments of 8 can we make? Use this for the counter later on
    ConsoleWrite ("WorkArealengthCount="&$iWorkAreaLengthCount&@CR) 
    Local $sPiece = 0 ; The string segment holder
    Global $aWorkArea ; The array to write to
        Do
            $sPiece = StringLeft ($WorkAreaData, 8) ; Grab the first 8 characters and put them into $sPiece
            _ArrayAdd ($aWorkArea, $sPiece) ; Put what $sPiece is into the array
            StringTrimLeft ($WorkAreaData, 8) ;Trim off the first 8 characters that were just copied

            $i = $i + 1 ;Count up 1
            ConsoleWrite ($i&@CR)
        Until $i >= $iWorkAreaLengthCount ; Keep going until $i >= $iWorkAreaLengthCount

    ConsoleWrite ("Done"&@CR)
    Local $sFile = FileOpen ("main.txt");Open the text file
    _FileWriteFromArray ($sFile , $aWorkArea);Write the array to the text file
    FileClose ($sFile); Close the text file
EndFunc

 

A little secondary problem, that isn't throwing any errors to speak of, is the file is never written to. It remains blank even though it is in the same directory as the script and the script isn't responding with any errors. This isn't my main focus though as it just lets me see what the array is rather than using the console which can be a little cramped at times.

Any advise would be appreciated, thank you!

[alien4u]

$sPiece = StringLeft ($WorkAreaData, +8)

Hi @Reizvoller

I don't know how BIG could be your string but sometimes we just over complicate things.

Like I see it to save your string to an Array 1D or 2D you need to split that string, or you need some kind of delimiter.
So you split your string on segments of 8 character each one, because you don't have any other way to split your string? no delimiters? spaces? anything?

Anyways in order to make this more efficient what I would probably do is to avoid another string operation:

StringTrimLeft ($WorkAreaData, 8)

Try to avoid this operation, maybe if you sum first position to second one?

$sPiece = StringLeft ($WorkAreaData, 8)

Maybe If $i >= 1 Then

$sPiece = StringLeft ($WorkAreaData, +8)

I mean increment StringLeft position by 8 because is always your increment factor, on each loop cicle you could save next StringLeft left position and use that.

I thinks avoiding StringTrimLeft will give you better performance.

Kind Regards
Alien.

[Reizvoller]

Greetings,

Fantastic idea regarding not doing the "StringTrim" and the subsequent +8 on the StringLeft.

Truth be told I am starting to feel like I am attempting to do something I shouldn't.

The string is a bunch of Hexadecimal color values all in a line without any spaces further than a "00" but some colors have "00" in them hence me not using it as a delimiter for this process.

The string is a bit over 1.7million characters which is off of about 217500 pixel values.

This process is to hunt through a work area, see if there is something, then continue along.

After putting more thought into it I feel I need to find another method to tackle my goal but this was great learning along the way!

[alien4u]

Anyways if this are Hexadecimal color values always with a fixed length for example 4 characters you could take a high multiple of 4.

For example 400 this will make each array element store 100 colors values and this will make the arrays also shorten for parsing in the next stage of your code.

Kind Regards
Alien.

[czardas]

Perhaps this helps:

#include <Array.au3>
Local $aArray = _StringEqualSplit("12345678910123456789101234567890", 8)
_ArrayDisplay($aArray)

Func _StringEqualSplit($sString, $iNumChars)
    If Not IsString($sString) Or $sString = "" Then Return SetError(1, 0, 0)
    If Not IsInt($iNumChars) Or $iNumChars < 1 Then Return SetError(2, 0, 0)
    Return StringRegExp($sString, "(?s).{1," & $iNumChars & "}", 3)
EndFunc

 

Edited April 8, 2016 by czardas

[markyrocks]

$file=fileopen ("file.txt",1)

 

Your file is opening in read only mode

 

Edit.  I believe you could use 0x as delimiter then just add it back later

2d array simple example 

Dim $array[10][10]

For $col=0 to ubound($array,2)-1
For $row=0 to ubound($array1)-1
$array[$row][$col]
MsgBox("","", $$array[$row][$col])
;this basically just shuffle through the array and display the results

Next
Next

 

Edited April 9, 2016 by markyrocks
made a mistake

[AutoBert]

This funcs do the job:

expandcollapse popup

Func _FFArrayFromRawData($String, $iRows = 0, $iCols = 0)
    #cs
        _FFArrayFromRawData($String, [$iRows = 0, [$iCols = 0]])
        Params:     $String     the String with raw pixel data as returned from _FFArrayFromRawData
        $iRows      number of Rows in resulting array OPTIONAL
        $iCols      number of Cols in resulting array OPTIONAL
        possible
        Returns:    $aResult[$iPixelCount+1]    $iRows and $iCols = 0
        so all Pixels in a 1D array
        $aResult[$iRows]            1D array
        $aResult[$iRows][$iCols]    2D array
        also
        setting:    @Error      (0 no error, 1 error occured)
        @extended   10 no valid RawDataString
        11 Arraydimensions to big
        1 Array is 1D (info only)
        2 Array is 2D (info only)

        Author:     HarryBotter 2016
    #ce
    Local $aResult[0], $iPixNo, $iEx
    Local $iLen = StringLen($String)
    If Mod($iLen, 8) Then ;no valid RawDataString
        SetError(1, 10)
        Return
    EndIf
    If $iRows * $iCols * 8 > $iLen Then ;Arraydimensions to big
        SetError(1, 11)
        Return
    EndIf
    If $iRows = 0 And $iCols = 0 Then
        $iRows = $iLen / 8
        ReDim $aResult[$iRows]
        $iEx = 1
    Else
        If $iCols = 0 Then
            ReDim $aResult[$iRows]
            $iEx = 1
        Else
            ReDim $aResult[$iRows][$iCols]

            $iEx = 2
        EndIf
    EndIf
    For $i = 0 To $iRows - 1
        If $iEx > 1 Then
            For $j = 0 To $iCols - 1
                $iPixNo = $i * $iCols + $j
                $aResult[$i][$j] = _FFGetPixelColor($iPixNo, $String)
            Next
        Else
            $aResult[$i] = _FFGetPixelColor($i, $String)
        EndIf
    Next
    SetError(0, $iEx, $aResult)
    Return $aResult
EndFunc   ;==>_FFArrayFromRawData

Func _FFGetPixelColor($iPixNo, $String)
    #cs
        Params:     $iPixNo     the number of Pixel for wich the color is extracted
        $String     the String with raw pixel data as returned from _FFArrayFromRawData

        Remarks:    no Errorhandling only for using together with _FFArrayFromRawData

        Author:     HarryBotter 2016
    #ce
    Local $sRet = StringMid($String, $iPixNo * 8 + 1, 8)
    ;ConsoleWrite($iPixNo & ': ' & $sRet & @CRLF)
    Return $sRet
EndFunc   ;==>_FFGetPixelColor

i already posted you the link to script using this:

#include "FastFind.au3"
#include <Array.au3>

$tdStart = TimerInit()
FFSnapShot(1, 1, 100, 100, 1)
ConsoleWrite('SnapShot after '&TimerDiff($tdStart)&' ms returned with: '  & @error & @CRLF)
Local $sData = FFGetRawData(1)
ConsoleWrite('GetRawData: ' & @error & @CRLF)
Local $aPixels = _FFArrayFromRawData($sData, 100, 100)
If @extended=2 Then
    $iPixelCount=UBound($aPixels,1)*UBound($aPixels,1)
Else
    $iPixelCount=UBound($aPixels)
EndIf

ConsoleWrite('ArrayFromRawData: ' & @error & @TAB & @extended & @TAB & TimerDiff($tdStart) & ' ms'& @CRLF)
ConsoleWrite('Average Time: ' & TimerDiff($tdStart) / $iPixelCount &'ms Pixels: ' & $iPixelCount & @CRLF)
_ArrayDisplay($aPixels)

Without the _ArrayDisplay its very fast. Using smaller workarea will save absolut time.

Edited April 8, 2016 by AutoBert

[Reizvoller]

3 hours ago, alien4u said:

Anyways if this are Hexadecimal color values always with a fixed length for example 4 characters you could take a high multiple of 4.

For example 400 this will make each array element store 100 colors values and this will make the arrays also shorten for parsing in the next stage of your code.

Kind Regards
Alien.

Shortening them would indeed be handy!

3 hours ago, czardas said:

Perhaps this helps:

#include <Array.au3>
Local $aArray = _StringEqualSplit("12345678910123456789101234567890", 8)
_ArrayDisplay($aArray)

Func _StringEqualSplit($sString, $iNumChars)
    If Not IsString($sString) Or $sString = "" Then Return SetError(1, 0, 0)
    If Not IsInt($iNumChars) Or $iNumChars < 1 Then Return SetError(2, 0, 0)
    Return StringRegExp($sString, "(?s).{1," & $iNumChars & "}", 3)
EndFunc

 

This is an interesting approach! I am going to try it right away, even if it is just for the learning experience

2 hours ago, markyrocks said:

$file=fileopen ("file.txt",1)

 

Your file is opening in read only mode

 

Edit.  I believe you could use 0x as delimiter then just add it back later

2d array simple example 

Dim $array[10][10]

For $col=0 to ubound($array)-1
For $row=0 to ubound($array)-1
$array[$row][$col]
MsgBox("","", $$array[$row][$col])
;this basically just shuffle through the array and display the results

Next
Next

 

I KNEW I was over looking something super simple with the FileWrite! I was staring at it going "There has gotta be a setting I am just not even thinking of..." Thank you! 

As for using 0x , I actually don't think that the function FFGetRawData throws out the 0x too or else this would be ideal.

19 minutes ago, AutoBert said:

This funcs do the job:

expandcollapse popup

Func _FFArrayFromRawData($String, $iRows = 0, $iCols = 0)
    #cs
        _FFArrayFromRawData($String, [$iRows = 0, [$iCols = 0]])
        Params:     $String     the String with raw pixel data as returned from _FFArrayFromRawData
        $iRows      number of Rows in resulting array OPTIONAL
        $iCols      number of Cols in resulting array OPTIONAL
        possible
        Returns:    $aResult[$iPixelCount+1]    $iRows and $iCols = 0
        so all Pixels in a 1D array
        $aResult[$iRows]            1D array
        $aResult[$iRows][$iCols]    2D array
        also
        setting:    @Error      (0 no error, 1 error occured)
        @extended   10 no valid RawDataString
        11 Arraydimensions to big
        1 Array is 1D (info only)
        2 Array is 2D (info only)

        Author:     HarryBotter 2016
    #ce
    Local $aResult[0], $iPixNo, $iEx
    Local $iLen = StringLen($String)
    If Mod($iLen, 8) Then ;no valid RawDataString
        SetError(1, 10)
        Return
    EndIf
    If $iRows * $iCols * 8 > $iLen Then ;Arraydimensions to big
        SetError(1, 11)
        Return
    EndIf
    If $iRows = 0 And $iCols = 0 Then
        $iRows = $iLen / 8
        ReDim $aResult[$iRows]
        $iEx = 1
    Else
        If $iCols = 0 Then
            ReDim $aResult[$iRows]
            $iEx = 1
        Else
            ReDim $aResult[$iRows][$iCols]

            $iEx = 2
        EndIf
    EndIf
    For $i = 0 To $iRows - 1
        If $iEx > 1 Then
            For $j = 0 To $iCols - 1
                $iPixNo = $i * $iCols + $j
                $aResult[$i][$j] = _FFGetPixelColor($iPixNo, $String)
            Next
        Else
            $aResult[$i] = _FFGetPixelColor($i, $String)
        EndIf
    Next
    SetError(0, $iEx, $aResult)
    Return $aResult
EndFunc   ;==>_FFArrayFromRawData

Func _FFGetPixelColor($iPixNo, $String)
    #cs
        Params:     $iPixNo     the number of Pixel for wich the color is extracted
        $String     the String with raw pixel data as returned from _FFArrayFromRawData

        Remarks:    no Errorhandling only for using together with _FFArrayFromRawData

        Author:     HarryBotter 2016
    #ce
    Local $sRet = StringMid($String, $iPixNo * 8 + 1, 8)
    ;ConsoleWrite($iPixNo & ': ' & $sRet & @CRLF)
    Return $sRet
EndFunc   ;==>_FFGetPixelColor

i already posted you the link unsing to script using this in

I need to save CodeBot to my favorites , you have referenced it before and it seems like a valuable source of data. Your solution is very intricuit and will take a little time for me to review and employ for myself. Thank you for the information!

<><><>

So many replies to this little question of mine, I really apreciate every moment that was put into writing back to me. Thank you all very much! I am headed out of the office for the weekend but will be back on this problem ( and probably this forum ) come Monday

Thank you again!~

~~~Reiz

[AutoBert]

5 hours ago, alien4u said:

Anyways if this are Hexadecimal color values always with a fixed length for example 4 characters you could take a high multiple of 4.

It's a String each Pixel is represented in BGRA-Format 2 chars for Blue, 2 chars for green, 2 chars for red and 2 chars for the alphachannel

 

2 hours ago, Reizvoller said:

Your solution is very intricuit and will take a little time for me to review and employ for myself.

It's not mine. I had a small look in FastFind's helpfile and i think there a better solutions possible than creating a array and search with AutoItscript for a pixel with specific color. Using the wrapped DLL-calls like FFGetPixel, FFNearestPixel, FFBestSpot should be perfomater you don't need the 2 Steps:

• FFGetRawData
• creating a array from resultstring for easier searching

and then searching in the array. If knowing the coords for a pixel you want know the colour use FFGetPixel. If you want the nearest pixel from a given coordinate in defined colour (shadevariantions are possible) user FFNearesPixel. For a Spot of Pixels you can use FFBestSpot, FFNearestSpot.

As this functions are wrapped DLL-calls for the FF.DLL it's much faster than the best optimized AutoIt code. FF.au3 is very complex and so it needs time to explore all the possiblilies. But this time is not wasted for you, not using a FF'-func when it's possible is wasted time. Only in case you know how to include and execute ASM-routines in autoit (I don't) you can build faster algorithm than the FF-funcs.

[Reizvoller]

On 4/8/2016 at 7:06 PM, AutoBert said:

It's a String each Pixel is represented in BGRA-Format 2 chars for Blue, 2 chars for green, 2 chars for red and 2 chars for the alphachannel

 

It's not mine. I had a small look in FastFind's helpfile and i think there a better solutions possible than creating a array and search with AutoItscript for a pixel with specific color. Using the wrapped DLL-calls like FFGetPixel, FFNearestPixel, FFBestSpot should be perfomater you don't need the 2 Steps:

• FFGetRawData
• creating a array from resultstring for easier searching

and then searching in the array. If knowing the coords for a pixel you want know the colour use FFGetPixel. If you want the nearest pixel from a given coordinate in defined colour (shadevariantions are possible) user FFNearesPixel. For a Spot of Pixels you can use FFBestSpot, FFNearestSpot.

As this functions are wrapped DLL-calls for the FF.DLL it's much faster than the best optimized AutoIt code. FF.au3 is very complex and so it needs time to explore all the possiblilies. But this time is not wasted for you, not using a FF'-func when it's possible is wasted time. Only in case you know how to include and execute ASM-routines in autoit (I don't) you can build faster algorithm than the FF-funcs.

Howdy!

I applied the idea of  using just PixelSearch for another process for the same web interface. Trouble is, with this next step, there are many items that will have the color I am looking for and the location is going to change. This problem is actually why I am making this little script because this particular task is very cumbersome.

The entire thing, the work space,  renders as a big BMP and is not searchable as text. 

 

So the strategy was this ...

1. Get reference RAW data for the areas of interest that I want to automatically navigate to/find
   1. Get a snapShot of the wanted area, convert to RAW, save on a .txt
2. Open the work area
3. Snap shot work area
4. Search for the reference RAW in the WorkArea SnapShot RAW
5. Scroll down x units 
6. Go back to step 2

It isn't the most elegant of solutions but with the interface I have, it's really my only option ( that I can see at the moment. )

I suppose the better way would be to find a more finite way to specify the target areas I am wanting the script to find. Like a 1 pixel tall by 100 pixels wide rather than an area of 300x500 pixels.

I am going to see about finding other ways to search through this thing in the meantime. I think that my described method is too lengthy of a process to really be useful when compared against just manual searching. 

Edited April 12, 2016 by Reizvoller

[AutoBert]

You don't need FFGetRawData, FFGetPixel searches in SnapShot too and this much faster. I think (not tested, i am no gamer or gamebotter) that building 2DArray with FFGetPixel is Faster than the way Hairy Botter showed. 

[Reizvoller]

23 minutes ago, AutoBert said:

You don't need FFGetRawData, FFGetPixel searches in SnapShot too and this much faster. I think (not tested, i am no gamer or gamebotter) that building 2DArray with FFGetPixel is Faster than the way Hairy Botter showed. 

Quick note this is NOT for a game in any capacity! 

But your information is quite useful and will be tested once I get the chance, thank you.