Jump to content
Reizvoller

BIG String to Array

Recommended Posts

Reizvoller

Greetings,

I am getting the "RawData" return from FastFind's "FFGetRawData" function and then attempting to make it into an array.

What I have here does the job as it should but it is a slow process. I feel I may be overlooking a more effecient way to handle this matter. In the end ( in an ideal situation ) I'd like to end up with a 2D array with 580 col and 375 rows. Currently it's just a 1D array as I am still trying to get my head around how to make it into a 2D. BUT... My question is this ...

Is there a better way to handle what I am trying to do? 

Func trimmer ()
    Local $i = 0 ;The "Counter"
    Local $WorkAreaLength = StringLen ($WorkAreaData) ;Get the length of the string
    ConsoleWrite ("WorkAreaLegnth="&$WorkAreaLength&@CR)
    Local $iWorkAreaLengthCount = $WorkAreaLength / 8 ;How many segments of 8 can we make? Use this for the counter later on
    ConsoleWrite ("WorkArealengthCount="&$iWorkAreaLengthCount&@CR) 
    Local $sPiece = 0 ; The string segment holder
    Global $aWorkArea ; The array to write to
        Do
            $sPiece = StringLeft ($WorkAreaData, 8) ; Grab the first 8 characters and put them into $sPiece
            _ArrayAdd ($aWorkArea, $sPiece) ; Put what $sPiece is into the array
            StringTrimLeft ($WorkAreaData, 8) ;Trim off the first 8 characters that were just copied

            $i = $i + 1 ;Count up 1
            ConsoleWrite ($i&@CR)
        Until $i >= $iWorkAreaLengthCount ; Keep going until $i >= $iWorkAreaLengthCount

    ConsoleWrite ("Done"&@CR)
    Local $sFile = FileOpen ("main.txt");Open the text file
    _FileWriteFromArray ($sFile , $aWorkArea);Write the array to the text file
    FileClose ($sFile); Close the text file
EndFunc

 

A little secondary problem, that isn't throwing any errors to speak of, is the file is never written to. It remains blank even though it is in the same directory as the script and the script isn't responding with any errors. This isn't my main focus though as it just lets me see what the array is rather than using the console which can be a little cramped at times.

Any advise would be appreciated, thank you!

Share this post


Link to post
Share on other sites
alien4u
$sPiece = StringLeft ($WorkAreaData, +8)

Hi @Reizvoller

I don't know how BIG could be your string but sometimes we just over complicate things.

Like I see it to save your string to an Array 1D or 2D you need to split that string, or you need some kind of delimiter.
So you split your string on segments of 8 character each one, because you don't have any other way to split your string? no delimiters? spaces? anything?

Anyways in order to make this more efficient what I would probably do is to avoid another string operation:

StringTrimLeft ($WorkAreaData, 8)

Try to avoid this operation, maybe if you sum first position to second one?

$sPiece = StringLeft ($WorkAreaData, 8)

Maybe If $i >= 1 Then

$sPiece = StringLeft ($WorkAreaData, +8)

I mean increment StringLeft position by 8 because is always your increment factor, on each loop cicle you could save next StringLeft left position and use that.

I thinks avoiding StringTrimLeft will give you better performance.

Kind Regards
Alien.

Share this post


Link to post
Share on other sites
Reizvoller

Greetings,

Fantastic idea regarding not doing the "StringTrim" and the subsequent +8 on the StringLeft.

Truth be told I am starting to feel like I am attempting to do something I shouldn't.

The string is a bunch of Hexadecimal color values all in a line without any spaces further than a "00" but some colors have "00" in them hence me not using it as a delimiter for this process.

The string is a bit over 1.7million characters which is off of about 217500 pixel values.

This process is to hunt through a work area, see if there is something, then continue along.

After putting more thought into it I feel I need to find another method to tackle my goal but this was great learning along the way! :)

Share this post


Link to post
Share on other sites
alien4u

Anyways if this are Hexadecimal color values always with a fixed length for example 4 characters you could take a high multiple of 4.

For example 400 this will make each array element store 100 colors values and this will make the arrays also shorten for parsing in the next stage of your code.

Kind Regards
Alien.

Share this post


Link to post
Share on other sites
czardas

Perhaps this helps:

#include <Array.au3>
Local $aArray = _StringEqualSplit("12345678910123456789101234567890", 8)
_ArrayDisplay($aArray)

Func _StringEqualSplit($sString, $iNumChars)
    If Not IsString($sString) Or $sString = "" Then Return SetError(1, 0, 0)
    If Not IsInt($iNumChars) Or $iNumChars < 1 Then Return SetError(2, 0, 0)
    Return StringRegExp($sString, "(?s).{1," & $iNumChars & "}", 3)
EndFunc

 

Edited by czardas

Share this post


Link to post
Share on other sites
markyrocks

$file=fileopen ("file.txt",1)

 

Your file is opening in read only mode

 

Edit.  I believe you could use 0x as delimiter then just add it back later

2d array simple example 

Dim $array[10][10]

For $col=0 to ubound($array,2)-1
For $row=0 to ubound($array1)-1
$array[$row][$col]
MsgBox("","", $$array[$row][$col])
;this basically just shuffle through the array and display the results

Next
Next

 

Edited by markyrocks
made a mistake

Share this post


Link to post
Share on other sites
AutoBert

This funcs do the job:

Func _FFArrayFromRawData($String, $iRows = 0, $iCols = 0)
    #cs
        _FFArrayFromRawData($String, [$iRows = 0, [$iCols = 0]])
        Params:     $String     the String with raw pixel data as returned from _FFArrayFromRawData
        $iRows      number of Rows in resulting array OPTIONAL
        $iCols      number of Cols in resulting array OPTIONAL
        possible
        Returns:    $aResult[$iPixelCount+1]    $iRows and $iCols = 0
        so all Pixels in a 1D array
        $aResult[$iRows]            1D array
        $aResult[$iRows][$iCols]    2D array
        also
        setting:    @Error      (0 no error, 1 error occured)
        @extended   10 no valid RawDataString
        11 Arraydimensions to big
        1 Array is 1D (info only)
        2 Array is 2D (info only)

        Author:     HarryBotter 2016
    #ce
    Local $aResult[0], $iPixNo, $iEx
    Local $iLen = StringLen($String)
    If Mod($iLen, 8) Then ;no valid RawDataString
        SetError(1, 10)
        Return
    EndIf
    If $iRows * $iCols * 8 > $iLen Then ;Arraydimensions to big
        SetError(1, 11)
        Return
    EndIf
    If $iRows = 0 And $iCols = 0 Then
        $iRows = $iLen / 8
        ReDim $aResult[$iRows]
        $iEx = 1
    Else
        If $iCols = 0 Then
            ReDim $aResult[$iRows]
            $iEx = 1
        Else
            ReDim $aResult[$iRows][$iCols]

            $iEx = 2
        EndIf
    EndIf
    For $i = 0 To $iRows - 1
        If $iEx > 1 Then
            For $j = 0 To $iCols - 1
                $iPixNo = $i * $iCols + $j
                $aResult[$i][$j] = _FFGetPixelColor($iPixNo, $String)
            Next
        Else
            $aResult[$i] = _FFGetPixelColor($i, $String)
        EndIf
    Next
    SetError(0, $iEx, $aResult)
    Return $aResult
EndFunc   ;==>_FFArrayFromRawData

Func _FFGetPixelColor($iPixNo, $String)
    #cs
        Params:     $iPixNo     the number of Pixel for wich the color is extracted
        $String     the String with raw pixel data as returned from _FFArrayFromRawData

        Remarks:    no Errorhandling only for using together with _FFArrayFromRawData

        Author:     HarryBotter 2016
    #ce
    Local $sRet = StringMid($String, $iPixNo * 8 + 1, 8)
    ;ConsoleWrite($iPixNo & ': ' & $sRet & @CRLF)
    Return $sRet
EndFunc   ;==>_FFGetPixelColor

i already posted you the link to script using this:

#include "FastFind.au3"
#include <Array.au3>

$tdStart = TimerInit()
FFSnapShot(1, 1, 100, 100, 1)
ConsoleWrite('SnapShot after '&TimerDiff($tdStart)&' ms returned with: '  & @error & @CRLF)
Local $sData = FFGetRawData(1)
ConsoleWrite('GetRawData: ' & @error & @CRLF)
Local $aPixels = _FFArrayFromRawData($sData, 100, 100)
If @extended=2 Then
    $iPixelCount=UBound($aPixels,1)*UBound($aPixels,1)
Else
    $iPixelCount=UBound($aPixels)
EndIf

ConsoleWrite('ArrayFromRawData: ' & @error & @TAB & @extended & @TAB & TimerDiff($tdStart) & ' ms'& @CRLF)
ConsoleWrite('Average Time: ' & TimerDiff($tdStart) / $iPixelCount &'ms Pixels: ' & $iPixelCount & @CRLF)
_ArrayDisplay($aPixels)

Without the _ArrayDisplay its very fast. Using smaller workarea will save absolut time.

Edited by AutoBert

Share this post


Link to post
Share on other sites
Reizvoller
3 hours ago, alien4u said:

Anyways if this are Hexadecimal color values always with a fixed length for example 4 characters you could take a high multiple of 4.

For example 400 this will make each array element store 100 colors values and this will make the arrays also shorten for parsing in the next stage of your code.

Kind Regards
Alien.

Shortening them would indeed be handy!

3 hours ago, czardas said:

Perhaps this helps:

#include <Array.au3>
Local $aArray = _StringEqualSplit("12345678910123456789101234567890", 8)
_ArrayDisplay($aArray)

Func _StringEqualSplit($sString, $iNumChars)
    If Not IsString($sString) Or $sString = "" Then Return SetError(1, 0, 0)
    If Not IsInt($iNumChars) Or $iNumChars < 1 Then Return SetError(2, 0, 0)
    Return StringRegExp($sString, "(?s).{1," & $iNumChars & "}", 3)
EndFunc

 

This is an interesting approach! I am going to try it right away, even if it is just for the learning experience :)

2 hours ago, markyrocks said:

$file=fileopen ("file.txt",1)

 

Your file is opening in read only mode

 

Edit.  I believe you could use 0x as delimiter then just add it back later

2d array simple example 

Dim $array[10][10]

For $col=0 to ubound($array)-1
For $row=0 to ubound($array)-1
$array[$row][$col]
MsgBox("","", $$array[$row][$col])
;this basically just shuffle through the array and display the results

Next
Next

 

I KNEW I was over looking something super simple with the FileWrite! I was staring at it going "There has gotta be a setting I am just not even thinking of..." Thank you! 

As for using 0x , I actually don't think that the function FFGetRawData throws out the 0x too or else this would be ideal.

19 minutes ago, AutoBert said:

This funcs do the job:

Func _FFArrayFromRawData($String, $iRows = 0, $iCols = 0)
    #cs
        _FFArrayFromRawData($String, [$iRows = 0, [$iCols = 0]])
        Params:     $String     the String with raw pixel data as returned from _FFArrayFromRawData
        $iRows      number of Rows in resulting array OPTIONAL
        $iCols      number of Cols in resulting array OPTIONAL
        possible
        Returns:    $aResult[$iPixelCount+1]    $iRows and $iCols = 0
        so all Pixels in a 1D array
        $aResult[$iRows]            1D array
        $aResult[$iRows][$iCols]    2D array
        also
        setting:    @Error      (0 no error, 1 error occured)
        @extended   10 no valid RawDataString
        11 Arraydimensions to big
        1 Array is 1D (info only)
        2 Array is 2D (info only)

        Author:     HarryBotter 2016
    #ce
    Local $aResult[0], $iPixNo, $iEx
    Local $iLen = StringLen($String)
    If Mod($iLen, 8) Then ;no valid RawDataString
        SetError(1, 10)
        Return
    EndIf
    If $iRows * $iCols * 8 > $iLen Then ;Arraydimensions to big
        SetError(1, 11)
        Return
    EndIf
    If $iRows = 0 And $iCols = 0 Then
        $iRows = $iLen / 8
        ReDim $aResult[$iRows]
        $iEx = 1
    Else
        If $iCols = 0 Then
            ReDim $aResult[$iRows]
            $iEx = 1
        Else
            ReDim $aResult[$iRows][$iCols]

            $iEx = 2
        EndIf
    EndIf
    For $i = 0 To $iRows - 1
        If $iEx > 1 Then
            For $j = 0 To $iCols - 1
                $iPixNo = $i * $iCols + $j
                $aResult[$i][$j] = _FFGetPixelColor($iPixNo, $String)
            Next
        Else
            $aResult[$i] = _FFGetPixelColor($i, $String)
        EndIf
    Next
    SetError(0, $iEx, $aResult)
    Return $aResult
EndFunc   ;==>_FFArrayFromRawData

Func _FFGetPixelColor($iPixNo, $String)
    #cs
        Params:     $iPixNo     the number of Pixel for wich the color is extracted
        $String     the String with raw pixel data as returned from _FFArrayFromRawData

        Remarks:    no Errorhandling only for using together with _FFArrayFromRawData

        Author:     HarryBotter 2016
    #ce
    Local $sRet = StringMid($String, $iPixNo * 8 + 1, 8)
    ;ConsoleWrite($iPixNo & ': ' & $sRet & @CRLF)
    Return $sRet
EndFunc   ;==>_FFGetPixelColor

i already posted you the link unsing to script using this in

I need to save CodeBot to my favorites , you have referenced it before and it seems like a valuable source of data. Your solution is very intricuit and will take a little time for me to review and employ for myself. Thank you for the information!

<><><>

So many replies to this little question of mine, I really apreciate every moment that was put into writing back to me. Thank you all very much! I am headed out of the office for the weekend but will be back on this problem ( and probably this forum ) come Monday :)

Thank you again!~

~~~Reiz

Share this post


Link to post
Share on other sites
AutoBert
5 hours ago, alien4u said:

Anyways if this are Hexadecimal color values always with a fixed length for example 4 characters you could take a high multiple of 4.

It's a String each Pixel is represented in BGRA-Format 2 chars for Blue, 2 chars for green, 2 chars for red and 2 chars for the alphachannel

 

2 hours ago, Reizvoller said:

Your solution is very intricuit and will take a little time for me to review and employ for myself.

It's not mine. I had a small look in FastFind's helpfile and i think there a better solutions possible than creating a array and search with AutoItscript for a pixel with specific color. Using the wrapped DLL-calls like FFGetPixel, FFNearestPixel, FFBestSpot should be perfomater you don't need the 2 Steps:

  • FFGetRawData
  • creating a array from resultstring for easier searching

and then searching in the array. If knowing the coords for a pixel you want know the colour use FFGetPixel. If you want the nearest pixel from a given coordinate in defined colour (shadevariantions are possible) user FFNearesPixel. For a Spot of Pixels you can use FFBestSpot, FFNearestSpot.

As this functions are wrapped DLL-calls for the FF.DLL it's much faster than the best optimized AutoIt code. FF.au3 is very complex and so it needs time to explore all the possiblilies. But this time is not wasted for you, not using a FF'-func when it's possible is wasted time. Only in case you know how to include and execute ASM-routines in autoit (I don't) you can build faster algorithm than the FF-funcs.

Share this post


Link to post
Share on other sites
Reizvoller
On 4/8/2016 at 7:06 PM, AutoBert said:

It's a String each Pixel is represented in BGRA-Format 2 chars for Blue, 2 chars for green, 2 chars for red and 2 chars for the alphachannel

 

It's not mine. I had a small look in FastFind's helpfile and i think there a better solutions possible than creating a array and search with AutoItscript for a pixel with specific color. Using the wrapped DLL-calls like FFGetPixel, FFNearestPixel, FFBestSpot should be perfomater you don't need the 2 Steps:

  • FFGetRawData
  • creating a array from resultstring for easier searching

and then searching in the array. If knowing the coords for a pixel you want know the colour use FFGetPixel. If you want the nearest pixel from a given coordinate in defined colour (shadevariantions are possible) user FFNearesPixel. For a Spot of Pixels you can use FFBestSpot, FFNearestSpot.

As this functions are wrapped DLL-calls for the FF.DLL it's much faster than the best optimized AutoIt code. FF.au3 is very complex and so it needs time to explore all the possiblilies. But this time is not wasted for you, not using a FF'-func when it's possible is wasted time. Only in case you know how to include and execute ASM-routines in autoit (I don't) you can build faster algorithm than the FF-funcs.

Howdy!

I applied the idea of  using just PixelSearch for another process for the same web interface. Trouble is, with this next step, there are many items that will have the color I am looking for and the location is going to change. This problem is actually why I am making this little script because this particular task is very cumbersome.

The entire thing, the work space,  renders as a big BMP and is not searchable as text. 

 

So the strategy was this ...

  1. Get reference RAW data for the areas of interest that I want to automatically navigate to/find
    1. Get a snapShot of the wanted area, convert to RAW, save on a .txt
  2. Open the work area
  3. Snap shot work area
  4. Search for the reference RAW in the WorkArea SnapShot RAW
  5. Scroll down x units 
  6. Go back to step 2

It isn't the most elegant of solutions but with the interface I have, it's really my only option ( that I can see at the moment. )

I suppose the better way would be to find a more finite way to specify the target areas I am wanting the script to find. Like a 1 pixel tall by 100 pixels wide rather than an area of 300x500 pixels.

I am going to see about finding other ways to search through this thing in the meantime. I think that my described method is too lengthy of a process to really be useful when compared against just manual searching. 

Edited by Reizvoller

Share this post


Link to post
Share on other sites
AutoBert

You don't need FFGetRawData, FFGetPixel searches in SnapShot too and this much faster. I think (not tested, i am no gamer or gamebotter) that building 2DArray with FFGetPixel is Faster than the way Hairy Botter showed. 

Share this post


Link to post
Share on other sites
Reizvoller
23 minutes ago, AutoBert said:

You don't need FFGetRawData, FFGetPixel searches in SnapShot too and this much faster. I think (not tested, i am no gamer or gamebotter) that building 2DArray with FFGetPixel is Faster than the way Hairy Botter showed. 

Quick note this is NOT for a game in any capacity! 

But your information is quite useful and will be tested once I get the chance, thank you. 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • nooneclose
      By nooneclose
      I need help turning this string "20180913221626" into a formatted time string.
      I need to go from this: 20180913221626
      to this: 09/13/2018 10:16 PM
      I do not always know what the date will be so I can not just use a variable I need to actually convert/format. 
      I did see an older post in the forms that was basically the same question only the other guy did not post the working code and I can not figure out how to use _AD_GetObjectProperties properly to get what I want. 
      As always any help would be appreciated. 
      Here is the code I use to find the date, but it always gives me the unformatted version. 
      ;retrieve the items object $oItem = $oOutlook.Session.GetItemFromID($aItems[1][0], Default) $oItem.GetInspector $eSentOn = $oItem.SentOn ; When was the email sent? MsgBox("", "Sent On of the email", $eSentOn) ;******************************************************************************* ; Formats the date and time from the email ;******************************************************************************* ;Local $fDatenTime = _DateTimeFormat($eSentOn, 1) ;MsgBox("", "Formatted email time", $fDatenTime) $aProperties = _AD_GetObjectProperties($eSentOn) _ArrayDisplay($aProperties, "Did the conversion work?")  
    • Skeletor
      By Skeletor
      Hi Virtual People,
      My array works perfectly fine. However, what is the best practice if the line in the array doesn't have the correct amount of columns and if I can add a placeholder?

       
      For $count = 1 To _FileCountLines($FileRead1) Step 1 $string = FileReadLine($FileRead1, $count) $input = StringSplit($string, ",", 1) $value1 = $input[1] $value2 = $input[2] $value3 = $input[3] _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value2, "A1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value1, "B1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value3, "C1") Next  
    • MrCheese
      By MrCheese
      hi all,
      reviewing the forum, this thread is applicable: 
       
       
      I wanted to know if there is now a better way to do this?
      In essence, I load a tab delimited txt file into an array (works well). I used tab, as some fields in the original csv contains commas.
      However, I needed autoit to manipulate this array, and output it as a csv.
      IF my array contains items with a comma, without double quotes around the field, then how best do I get a csv out of this?
      My current workaround is to filewritefromarray tab delimited, then open it in excel and save as a csv. I will need to check this to see how the address fields behave that contain a comma.
       
      Any thoughts would be appreciated.
       
    • Skeletor
      By Skeletor
      Hi All,

      I would like to know how you would take a FileLineRead and insert it into an array which then inserts it into Excel?
      One thing to know is the files content is broken up, so I only use half of the content within $FileRead1.
      So its imperative that the $value1, $value2, etc variables be used. 
      Code below:
      $FileRead1 = FileReadLine("C:\temp\sample.txt",1) For $count = 1 To _FileCountLines($FileRead1) Step 1 $string = FileReadLine($FileRead1, $count) $input = StringSplit($string, ",", 1) $value1 = $input[1] $value2 = $input[2] $value3 = $input[3] $value4 = $input[4] _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value1, "A1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value2, "B1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value3, "C1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value4, "D1") Next  
    • AnonymousX
      By AnonymousX
      Hello,
      I'm trying to write a script that moves copies excel cells into an array. I'll than manipulate the values and send array into another program. 
      I don't want range to be specific to a workbook, or sheet, or set of cells.
      I want user to be able to highlight desired cells and to copy either normally ("Ctrl+C") or by a hotkey ("Alt+C"). 
      Could someone help me with this?
      Thank you,
      I've tried to write the framework: (edited)
      #include <MsgBoxConstants.au3> #include <Array.au3> #include <Excel.au3> HotKeySet("!v", "Pastedata") While True Sleep(1000) WEnd func Makearray() local $bArray ;User has cells already copied ;Convert clipboard into an array ;I don;t know how excel stores data to clipboard so don;t know how to bring it into array _Arraydisplay($bArray) MsgBox(0,0,$bArray) return $bArray endfunc func Pastedata() Local $aArray MsgBox(0,0,"wait",1) ;make array based on assumption user has already copied a range to clipboard $aArray = Makearray() ;paste code ;don;t worry about this I got the rest endfunc  
×