Jump to content

saving array with nested array inside


frank10
 Share

Recommended Posts

I would like to use an array with this structure and save/read it.

global $arr1 = [[1,"2"],[3,"4"],[5,"6"]]
global $arr2 = [[7,"8"],[9,"10"],[11,"12"]]
global $arrFinal = [ ["test1", $arr1, $arr2], ["test2", $arr1, $arr2] ]
;~ _ArrayDisplay($arrFinal)

_FileWriteFromArray( @ScriptDir & "\arrFinal.txt" , $arrFinal)

But _FileWriteFromArray() works only with plain 2D array, not with array with 2d array nested in it...

Is there an UDF for this?

Or I could change the structure of the array... I don't know how, because it seems more logic as is:

each row has a test name, followed by 2 groups of data containing some histogram stats bins, each referring to different timeset, now grouped in 2 array nested, so each row has all the data logically grouped.

i.e.

Array rows  = "testXX"  --> col1 arr of histogram bins for timeset1--> hist00 = bin0data0, hist01= bin0data1 etc // --> col2 arr of histogram bins for timeset2 etc

Edited by frank10
Link to comment
Share on other sites

I assume:

  • Every row starts with a test name in column 0
  • Data group 1 starts in column 1
  • Each timeset consists of 2 values
  • The number of timesets in each data group could be variable

I suggest a 2D array with a cell holding a separator between data group 1 and datagroup 2. The following example uses "||" as separator.

#include <Array.au3>

Global $aFinal = [["test1", 1, "2", 3, "4", 5, "6", "||", 7, "8", 9, "10", 11, "12"], _ 
        ["test2", 1, "2", 3, "4", 5, "6", 7, "8", "||", 9, "10", 11, "12", 13, "14", 15, "16"]]
_ArrayDisplay($aFinal)

To process data group 1 you start in column 1 and process all timesets until you hit the separator
To process data group 2 you loop through the row until you hit the separator and begin to process the timesets starting in the next column

Edited by water

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
OutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - Download
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
PowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - Wiki
Task Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki

Standard UDFs:
Excel - Example Scripts - Wiki
Word - Wiki

Tutorials:
ADO - Wiki
WebDriver - Wiki

 

Link to comment
Share on other sites

Ok, thank you. I tested and it works even if it's better to use a different separator like "#" if using default _FileWriteFromArray  and _FileReadToArray otherwise it fails reloading.

Anyway, it's more difficult to get data of a specific bin: I put 2 data in each bin: i.e bin0 correspond to data from 0 to 10000, data0 is the number of samples and data1 is the percentage, and so on for the other bins.

So with array nested, I could ask for percentage of bin3 with arr[3][1], (after de-nested the hist arr from the container arr).

With your unique all-2d array you can't know immediately how many bins you have (they were the rows->Ubound), and you must count each row if odd or even to discriminate between samples and percentages. Doable, but less immediate. Even recostructing the hist array should be difficult without knowing if I always mantain 2 data or more data for each bin.

Infact this time I will use this structure, but next time I could store 3 data for each bins or put different bins for timesets and so on. And I should change every time also the decoding logic of the unique all-2d arr.

 

If there are no other ideas, I'll try to make a custom load/save script and post it.

Link to comment
Share on other sites

you might want to use XML or JSON for this kind of data structure (AutoIt has UDFs for both). this will give you additional flexibility if you find yourself in need to expand the nesting level or have a non-consistent structure across a level.

Signature - my forum contributions:

Spoiler

UDF:

LFN - support for long file names (over 260 characters)

InputImpose - impose valid characters in an input control

TimeConvert - convert UTC to/from local time and/or reformat the string representation

AMF - accept multiple files from Windows Explorer context menu

DateDuration -  literal description of the difference between given dates

Apps:

Touch - set the "modified" timestamp of a file to current time

Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes

SPDiff - Single-Pane Text Diff

 

Link to comment
Share on other sites

Anyway, this is what I made to load/save nested arrays:

2 Func to Load / Save:

#include <Array.au3>

Global $sFile = @ScriptDir & "\arrFinal.txt"
Global $arrFinal

global $arr0 = [[1,"2"],[3,"4"],[5,"6"]]
global $arr1 = [1,2,3,"4",5,"6"]
global $arr2 = [ [7,8],[9,10],[11,12,"data1"],["data9","data10",134],[11,12,1] ]
global $arrFinal = [ ["test1", $arr0, "otherCol1",$arr2], ["test2", $arr1, "otherCol2"] ]

_ArrayDisplay($arrFinal, "original")
 
_FileSaveNestedArray($sFile, $arrFinal)
_ArrayDisplay($arrFinal, "after save")

Global $arrFinal
_FileReadNestedArray($sFile, $arrFinal)
_ArrayDisplay($arrFinal, "after restore")

; check nested arrays after loaded:
_ArrayDisplay( $arrFinal[0][3] ,"row0 col3")
_ArrayDisplay( $arrFinal[1][1] ,"row1 col1")

_FileSaveNestedArray($sFile, $arrFinal)


func _FileReadNestedArray($fileName, ByRef $arrFinal) 
    _FileReadToArray($fileName,$arrFinal, $FRTA_NOCOUNT,"|")
    _ArrayDisplay($arrFinal, "after loaded")

    For $i=0 to UBound($arrFinal)-1
        for $j = 0 to UBound($arrFinal,2) -1
            local $arr = $arrFinal[$i][$j], $arr1
            if StringInStr($arr, "#") Then 
                $arr1 = StringSplit($arr, "#")
                local $maxCol = 1
                for $x = 1 to UBound($arr1) -1 
                    StringReplace($arr1[$x], "@", "")
                    if @extended +1 > $maxCol Then $maxCol = @extended +1
                next
                local $arrTemp[0][$maxCol]
                _ArrayAdd($arrTemp,$arr, 0,"@","#")
                $arrFinal[$i][$j] = $arrTemp
            elseif StringInStr($arr, "@") Then 
                $arr1 = StringSplit($arr, "@")
                _ArrayDelete($arr1,0)
                $arrFinal[$i][$j] = $arr1
            EndIf
        Next
    Next
EndFunc

func _FileSaveNestedArray($fileName, $arrFinal) 
    local $arrToSave = $arrFinal
    For $i=0 to UBound($arrToSave)-1
        for $j = 0 to UBound($arrToSave,2)-1
            local $arr = $arrToSave[$i][$j]
            if IsArray($arr) Then
                $arrToSave[$i][$j] = _ArrayToString($arr,"@",-1,-1,"#")
            Else
                $arrToSave[$i][$j] = $arr
            endif
        Next
    Next
    _ArrayDisplay($arrToSave, "before save")
    _FileWriteFromArray( @ScriptDir & "\arrFinal.txt" , $arrToSave)
EndFunc

The only drawback, you can't use "@" and "#" (nor "|") chars in the nested array content. Maybe we can change them with more unusual chars...

If you want, test it further.

Edited by frank10
Link to comment
Share on other sites

1 hour ago, frank10 said:

bin0 correspond to data from 0 to 10000

Looks like you're going to store a significant number of values. Then it could be better suited to store the data in a database. AutoIt has support for SQLite, which can manage unlimited volumes of data and query in simple or complex ways.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

No, those aren't the samples occurred, they are the range values each sample can reach:

i.e.

test1 get these value results:

1450,56777,110033,35466 etc

So, I create bins to group the value results (0-10000,10000-50000 etc) and look in which range there are more dense results, sparse and so on, other stats.

Edited by frank10
Link to comment
Share on other sites

So irrespective of the number of values, You'll have better time using SQLite to extract this type of information.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

I can't for this particular type of script, because, as I wrote before, I need to download these data from an HTML web page and reupload them modified by autoit  using WD_driver.

So, I must cope with JS and autoit arrays, possibly with JSON too to exchange them.

 

BTW,

I was already using JSON (jsmn) with WD_driver, but I believe it can't encode 2D array, do you know if it's possible?

Edited by frank10
Link to comment
Share on other sites

On 2/8/2021 at 9:06 PM, frank10 said:

but I believe it can't encode 2D array, do you know if it's possible?

JSON syntax is not able to distinguish n-dimensional arrays from array-in-array structures.
A 2D array in JSON would therefore be mapped as an array-in array.
Attached is a JSON UDF in pure AutoIt. With this one would accomplish this as follows:

#include <JSON.au3>

global $arr0 = [[1,"2"],[3,"4"],[5,"6"]]
global $arr1 = [1,2,3,"4",5,"6"]
global $arr2 = [ [7,8],[9,10],[11,12,"data1"],["data9","data10",134],[11,12,1] ]
global $arrFinal = [ ["test1", $arr0, "otherCol1",$arr2], ["test2", $arr1, "otherCol2"] ]
_ArrayDisplay($arrFinal)

; serialize a AutoIt-variable into a  JSON string
$sSerialized = _JSON_Generate($arrFinal)
ConsoleWrite($sSerialized)

; or compact json:
;~ $sSerialized = _JSON_Generate($arrFinal, '', '', '', '', '', '')
;~ ConsoleWrite($sSerialized)

; rebuild AutoIt 2D-Array from JSON-String
$aRebuild = _JSON_Parse($sSerialized)
$aRebuild = __ArrayAinATo2d($aRebuild)
_ArrayDisplay($aRebuild)

 

 

 

Edited by AspirinJunkie
Removed attached Json.au3 because there is now a thread for this
Link to comment
Share on other sites

10 hours ago, AspirinJunkie said:

JSON syntax is not able to distinguish n-dimensional arrays from array-in-array structures.
A 2D array in JSON would therefore be mapped as an array-in array.
Attached is a JSON UDF in pure AutoIt. With this one would accomplish this as follows:

#include <JSON.au3>

global $arr0 = [[1,"2"],[3,"4"],[5,"6"]]
global $arr1 = [1,2,3,"4",5,"6"]
global $arr2 = [ [7,8],[9,10],[11,12,"data1"],["data9","data10",134],[11,12,1] ]
global $arrFinal = [ ["test1", $arr0, "otherCol1",$arr2], ["test2", $arr1, "otherCol2"] ]
_ArrayDisplay($arrFinal)

; serialize a AutoIt-variable into a  JSON string
$sSerialized = _JSON_Generate($arrFinal)
ConsoleWrite($sSerialized)

; or compact json:
;~ $sSerialized = _JSON_Generate($arrFinal, '', '', '', '', '', '')
;~ ConsoleWrite($sSerialized)

; rebuild AutoIt 2D-Array from JSON-String
$aRebuild = _JSON_Parse($sSerialized)
$aRebuild = __ArrayAinATo2d($aRebuild)
_ArrayDisplay($aRebuild)

I tried it but it's not working:

The $aRebuild array apparently has the same structure of the original array but if you check the same row0col3 it has very different structure.

Check

_ArrayDisplay($arrFinal[0][3], "check original row0 col3")
and
_ArrayDisplay($aRebuild[0][3], "check rebuilt row0 col3")
 
More, after
_JSON_Generate($arrFinal)
it changes also the original array $arrFinal structure, so you can't check anymore $arrFinal[0][3] ...
 
Edited by frank10
Link to comment
Share on other sites

11 hours ago, jguinch said:

@frank10  :  maybe you will find what you want here :

It seems the fonctions ArrayToDeclarationString and ArrayDeclareFromString are what you need 😉

thank you, but it seems it can't do array in array:

I tried:

#include <Array.au3>
#Include "ArrayMultiDim.au3"


Opt("WinTitleMatchMode", 2)

Global $arrFinal

global $arr0 = [[1,"2"],[3,"4"],[5,"6"]]
global $arr1 = [1,2,3,"4",5,"6"]
global $arr2 = [ [7,8],[9,10],[11,12,"data1"],["data9","data10",134],[11,12,1] ]
global $arrFinal = [ ["test1", $arr0, "otherCol1",$arr2], ["test2", $arr1, "otherCol2"] ]

local $sArrFinal = _ArrayToDeclarationString($arrFinal)
Consolewrite($sArrFinal & @CRLF)
_ArrayDisplay($arrFinal, "original")


local $arrFromString = _ArrayDeclareFromString($sArrFinal)
_ArrayDisplay($arrFromString, "check")

But it fails creating the string and so recreating the original array.

Edited by frank10
Link to comment
Share on other sites

1 hour ago, frank10 said:

The $aRebuild array apparently has the same structure of the original array but if you check the same row0col3 it has very different structure.

Check

_ArrayDisplay($arrFinal[0][3], "check original row0 col3")
and
_ArrayDisplay($aRebuild[0][3], "check rebuilt row0 col3")

 

As i told you JSON-Syntax is not able to map 2D-arrays as such.
They are all mapped as array-in arrays.
Of course, this also applies to nested subarrays.
You must therefore of course also explicitly convert them back to 2D arrays if you absolutely want to have 2D arrays instead of array-in arrays:

#include <JSON.au3>

global $arr0 = [[1,"2"],[3,"4"],[5,"6"]]
global $arr1 = [1,2,3,"4",5,"6"]
global $arr2 = [ [7,8],[9,10],[11,12,"data1"],["data9","data10",134],[11,12,1] ]
global $arrFinal = [ ["test1", $arr0, "otherCol1",$arr2], ["test2", $arr1, "otherCol2"] ]

_ArrayDisplay($arrFinal[0][3], "$arrFinal[0][3] before")

; serialize a AutoIt-variable into a  JSON string
$sSerialized = _JSON_Generate($arrFinal, '', '', '', '', '', '')

; rebuild AutoIt 2D-Array from JSON-String
$aRebuild = _JSON_Parse($sSerialized)
$aRebuild = __ArrayAinATo2d($aRebuild)

; convert sub-2D-Array from Array-InArray-Structure to 2D-Array
$aRebuild[0][3] = __ArrayAinATo2d($aRebuild[0][3])

_ArrayDisplay($aRebuild[0][3], "$arrFinal[0][3] after")

You won't automatically get 2D arrays from a JSON string. You have to define yourself which parts should be explicitly treated as such.

1 hour ago, frank10 said:

it changes also the original array $arrFinal structure, so you can't check anymore $arrFinal[0][3] ...

Remove the ByRef from _JSON_Generate

Edited by AspirinJunkie
Link to comment
Share on other sites

1 hour ago, AspirinJunkie said:

As i told you JSON-Syntax is not able to map 2D-arrays as such.
They are all mapped as array-in arrays.
Of course, this also applies to nested subarrays.
You must therefore of course also explicitly convert them back to 2D arrays if you absolutely want to have 2D arrays instead of array-in arrays:

#include <JSON.au3>

global $arr0 = [[1,"2"],[3,"4"],[5,"6"]]
global $arr1 = [1,2,3,"4",5,"6"]
global $arr2 = [ [7,8],[9,10],[11,12,"data1"],["data9","data10",134],[11,12,1] ]
global $arrFinal = [ ["test1", $arr0, "otherCol1",$arr2], ["test2", $arr1, "otherCol2"] ]

_ArrayDisplay($arrFinal[0][3], "$arrFinal[0][3] before")

; serialize a AutoIt-variable into a  JSON string
$sSerialized = _JSON_Generate($arrFinal, '', '', '', '', '', '')

; rebuild AutoIt 2D-Array from JSON-String
$aRebuild = _JSON_Parse($sSerialized)
$aRebuild = __ArrayAinATo2d($aRebuild)

; convert sub-2D-Array from Array-InArray-Structure to 2D-Array
$aRebuild[0][3] = __ArrayAinATo2d($aRebuild[0][3])

_ArrayDisplay($aRebuild[0][3], "$arrFinal[0][3] after")

You won't automatically get 2D arrays from a JSON string. You have to define yourself which parts should be explicitly treated as such.

Got it, great!

 

I compared it to JS  JSON.stringify() and it has 1 more [....]:

;autoit stringified: [["test1",[[60,55997],[9,4],[2,1],[129,57],[88,39],[0,0]],[[365,44398],[127,9],[24,2],[436,31],[736,52],[91,6]]]]
;JS stringified:     ["test1",[[60,55997],[9,4],[2,1],[129,57],[88,39],[0,0]],[[365,44398],[127,9],[24,2],[436,31],[736,52],[91,6]]]

Is this normal? Shouldn't be equal results?

Before reuploading to JS it must be removed the start+end brackets.

EDIT

Sorry, the example refers only to one row, if I have multiple rows I'll use another level of []...

 

 

Anyway, shouldn't it be better if the _JSON_Generate doesn't change the original array structure?

Edited by frank10
Link to comment
Share on other sites

17 minutes ago, frank10 said:

I compared it to JS  JSON.stringify() and it has 1 more [....]:

Where are these results from?
Both don't describe the data-structure showed in your example.
The correct expression instead (as it generated by the script) is the following:

[["test1",[[1,"2"],[3,"4"],[5,"6"]],"otherCol1",[[7,8,""],[9,10,""],[11,12,"data1"],["data9","data10",134],[11,12,1]]],["test2",[1,2,3,"4",5,"6"],"otherCol2",""]]

you can visualize json-strings into a more manageable form with various tools like jsoneditoronline.org or jsonviewer.stack.hu.

 

20 minutes ago, frank10 said:

Anyway, shouldn't it be better if the _JSON_Generate doesn't change the original array structure?

yep

Link to comment
Share on other sites

2 hours ago, frank10 said:

thank you, but it seems it can't do array in array:

I confirm it cannot be used withs arrays containing arrays, in the same way you cannot declare such an array in a single declaration

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...