LarsJ

Data display functions based on virtual listviews

7 posts in this topic

#1 ·  Posted (edited)

This project implements data display functions based on _ArrayDisplay and _ArrayDisplayEx (virtual listview).

Four functions are planned and maybe there’ll also come a fifth function. First post contains documentation common to all functions. In order to avoid first post being too long, posts 2 - 5 are reserved for documentation that's specific to the four planned functions. The first two functions _ArrayDisplayEx and CSVfileDisplay are presented below. The other functions will be presented in the coming weeks.

The functions are mainly intended for large amounts of data with a large number of rows. The GUI and listview are implemented for this purpose. Thus, the left column in the listview is a row-number column. This is convenient when there are many rows. Through an input control at bottom of the GUI, you can focus on a specific row. When there are many rows it's difficult to scroll to a specific row with the mouse.

The primary purpose of the functions is testing and debugging. Secondary to display data for end users. Therefore, features that are directly available in the function definition through parameters and flags are reduced to a minimum, while additional features that are especially interesting for end users must be specified in a single parameter which is the last parameter in the function definition.

The definition of _ArrayDisplayEx looks like this:

; Displays a 1D or 2D array in a virtual ListView
Func _ArrayDisplayEx( _
  $aArray, _          ; 1D/2D array eg. StringSplit("Mo,Tu,We,Th,Fr,Sa,Su", ",")
  $sTitle = "", _     ; GUI title bar text, default title is set to "ArrayDisplayEx"
  $sHeader = "", _    ; ListView header column names, default is "Col0|Col1|Col2|...|ColN"
  $iFlags = 0x0000, _ ; Set additional options through flag values
  $aFeatures = "" )   ; 2D array of feature type/info pairs

All functions uses a virtual listview to display data. A normal listview has two main features: Data storage and data display. A virtual listview has only one main feature: Data display. Data is stored in a data source outside the listview. Because data does not have to be inserted into the rows of a virtual listview, the listview shows up instantly. The number of rows is limited only to the data source.

Data display in the virtual listview is implemented through subclassing. This means that the code will not interfere with WM_NOTIFY message handlers in your own code. And because there are a lot of messages involved in updating the cell texts in a virtual listview, the subclassing technique is also performance optimizing.

The first parameter in the function definition is the data source. The data source is the main difference between the display functions. Depending on the function more than one parameter can be used to support the data source.

After one or more data source parameters follows title, header and flag parameters. The last parameter is an array of additional features. These four parameters are common to all display functions.

 

Additional features

Only basic functionality used for testing and debugging is implemented directly in the display functions. It's possible to use a wide range of additional features that are especially interesting when the display functions are used to display data for end users.

Usually only a few features are used at a time. To avoid adding a lot of parameters to each display function of which only a few are used, additional features must be specified through a single $aFeatures parameter, which is the last parameter in the function definition.

Also in order to avoid adding a lot of extra code to the functions that are rarely used, most of the features are implemented in separate include files. These files must be included in order to use the features.

$aFeatures parameter is checked for errors under function initializing. In most cases, an error just means that the feature is skipped. Only in case of serious mistakes eg. a missing include file the function will return with an error code in @error. If verbose mode is turned on ($iFlags = 8), the error will be displayed in a MsgBox.

The following additional features can be used:

  • Column alignment
  • Min. column width
  • Max. column width
  • Column text formats
  • Column background colors
  • Listview background color
  • Listview header background colors
  • Sort rows in data source by index
  • Sort columns in data source by index
  • User supplied function

See HelpFiles\DisplayFeatures.txt in zip-file for documentation of each feature. See Examples\ folder for examples.

 

Missing features

Compared to the official _ArrayDisplay and the old version of _ArrayDisplayEx some features are omitted in the new functions. It's not possible to specify a data range to only display a small section of data. But rows and columns can be sorted by row and column indices which can replace the range feature. This is demonstrated in examples.

Data copying that could be performed through two GUI buttons is omitted. Because the listviews are virtual listviews potentially millions of cells can be selected and copied. Copying such large amounts of data can take a long time. Since data may in any case be copied directly from the data source, the copying functionality is omitted to avoid long-term copying.

It's also not possible to change rows and columns. No transpose functionality.

 

Function examples

The zip-file is organized in this way:

  • DataDisplay\ - Top folder
    • Common\ - Code common to all functions
    • Examples\ - Examples for each function
    • Functions\ - Implements the functions
    • HelpFiles\ - Docu of additional features
    • Resources\ - Resources used in examples

Common\ and Functions\ folders contains all the code used to implement the display functions. The Functions\ folder is further divided into subfolders for each display function.

The Examples\ folder is also divided into subfolders for each display function. For each function the features are demonstrated one by one.

If more parameters are used in relation to the data source, the utilization of the parameters is demonstrated in "Examples\<Display function>\0) Data source\".

For most examples the data source is created on the fly. For this purpose, a few small UDFs are used. These UDFs are stored in Resources\.

Using functions
To use a display function in your own project copy the entire DataDisplay\ folder to the project folder. You can rename DataDisplay to Display or similar if you want a shorter name. You can delete Examples\, HelpFiles\ and Resources\. You can also delete display functions in the Functions\ folder that you're not using. Then include the function you need in your code.

Usage of the display functions is demonstrated in "Examples\<Display function>\3) Using function\". Here the code in "Examples\ArrayDisplayEx\3) Using function\My Project\Script1.au3":

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "Display\Functions\ArrayDisplayEx\ArrayDisplayEx.au3"

Opt( "MustDeclareVars", 1 )

Example()

Func Example()
  Local $iRows = 10000, $iCols = 10
  Local $aArray[$iRows][$iCols]

  For $i = 0 To $iRows - 1
    For $j = 0 To $iCols - 1
      $aArray[$i][$j] = $i & "/" & $j
    Next
  Next

  _ArrayDisplayEx( $aArray )
EndFunc

 

And in "Examples\ArrayDisplayEx\3) Using function\My Project\Script2.au3":

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "Display\Display.au3"

Opt( "MustDeclareVars", 1 )

Example()

Func Example()
  Local $iRows = 10000, $iCols = 10
  Local $aArray[$iRows][$iCols]

  For $i = 0 To $iRows - 1
    For $j = 0 To $iCols - 1
      $aArray[$i][$j] = $i & "/" & $j
    Next
  Next

  _ArrayDisplayEx( $aArray, "", "", 0, GetDisplayFeatures() )
EndFunc

Display\Display.au3:

#include-once
#include "Functions\ArrayDisplayEx\ArrayDisplayEx.au3"
#include "Functions\ArrayDisplayEx\ArrayDisplayEx_ColumnColors.au3"

Func GetDisplayFeatures()
  ; ArrayDisplayEx features
  Local $aAlignment = [ [ -1, "R" ], [ 0, "C" ], [ 1, "L" ] ]

  Local $aColColors = [ [ 1, 0xCCFFCC ], [ 3, 0xFFFFCC ] ]

  Local $aFeatures  = [ [ "ColAlign",  $aAlignment ], _
                        [ "ColColors", $aColColors ] ]
  Return $aFeatures
EndFunc

 

Zip-file

The organization of the zip-file is described in the section above.

You need AutoIt 3.3.10 or later. Tested on Windows 10, Windows 7 and Windows XP.

Comments are welcome. Let me know if there are any issues.

DataDisplay.7z

Edited by LarsJ
Added new paragraph in top of post
4 people like this

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

_ArrayDisplayEx

This is copied from the top of ArrayDisplayEx.au3:

; Displays a 1D or 2D array in a virtual ListView
Func _ArrayDisplayEx( _
  $aArray, _          ; 1D/2D array eg. StringSplit("Mo,Tu,We,Th,Fr,Sa,Su", ",")
  $sTitle = "", _     ; GUI title bar text, default title is set to "ArrayDisplayEx"
  $sHeader = "", _    ; ListView header column names, default is "Col0|Col1|Col2|...|ColN"
  $iFlags = 0x0000, _ ; Set additional options through flag values
  $aFeatures = "" )   ; 2D array of feature type/info pairs

  ; $iFlags values
  ; Add required values together
  ; 0x0000 => Left aligned text
  ; 0x0002 => Right aligned text
  ; 0x0004 => Centered text
  ; 0x0008 => Verbose mode
  ; 0x0010 => Half-height ListView
  ; 0x0020 => No grid lines in ListView

  ; $aFeatures parameter
  ; Features available in _ArrayDisplayEx:
  ; "ColAlign"    => Column alignment
  ; "ColWidthMin" => Min. column width
  ; "ColWidthMax" => Max. column width
  ; "ColFormats"  => Column text formats
  ; "ColColors"   => Column background colors
  ; "BackColor"   => Listview background color
  ; "HdrColors"   => Listview header background colors
  ; "SortRows"    => Sort rows in data source by index
  ; "SortCols"    => Sort columns in data source by index
  ; "UserFunc"    => User supplied function
  ; See DisplayFeatures.txt for docu of features

  ; Error codes in @error
  ; 1 => No array variable passed to function
  ; 2 => Larger than 2D array passed to function
  ; 3 => Missing include file

Usage of additional features is demonstrated in Examples\ArrayDisplayEx\1) Single features\.

How to use _ArrayDisplayEx in your own project is demonstrated in Examples\ArrayDisplayEx\3) Using function\.

Edited by LarsJ
2 people like this

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

CSVfileDisplay

This is copied from the top of CSVfileDisplay.au3:

; Displays a CSV file in a virtual ListView
Func CSVfileDisplay( _
  $sCSVfile, _         ; Name of CSV file (plain text file)
  $iEncoding = 128, _  ; File encoding, default is UTF-8 with BOM
  $sSeparator = "|", _ ; Field separator character, default is "|"
  $iMax_Fields = 0, _  ; Max. number of data fields in a line in the CSV file
  $sTitle = "", _      ; GUI title bar text, default title is set to "CSVfileDisplay"
  $sHeader = "", _     ; ListView header column names, default is "Col0|Col1|Col2|...|ColN"
  $iFlags = 0x0100, _  ; Set additional options through flag values
  $aFeatures = "" )    ; 2D array of feature type/info pairs

  ; $iFlags values
  ; Add required values together
  ; 0x0000 => Left aligned text
  ; 0x0002 => Right aligned text
  ; 0x0004 => Centered text
  ; 0x0008 => Verbose mode
  ; 0x0010 => Half-height ListView
  ; 0x0020 => No grid lines in ListView
  ; 0x0100 => Calculate $iMax_Fields from first line
  ; 0x0200 => Calculate $iMax_Fields from first 10 lines
  ; 0x0400 => Calculate $iMax_Fields from first 100 lines

  ; $aFeatures parameter
  ; Features available in CSVfileDisplay:
  ; "ColAlign"    => Column alignment
  ; "ColWidthMin" => Min. column width
  ; "ColWidthMax" => Max. column width
  ; "ColFormats"  => Column text formats
  ; "ColColors"   => Column background colors
  ; "BackColor"   => Listview background color
  ; "HdrColors"   => Listview header background colors
  ; "SortRows"    => Sort rows in data source by index
  ; "SortCols"    => Sort columns in data source by index
  ; "UserFunc"    => User supplied function
  ; See DisplayFeatures.txt for docu of features

  ; Error codes in @error
  ; 1 => CSV file does not exist
  ; 2 => Invalid CSV file passed to function
  ; 3 => Missing include file

The first four function parameters are related to the data source, the CSV file. $sCSVfile is the file name, $iEncoding is the file encoding, $sSeparator is the field separator character and $iMax_Fields is the maximum number of data fields in a line in the CSV file.

The default value of $iMax_Fields is zero which means that the number of data fields is calculated automatically by inspecting the lines in top of the CSV file. How many lines that are inspected is decided through three flag values 0x0100, 0x0200, and 0x0400 which means that 1, 10 and 100 lines in top of the file are inspected.

Because this inspection takes time it's limited to 100 lines. If the lines from number 101 to the end of file contains more fields than the first 100 lines you have to set $iMax_Fields manually.

If the lines in the file contains 10 fields but you only want to display 7 fields you can set $iMax_Fields = 7.

CSVfileDisplay main code
The CSV file is loaded into a 1D array, $aCSVfile, with FileReadToArray(). Through $iFrom and $iTo parameters of the caching mechanism in the virtual listview a range of rows in $aCSVfile are split into single fields by the separator character and stored in a 2D array, $aDisplay. The visible listview cells can now be filled with data from $aDisplay.

CSVfileDisplay examples
Usage of $iMax_Fields and 0x0100, 0x0200, and 0x0400 flags is demonstrated in Examples\CSVfileDisplay\0) Data source\.

Usage of additional features is demonstrated in Examples\CSVfileDisplay\1) Single features\.

How to use CSVfileDisplay in your own project is demonstrated in Examples\CSVfileDisplay\3) Using function\.

Wrapper function
In most examples, the parameters $iEncoding, $sSeparator and $iMax_Fields are the same. Therefore, it's convenient to use a wrapper function to execute CSVfileDisplay(). Examples\CSVfileDisplay\CSVfileDisplayWrapper.au3 contains the wrapper function CSVfileDisplayWr() that's used in most of the examples:

#include-once
#include "..\..\Functions\CSVfileDisplay\CSVfileDisplay.au3"
#include "..\..\Resources\CSVfileDisplay\Conv2DArray.au3"
#include "..\..\Resources\CSVfileDisplay\Save1DArray.au3"

Func CSVfileDisplayWr( _
  $sCSVfile, _         ; Name of CSV file (plain text file)
  $sTitle = "", _      ; GUI title bar text, default title is set to "CSVfileDisplay"
  $sHeader = "", _     ; ListView header column names, default is "Col0|Col1|Col2|...|ColN"
  $iFlags = 0x0100, _  ; Set additional options through flag values
  $aFeatures = "" )    ; 2D array of feature type/info pairs

  CSVfileDisplay( $sCSVfile, 128, "|", 0, $sTitle, $sHeader, $iFlags, $aFeatures )
EndFunc

Note that CSVfileDisplayWr() is very similar to _ArrayDisplayEx(). CSVfileDisplayWrapper.au3 also includes a couple of UDFs that are used to create CSV files on the fly.

Edited by LarsJ
Copy/paste error in first code box
2 people like this

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

17 hours ago, LarsJ said:

... Comments are welcome ....

@LarsJ,   thank you for this nice share!
.... only a little perplexity about the need to edit a separate include file to set features. Anyway i see that if you want you can also set the features locally avoiding to use the "external custom include" as I'm doing in this little test. (save this little test in this path: Examples\ArrayDisplayEx\3) Using function\My Project) 

Spoiler
#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "Display\Display.au3"
#include <date.au3>
#include <array.au3>

Opt("MustDeclareVars", 1)

example()

Func example()
    Local $aMonth = _GenerateMonth() ; generate a little array of current month
    ;
    _ArrayDisplayEx( _
            _ArrayExtract($aMonth, 1, 6, 0, 6), _ ; ............. Array of values to be displayed
            "A calendar", _ ; ................................... Title of the window
            _ArrayToString($aMonth, "|", 0, 0, "", 0, 6), _ ; ... String wit titles of each column
            0x0004, _ ; ......................................... additional options
            CSS_Display() _ ; ................................... 2D array of feature type/info pairs
            )
EndFunc   ;==>example

; #FUNCTION# ====================================================================================================================
; Author ........: Chimp
; Modified.......:
; ===============================================================================================================================
Func _GenerateMonth($iYear = @YEAR, $iMonth = @MON, $ISO = True)
    ; this function creates a month calendar into an 7x7 array
    Local $aMonth[7][7], $iDOW
    Local $iFirstDOW = $ISO ? _DateToDayOfWeekISO($iYear, $iMonth, 1) : _DateToDayOfWeek($iYear, $iMonth, 1)
    For $iDay = 1 To 7
        $aMonth[0][$iDay - 1] = $ISO ? _DateDayOfWeek(Mod($iDay, 7) + $ISO, $DMW_LOCALE_SHORTNAME) : _DateDayOfWeek($iDay, $DMW_LOCALE_SHORTNAME)
    Next
    For $iDay = 1 To _DateDaysInMonth($iYear, $iMonth)
        $iDOW = $ISO ? _DateToDayOfWeekISO($iYear, $iMonth, $iDay) : _DateToDayOfWeek($iYear, $iMonth, $iDay)
        $aMonth[Int(($iFirstDOW + $iDay - 2) / 7) + 1][$iDOW - 1] = $iDay
    Next
    Return $aMonth
EndFunc   ;==>_GenerateMonth

Func CSS_Display()

    ; Column colors:
    Local $aColColors = [[6, 0xFFFFCC],[7, 0xFF4500]] ; set colors of columns

    ; load the carrier array with features
    Local $aFeatures = [["ColColors", $aColColors]] ; fill the "aFeatures" array
    Return $aFeatures ; return the carrier with all features
EndFunc   ;==>CSS_Display

 

...from a better reading I see that embedding features in an include is just a further option available allowed by your udf (sorry for my misunderstanding)


If allowed just two little suggestions demands:
would be nice to be able to "hide" the first column where are shown the row numbers by setting a flag ( $iFlags)
would be also nice , as an option, to be able to use this data display as an embeddable GUI control instead of as well as a "standalone window"

Thanks againn for sharing!

Edited by Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites

#7 ·  Posted

Chimp, The functions are mainly intended for large amounts of data with a large number of rows, where it's reasonable to use as much code and virtual listviews. I've added a new paragraph to top of first post to stress this.

In a virtual listview with many rows and maybe also custom draw code to add column colors you'll get a huge amount of WM_NOTIFY messages. You can get 1000 messages per second. This means that you have to be very careful about the performance of the code. You cannot add a lot of Case or If statements to implement more functionality.

Because the purpose of these functions is large amounts of data with 10,000 rows or more where row numbers are important, and because of performance considerations, I don't think I'll add an option to remove the row number column.

The performance considerations is also the main reason why I've omitted some of the original features.

Embeddable GUI control
A main purpose of these functions is testing and debugging large arrays and CSV files in an easy and fast way, and also to display large arrays and CSV files for end users in an easy and fast way. Using the functions as an embeddable GUI control seems to be more cumbersome and time consuming. Sorry, but I don't think I'll add the code.

Regards Lars.

1 person likes this

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now