Optimized Array.au3

[GEOSoft]

What is the effect of removing the _ArrayDisplay() on the the size of your compiled script ?

One other thing not many people know is that you can run Obfuscator to strip all unused functions from the generated EXE by adding the following Directives in you script:

#Region;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Run_Obfuscator=y
#Obfuscator_Parameters=/cs 0 /cn 0 /cf 0 /cv 0 /sf 1 
#EndRegion;**** Directives created by AutoIt3Wrapper_GUI ****
#Include<array.au3>
Msgbox(0,"test","test")

in this case it reduced the EXE from 319 KB to 259 Kb

You need to use the latest Obfuscator available v 1.0.22 to avoid a bug that was introduced in one of th last version.

I don't have any active projects anymore so I would have to pull one out and change the code to use the regular array.au3 but your figure shows a 19% reduction in file size. Actually that's 60k just on that file alone. I don't know if anything else in your script was calling GUIConstants or not but, if it was then stripping unused constants would reduce it even more.

I just looked and the last time I compiled PX it was 632k so I already wouldn't need to add another 60k to that.

I just downloaded Obfuscator again and filed it away but I think I had already downloaded that version. I still faithfully check for updates I just don't install them.

[Jos]

I don't have any active projects anymore so I would have to pull one out and change the code to use the regular array.au3 but your figure shows a 19% reduction in file size. Actually that's 60k just on that file alone. I don't know if anything else in your script was calling GUIConstants or not but, if it was then stripping unused constants would reduce it even more.

I just looked and the last time I compiled PX it was 632k so I already wouldn't need to add another 60k to that.

I just downloaded Obfuscator again and filed it away but I think I had already downloaded that version. I still faithfully check for updates I just don't install them.

The reason its this much is because these includes are included additionally since they are refferenced by the Array.au3:

1:c:\program files\autoit3\include\array.au3

2:c:\program files\autoit3\include\guiconstants.au3

3:c:\program files\autoit3\include\guidefaultconstants.au3

4:c:\program files\autoit3\include\windowsconstants.au3

5:c:\program files\autoit3\include\aviconstants.au3

6:c:\program files\autoit3\include\comboconstants.au3

7:c:\program files\autoit3\include\datetimeconstants.au3

8:c:\program files\autoit3\include\editconstants.au3

9:c:\program files\autoit3\include\staticconstants.au3

10:c:\program files\autoit3\include\listboxconstants.au3

11:c:\program files\autoit3\include\listviewconstants.au3

12:c:\program files\autoit3\include\sliderconstants.au3

13:c:\program files\autoit3\include\treeviewconstants.au3

14:c:\program files\autoit3\include\updownconstants.au3

15:c:\program files\autoit3\include\guiconstantsex.au3

16:c:\program files\autoit3\include\buttonconstants.au3

17:c:\program files\autoit3\include\tabconstants.au3

18:c:\program files\autoit3\include\progressconstants.au3

19:c:\program files\autoit3\include\guilistview.au3

20:c:\program files\autoit3\include\misc.au3

21:c:\program files\autoit3\include\memory.au3

But lets get back on topic ... the new Array.au3 UDFs...

[randallc]

OK!

So if we

1. Ignore the column display error; everyone wants to ignore that as its too complex (sic!)

2. Don't allow column headers when debugging! - too complex!??.... [huh!?]

A. We still have a bug when there are more than 4000 elements if we are to change over [over 4000 is -not- working in the new version anyway; see my example script above]

B. If we are not going to fix columns, we can simplify more and get rid of Transpose!; it was only meant to rescue those who had it around the wrong way anyway, and would crash their columns. [and transpose is -not- working in the new version anyway; see my example script above]

I was hopeful you would all go my way [although not a complete UDF as it has 2 bugs as I have shown..]; the example of a UDF showing "proper" way to fix columns would be easy to point out, and avoid hassles with bug queries.!

============I'm not fussed, at long as its working properly!=================

Best, randall

Edited October 18, 2007 by randallc

[-Ultima-]

None of your functions worked in displaying more than 4000 items either when I tested them. My function didn't so much display an empty listview as it did pad it with empty items from the top for whatever reason. I'll try to fix it later, but you'll notice that it *did* display the 4000 items at the bottom after all the weird padding.

Edit: The 4000-item limit bug is a bug with the new _GUIListView_InsertItem(). You'll see _ArrayDisplay() show every single item properly with the older versions of AutoIt (that have the old _GUIListViewInsertItem()).

Edit: Nevermind, it was because I was calling the function with the parameters in the wrong place. $sText is supposed to be the second parameter.

Here's the fix:

For $i = ($iLVIAddUDFThreshold + 1) To ($iUBound - 1)
        _GUICtrlListView_InsertItem($hListView, "")
        _GUICtrlListView_SetItemText($hListView, $i, $avArrayText[$i], -1)
    Next

Edit: Regarding the column bug... are you referring to it not transposing the 1D array? If so, I purposely made it not transpose 1D arrays. Why would anyone want to transpose a 1D array...? Having a list run sideways doesn't make it any more readable (and limits it to 250 items too). Check a 2D array, and you'll see that it does work. As for removing it altogether, I actually would have done that had it not been for the "backwards compatibility" stipulation.

So basically, if I'm interpreting what you're saying correctly, then there isn't any bug with the columns. The function doesn't transpose 1D arrays by design.

Edit: I've updated the first post with the above fix for the item insertion over 4000...

Edited October 18, 2007 by -Ultima-

[randallc]

OK,

None of your functions

my functions?... _ArrayDisplay in beta works OK!

You seem to have tried to change everything in the UDF!

Try this one? - look at column 218 and beyond?

Randall

[PS I think you misinterpret my intentions; I am -not- trying to add functionality to a simple script; I am just trying to make it as fool-proof as possible....]

Edited October 18, 2007 by randallc

[-Ultima-]

You seem to have tried to change everything in the UDF!

No, I'm picking and choosing what's practical to be included in what should be a simple display function that's part of a standard UDF... I never tried to interpret your intentions O.o I only interpreted what you were referring to as a bug (as it wasn't clear)...

When I tested, your _ArrayDisplay() didn't show more than 4000 items. I tested the version in your attachment, and the version in the locked bug report thread. Dunno... Here are the screenshots of what I saw:

http://img81.imageshack.us/img81/5369/originalkg4.png (Original)

http://img218.imageshack.us/img218/1773/ra...lcbugfixhz6.png (randallc as posted in the bugfix thread)

http://img81.imageshack.us/img81/8894/randallchb7.png (randallc as included in the ZIP file from the last page)

http://img85.imageshack.us/img85/4568/ultimann7.png (Ultima as included in the ZIP file from the last page; the 4000th item was shown at the bottom as well)

Regardless, I've fixed the 4000 item bug, so it's no longer an issue anyway (not that it was an issue with the code itself; it was just an issue with translating it from the old GUI UDFs to the new GUI UDFs).

As for the truncated column "bug"... AFAIK, that's a known Windows issue with listviews. The number of columns has nothing to do with the problem; it's a problem with the total width of every single column. You can have 22 columns each 10 times wider than each of the 218 columns, and it'll end up showing the same bug. In fact, if you want more proof of this, just press Ctrl+{NUMPAD PLUS} on the listview in your example, and you'll see all the columns going past the same point getting truncated. That's another part of the reason I decided that it wouldn't be worth it to implement the 1D array transposing.

Note: If it seems like I'm being brief/blunt in my responses, don't take any offense from it; I'm running on limited time at the moment ;D

Edited October 19, 2007 by -Ultima-

[randallc]

Hi,

I don't have any gripes about this, and whether...

1. the _ArrayDisplay UDF stays exactly as it is

-or-

2. It changes to your inclusion with all the re-written arrays,

as long as -it works-

There seem to be so many fixes needed for your _ArrayDisplay, that I would treat any update to this separately to the larger re-write, personally.. [ie leave it as it is for now in the beta...] - but not my decision, of course!...

Best, Randall

[-Ultima-]

The only issues you've brought up recently have been:

1) [FIXED] "Copy Selected" not copying everything when nothing is selected

2) [FIXED] Improperly displaying over 4000 items

3) Column header truncation

4) Array transposing

5) Retrofitting column header customization into $sText

6) Setting the default window size based on @DesktopHeight and @DesktopWidth

7) Missing close button when compared with the original _ArrayDisplay()

I've previously responded to pretty much all of them as follows:

3) (Again) a Windows issue, and modifying column widths to attempt to work around that results in this kind of display, which makes it unreadable (almost entirely defeating the purpose of _ArrayDisplay() as a method to see the contents of the array). Really, I'd imagine this probably won't be fixed unless someone can come up with a more reasonable/viable solution. The column count limitation is probably as good as it'll get. Even though it's not perfect (it's far from perfect), it still maintains a higher level of readability than shrinking all of the columns.

4) Not a bug; I'm just not seeing any advantages to be gained by allowing it 1D arrays to be transposed. Transposing *does* work, just not with 1D arrays. I intentionally removed 1D array transposing because of the limitations associated with it due to the column truncation issue.

5) Doesn't fit into the scope of what the UDF's purposes are (notwithstanding my disagreement with the proposed implementation itself -- retrofitting it into $sText is an unclean implementation -- but that's not the bulk of the issue; the scope of the UDF is).

6) The window is resizable... :/

7) Redundant, considering the fact that I can readily count at least 7 ways of easily exiting the dialog off the top of my head:

- pressing Esc on the keyboard

- pressing Alt+F4 on the keyboard

- click the X button at the top-right corner

- double-click the icon at the top-left corner

- click the icon and selecting "Close"

- right-click the titlebar and selecting "Close"

- right-click the taskbar entry and selecting "Close"

_________________________

Of all of the still-open issues you've brought up, I feel that only one of them (the third one) might be fix-worthy, but again, it's a Windows limitation, and the only workaround you've proposed results in an unreadable display. The rest of the issues you've brought up that are open are purely preferential in nature -- nothing that absolutely needs to be fixed. I'm fairly certain I've addressed all of the issues you brought up already (either by actually fixing it, or by explaining why something is the way it is), so I can't be sure why there'd still be "so many fixes needed" for the function.

Regards,

Ultima

Edited October 20, 2007 by -Ultima-

[ChrisL]

-Ultima- I would like to get my _ArraySearch2d() included in array.au3 too http://www.autoitscript.com/forum/index.php?showtopic=57346 perhaps you/others can take a look and see if there are improvements that could be made to speed it up any

[-Ultima-]

I already added support for 2D arrays since my first release though The only difference I see in yours is that you allow searching on multiple columns, but that's trivially done via a for loop that changes the $iSubItem argument when calling _ArraySearch() as I currently have it. Really, I'd expect that columns are independent enough that there would never be a question of searching through a rectangular region of the array serially for a match (otherwise, there'd be no reason to organize them as a 2D array in the first place)...

-----

Oh, and by the way (for anyone using the Array.au3 from the first post), I found that I b0rked _ArraySort() up in a way that caused crashes on 2D arrays. I've fixed it in my copy, but haven't gotten around to uploading it. Why? I tried cleaning _ArraySort()'s helper functions up a few weeks ago, and managed to speed them up by perhaps 15-30% (or thereabouts). So why didn't I just release? Because I found that the speedup didn't seem to scale up well (I think?), so it might slow down for larger arrays. I'll try messing with it some more soon, though.

Edited November 21, 2007 by -Ultima-

[-Ultima-]

Here's a test script for the modified sort function (or at least what I have of it at the moment). The different test cases ($iTestCase) yield different results. For the most part, it's faster than the old _ArraySort(), but oddly enough, it's slower than the old function when sorting 2D arrays with lots of numbers. I'm not really sure why at the moment. The new function does "win" by a wide margin in test case 4 (~90% character strings) though, always beating the old _ArraySort() by around 25%.

At first, I blamed the slowdown on the insertion sort implementation worked into the 2D sort (with the overhead due to copying rows killing any gained speeds), but removing the insertion sort didn't improve the problem (and only served to slow case 4 down)...

*shrug*

[ATTACHMENT REMOVED]

Edit: So uh... yeah, insertion sort is the killer for 2D arrays. WHY is still a mystery to me. Anyhow, I managed to squeeze another 10-15% performance boost or so out of the 1D sorting (for a total of 40% speed increase over the original _ArraySort() on test case 4, and 25% on test case 2). No real progress with 2D array sorting, unless you count commenting out the insertion sort to speed up test case 2 (at the expense of slowing test case 4 down). Ugh. Attachment not updated.

Edited December 28, 2007 by -Ultima-

[Jos]

The ArraySort UDF has been worked on for quite a while by diffeent people to make the speed as optimal as possible and I prefer we leave it as is unless there is a real good reason for changing ...(would hate to brake that baby)

[-Ultima-]

I certainly haven't broken it; just try it It retains the mixed value logic (only optimized). I do believe it remains a stable sort, or at least as stable as Tylo last left it -- I'm not sure he really achieved a true stable sort algorithm, because sorting some array like [["a", 4], ["a", 2], ["a", 8]] by the first column resulted in the array being sorted as [["a", 8], ["a", 2], ["a", 4]] in both the old _ArraySort() and my modified one. Of course, after it gets sorted into that manner, it never changes again (and in that sense, it's stable).

For the most part, the improvements came from the use of "new" syntactical sugar (like $L += 1 instead of $L = $L + 1, the former of which is faster) in addition to some optimizations to the logic conditions (cut down on repetitive calls to IsNumber() where possible, use StringCompare() instead of two calls to String(), etc).

Really, I've tested the function quite extensively (which is the only reason I could keep track of its performance improvements), and the end result is exactly the same in terms of how the elements get sorted. Of course, I'm posting the test scripts so that other people could test it. It would be awesome if you could confirm whether I've improved performance without the expense of a change in the core of the algorithm itself. I'd like to think that just a 25-40% boost in performance for 1D arrays (without changing the algorithm itself) is good enough a reason for using the updated version

The updated version is here:

[ATTACHMENT REMOVED]

I commented the insertion sort out of the 2D sort helper function. For 2D integer arrays with less columns, my function is still faster by (I *think*) ~10%, but it becomes slower by the time it reaches something like 25-50 columns (slower by around 3% at 50, if I'm remembering correctly). 2D mixed/string arrays, on the other hand, remain faster than the old _ArraySort() even at 50 subitems.

Edited December 28, 2007 by -Ultima-

[-Ultima-]

Latest Changes

- Updated function headers to the new style of documentation
- _ArrayDelete() now returns the new size of the array
- _ArrayInsert() now returns the new size of the array
- _ArraySort() updated drastically; up to 40% faster than before (depends on array/data type)
- _ArrayDisplay() and _ArrayDisplayTree() moved outside of Array.au3 into ArrayDisplay.au3

Check the first post for the updates.

Yes, I've now integrated the new _ArraySort() that I posted a few weeks back because after many tests, I found it to work identically to the old _ArraySort() other than being faster. If anyone still isn't convinced, here's the test script:

_ArraySort_Test.au3

I check the sorted arrays to see if they're identical with some simple IsIdentical() function that checks the output arrays from both sorting functions (old and new), element by element.

Edited December 7, 2007 by -Ultima-

[-Ultima-]

Latest Changes

- Cleaned _ArrayDelete() up a little bit more
- Fixed _ArrayInsert() not actually doing anything other than replacing the specified element...
- Fixed _ArraySort() reversing entire array, regardless of $iStart and $iEnd on 1D arrays
- Fixed Au3Check warnings for _ArraySort()

- All benchmarks updated to reflect function usage more accurately
- Example scripts now correctly include <ArrayDisplay.au3>
- Other minor changes (nothing too noteworthy)

Silly little bugs. Check first post for updated attachments, as usual.

It turns out some of my functions had more of an improvement over the original functions than I had expected. Still, at the same time, some functions were a bit slower than I had initially benchmarked them to be (though all were still faster than the original functions anyway -- even if only marginally).

[MadBoy]

So when is it going to be applied? Can't wait to see those included in beta if it's going to happen of course. Regardless, -Ultima- nice work

Edited December 22, 2007 by MadBoy

[GaryFrost]

Here is a self contained version of your latest Array.au3 (no #includes)

I added the _ArrayDisplay in, not the ArrayDisplayTree (too complicated to make self contained).

See what you think.

Edited December 27, 2007 by GaryFrost
Removed attachment

[GaryFrost]

Here is a self contained version of your latest Array.au3 (no #includes)

I added the _ArrayDisplay in, not the ArrayDisplayTree (too complicated to make self contained).

See what you think.

I will need feed back on this.

Also the headers need to be checked, If I process the include for the next beta, the templates for the help are created from the headers.

Enjoy!!!

Edited December 27, 2007 by GaryFrost
Removed attachment

[-Ultima-]

Thanks for doing a bulk of the work, Gary. I've taken the code and worked _GUICtrlListView_GetSelectedIndices() into _ArrayDisplay() like so:

Case $hCopy
                Local $sClip = ""

                ; Get selected indices [ _GUICtrlListView_GetSelectedIndices($hListView, True) ]
                Local $aiCurItems[1] = [0]
                For $i = 0 To GUICtrlSendMsg($hListView, $_ARRAYCONSTANT_LVM_GETITEMCOUNT, 0, 0)
                    If GUICtrlSendMsg($hListView, $_ARRAYCONSTANT_LVM_GETITEMSTATE, $i, 0x2) Then
                        $aiCurItems[0] += 1
                        ReDim $aiCurItems[$aiCurItems[0] + 1]
                        $aiCurItems[$aiCurItems[0]] = $i
                    EndIf
                Next

                ; Generate clipboard text
                If Not $aiCurItems[0] Then
                    For $i = 0 To $iUBound - 1
                        $sClip &= $avArrayText[$i] & @CRLF
                    Next
                Else
                    For $i = 1 To UBound($aiCurItems) - 1
                        $sClip &= $avArrayText[$aiCurItems[$i]] & @CRLF
                    Next
                EndIf
                ClipPut($sClip)

You might want to consider replacing the relevant snippet of code in your _GUICtrlListView_GetSelectedIndices() function with what I used above:

$aiCurItems[0] += 1
                        ReDim $aiCurItems[$aiCurItems[0] + 1]
                        $aiCurItems[$aiCurItems[0]] = $i

Why? It's slower to call UBound() repeatedly, especially when the first element already keeps track of the size (and gets bumped up by one by the end of the If statement anyhow). I'm not sure if a similar thing happens in other places in the GUI*.au3 UDFs, but it's also worth checking out, if only for optimization's sake

As a quick note, the limits are evaluated before going into the For loop, so "GUICtrlSendMsg($hListView, $_ARRAYCONSTANT_LVM_GETITEMCOUNT, 0, 0)" is called only once (just in case anyone wasn't sure whether it would repeatedly call GUICtrlSendMsg() in that loop -- I know I wasn't sure until I tested it xD).

Meanwhile, I've found a bug in the listview filling section:

#include "array.au3"
local $a[4002][3]
$a[4001][2] = "find me if you can!"
_arraydisplay($a)

Quick, spot the mistake! ;D

You forgot to use a nested For loop inside the For loop at line 339 of your attachment, so only the first and second subitems are actually added to the listview. Not to worry, though; I'm currently working on tightening the GUI code some more, so I'm fixing the bug along with it. I'll try to post an updated version of Array.au3 later today, or tomorrow.

Edit:

Local $tItem = DllStructCreate($_ARRAYCONSTANT_tagLVITEM), $pItem = DllStructGetPtr($tItem)
    DllStructSetData($tItem, "Mask", $_ARRAYCONSTANT_LVIF_TEXT)

    For $i = ($iLVIAddUDFThreshold + 1) To ($iUBound - 1)
        $aItem = StringSplit($avArrayText[$i], $sSeparator)
        $iIndex = _Array_GUICtrlListView_InsertItem($hListView, $aItem[1], -1)
        For $j = 2 To $aItem[0]
            $tBuffer = DllStructCreate("char Text[" & (StringLen($aItem[$j]) + 1) & "]")
            $pBuffer = DllStructGetPtr($tBuffer)
            DllStructSetData($tBuffer, "Text", $aItem[$j])
            DllStructSetData($tItem, "item", $iIndex)
            DllStructSetData($tItem, "SubItem", $j-1)
            DllStructSetData($tItem, "Text", $pBuffer)
            GUICtrlSendMsg($hListView, $_ARRAYCONSTANT_LVM_SETITEMA, 0, $pItem)
        Next
    Next

For now, that's the fix. I pulled some assignments outside of the For loops because it's a waste of time/resources to do the same thing over and over. I'm still modifying some stuff, though, so things are still subject to change

Edited December 24, 2007 by -Ultima-

[-Ultima-]

I managed to get _ArrayDisplay() to be entirely self-contained. It no longer relies on external helper functions, and does all of the Dll*() stuff internally. It's much faster too. For example, trying to add 10000 items with 3 subitems using the above-provided version used to take 3.4 seconds, but now takes 1.9 seconds (on my computer anyhow). Adding 40000 items with 3 subitems used to take 17 seconds, but now takes around 7 seconds. Basically, it's scaling up a lot more nicely because the redundancy in calling/creating the DLL-related stuff was minimized.

In total, the function adds 8.45KB uncompiled code to Array.au3 now, but that's obviously a huge improvement over the previous several-hundred-KB-compiled situation Test, test, test away!

@GaryFrost: I'm not sure if you're going to remove your script, but I'd say not to, just in case I screwed something up in the transition (so that we have a known, mostly-working-with-a-minor-bug copy on hand)

[ATTACHMENT REMOVED]

Edited December 28, 2007 by -Ultima-