Variant Datatype Change

[Jon]

I've made some, well er, massive changes to the variant_datatype.cpp.

It was using 128 bytes of memory for each variant which I've cut down to 24 bytes using unions and planned ordering of the variables in the class (the compiler pads variables and sometimes wastes loads of space). This has a dramatic effect on the amount of memory in use when working with large arrays and overall has speeded things up by 10%. It also means that adding new types (binary, handles, dates, maybe) will be easy and won't use up much/if any extra resources. (And Valiks proposed variant changes shouldn't take up too much more space...)

I'd made a rather silly design decision of each variant having all the internal variables for arrays present even when there was no array which wasted about 80 bytes just by itself. This has been removed into a special struct that is only allocated when array functions are needed.

The final change is that I've reverted to the very old system of having different sub datatypes of int and double rather than just converting everything to doubles as is the case now. I had a play with VBscript and have done a similar method of deciding when to convert between ints, doubles and strings. ints will be auto-promoted to __int64 as required which raises the maximum possible integer range to -9223372036854775808 to 9223372036854775807 - i think.

There is a hidden function called VarType($var) that returns the subtype of a variant. Current types are:

1                    // 32bit Integer
2                    // 64bit Integer
3                    // Double float
4                    // String
5                    // Array of variants

This is obviously a massive change that may or may not be used so I've uploaded it into the unstable\xxx directory. I really need people to try out things with these, expressions, arrays, byrefs, memory usage, performance, overflow, etc. and make sure I've not really messed it up. I've tested it with my scripts that build autoit and the helpfiles but they are not really number crunchers...

Let me know what you think

http://www.autoitscript.com/autoit3/files/...ble/autoit/xxx/

Edited August 12, 2004 by Jon

[Jos]

... and overall has speeded things up by 10%. ....http://www.autoitscript.com/autoit3/files/...ble/autoit/xxx/

<{POST_SNAPBACK}>

Ran Tidy.au3 a couple of time for itself (1041 records), which is using arrays and searches etc a lot ...

current unstable 35-37 secs

New XXX Unstable 32-33 secs

looks Consistent with your findings....

[pekster]

Wow.

I ran a benchmark that I created to test the changes. You can see the code below. Basically, it ran a 5000 iteration loop that would re-dimention an array, and then store a value that required a fair amount of power-math to calculate. Using the 3.0.103 beta (August 6th release) I got a benchmark of 65.3 seconds. Using this test reelase, that was down to 35.8. Not bad!

I would also like to create a 2-dimentional array test. Same math function, but it'll expand both array directions, and fill in all newly created array values. I'll post my benchmark results, and a script when I finish that one.

The script I used:

Local $array[1]

$time = TimerStart()
For $i = 1 To 5000
  ReDim $array[$i]
  $array[$i - 1] = Math()
Next
$time = TimerStop($time)

MsgBox(0, "Result", "Benchmark is " & $time / 1000 & " seconds")

Func Math()
  Local $num = Random(2, 50), $i
  For $i = 1 To 10
    $num = $num ^ Round(Random(2, 1000), 3)
    $num = $num ^ (1 / Round(Random(2, 1000), 3))
  Next
  return $num
EndFunc

[Valik]

Jon, there is one minor caveat about my code change. The ID is stored as a 32-bit int. This was to kind of keep the size down (Obviously every Variant has this member), but if there is ever more than 2,147,483,648 Variant's created (Not in existence, just created, ID's are never re-used), then the ID part of the system will fail (I don't expect a full on crash, just unexpected behavior).

Did you also switch from using operator= to initialize data in the constructor to using the member initializer list? When I was messing with the Variant stuff, I think that everything could of been done in the initilizer list instead of inside the constructor's body. That might yield slightly better performance as well.

Edited August 12, 2004 by Valik

[Jon]

Jon, there is one minor caveat about my code change.  The ID is stored as a 32-bit int.  This was to kind of keep the size down (Obviously every Variant has this member), but if there is ever more than 2,147,483,648 Variant's created (Not in existence, just created, ID's are never re-used), then the ID part of the system will fail (I don't expect a full on crash, just unexpected behavior).

Did you also switch from using operator= to initialize data in the constructor to using the member initializer list?  When I was messing with the Variant stuff, I think that everything could of been done in the initilizer list instead of inside the constructor's body.  That might yield slightly better performance as well.

What was the initializer thingy again? I vaguely recall it but didn't understand it.

You could probably use a uint for the ID and simply allow it to wrap around to 0 and rely on the fact that a variant will not be existance by the time it wraps.

[pekster]

Okay, a 2-dimentional version of my previous test has produced the following results with a 150-loop benchmark:

3.0.103 beta (Aug 6th) clocked in at 55.2 seconds

New test release clocked 40.5 seocnds.

The test:

Local $array[1][1]

$time = TimerStart()

For $i = 1 To 150
  ReDim $array[$i][$i]
  For $k = 0 To $i - 2
    $array[$i - 1][$k] = Math()
    $array[$k][$i - 1] = Math()
  Next
  $array[$i - 1][$i - 1] = Math()
Next

$time = TimerStop($time)

MsgBox(0, "Result", "Benchmark is " & $time / 1000 & " seconds")

Func Math()
  Local $num = Random(2, 50), $i
  For $i = 1 To 10
    $num = $num ^ Round(Random(2, 1000), 3)
    $num = $num ^ (1 / Round(Random(2, 1000), 3))
  Next
  return $num
EndFunc

Nice work Jon, Valik, and any other contributing authors

[Nutster]

I would be interested in looking at the source for these changes too. Way to go. Now, let's see what I can do to break this.

[Jon]

Okay, a 2-dimentional version of my previous test has produced the following results with a 150-loop benchmark:

Nice work Jon, Valik, and any other contributing authors

Was it giving you the right answer at the end though?

[Jon]

I would be interested in looking at the source for these changes too.  Way to go.    Now, let's see what I can do to break this.  

it's in the xxx directory too. (variant_datatype.zip)

Should be easy to break, you can't change this amount of code without consequences...

[Valik]

Instead of doing this:

MyConstructor::MyConstructor()
{
    myMember1 = NULL;   // pointer to something or another
    myMember2 = "Hello";     // string object of some type
    myMember3 = 32; // int or similar
    myMember4 = false;  // boolean type
}

Use this:

MyConstructor::MyConstructor() : myMember1(NULL), myMember2("Hello"), myMember3(32), myMember4(false)
{
}

Also, if you think changing from int's to uint's on the ID is better, that's fine. It should be a fairly safe assumption, and short of long-running scripts, I wouldn't expect to cause anything to roll over, anway, not to mention, the ID is only used in a couple places at all (Currently).

Edited August 12, 2004 by Valik