Jump to content

Copying Data to/from C# or VB.NET Through IntPtr


Recommended Posts

Hi Lars,

This is again a realy nice example, which shows the interaction between AU3 and .NET.
Making the AU3 functionality more extensible and powerfull.

A nice real live use case Example for scripters would be to have PowerShell interact, bi-directional with AU3 using this technique.

The reason is the PS is much more accessible for most users / scripters than C# or vb.NET

using System;
using System.Collections.Generic;
using System.Management.Automation;
using System.Management.Automation.Runspaces;
using System.Threading.Tasks;

public class Program
{
    static void Main()
    {
        // create empty pipeline
        PowerShell ps = PowerShell.Create();

        // add command
        ps.AddCommand("Get-Date");

        // run command(s)
        Console.WriteLine("Date: {0}", ps.Invoke().First());

        Console.ReadLine();
    }
}

Since PS is based on .Net, that should be feasible I guess - using the System.Management.Automation class.

Thanks again for sharing !!

Link to post
Share on other sites

I'm not really a .NET coder and I'm definitely not a PowerShell coder. So I'll not go into this. But I need fast AutoIt code. And here C# and VB.NET come into the picture.

Link to post
Share on other sites
On 10/28/2021 at 10:25 AM, ptrex said:

Since PS is based on .Net, that should be feasible I guess - using the System.Management.Automation class.

Isn't it more or less already implemented in the example here that shows how to execute PowerShell code directly in an AutoIt script.

Link to post
Share on other sites

Sure  this was already implemented running PS scripts in a .NET PowerShell Host. 

But there are subtle technical difference and limitations involved. 

The prior PS Host that runs a script, is running in an Appdomain Object in the unmanaged environment.

So it runs in an isolated environment (and even in a seperate process if I remember well)

So the limitation is that you can't interact between the host and the PS process. 

 

While your example above makes interacting (Copying Data to/from C# or VB.NET Through IntPtr) between the 2 environments possible.

 

But well noticed of you, that the foundation blocks are already there. It is now a question of putting the pieces together, and test if it works as expected...

And example would be the copy console output from a PS script to Au3, or the other way around.

 

Link to post
Share on other sites

Something in this direction (no AutoIt available at the moment)
In powershell you can reach win32, .net and you need a way to communicate back in below example just sending a message to notepad but that can be an AutoIt window.

add-type -assemblyname System
add-type -assemblyname System.Drawing
#  [System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")
$winApi = add-type  -name user32 -passThru `
   -memberDefinition '

  [DllImport("user32.dll")] public static extern IntPtr FindWindowEx(IntPtr hwndParent, IntPtr hwndChildAfter,String lpszClass,String lpszWindow);

  [DllImport("User32.dll")] public static extern IntPtr SendMessage(IntPtr hWnd,int uMsg, int wParam, string lParam);'

$notepad_1 = start-process -passThru notepad

start-sleep 1

$hwndNotepad_1 = $notepad_1.MainWindowHandle

$hwndEdit_1    = $winApi::FindWindowEx($hwndNotepad_1, [IntPtr]::Zero, "Edit", $null)
$null          = $winApi::SendMessage($hwndEdit_1, 0x000C, 0, "This is the first notepad window.")

Class TestClass {
  [intptr]Test() {

   $bmp = New-Object System.Drawing.Bitmap(320, 240)

    for ($i = 0; $i -lt 100; $i++) {
       for ($j = 0; $j -lt 100; $j += 2) {
         $bmp.SetPixel($i, $j, 'Red')
         $bmp.SetPixel($i, $j + 1, [System.Drawing.Color]::FromArgb(0, 100, 200))
       }
    }

    $bmp.Save("test.bmp")

    $newintptr = New-Object system.Intptr 
    $newIntPtr = $bmp.GetHbitmap()
    return $newintptr
    }
}

$bmp1=new-object testclass

$null          = $winApi::SendMessage($hwndEdit_1, 0x000C, 0, "The pointer address " + $bmp1.test().tostring())


#[IntPtr]$Pointer = [System.Runtime.InteropServices.Marshal]::AllocHGlobal($Bytes.Length)
#[System.Runtime.InteropServices.Marshal]::Copy($Bytes, 0, $Pointer, $Bytes.Length)

In powershell you cannot create COM objects as far as I know but you could use

1. sendmessage as shown above

2. tcp client/server there are many powershell examples around to make a simple tcpip server and client in AutoIt should not be hard

3. .....

Link to post
Share on other sites

This explains more in detail why we are not able to have data flowing back and forth using the .Net Host in AutoIt.

Unless there is a way around it ...

Application Domain
Application Domain or AppDomain is one of the most powerful features of the .NET framework. AppDomain can be considered as a light-weight process. An application domain is a .NET concept whereas the process is an operating system concept. Both have many characteristics in common. A process can contain multiple app domains and each one provides a unit of isolation within that process. Most importantly, only those applications are written in .net, or in other words, only managed applications will have application domain.
 
Application domain provides isolation between code running in different app domains. App Domain is a logical container for code and data just like process and has separate memory space and access to resources. App domain also serves as a boundary like a process that does avoid any accidental or illegal attempts to access the data of an object in one running application from another.
 
System.AppDomain class provides you ways to deal with the application domain. It provides methods to create a new application domain, unload domain from memory, etc.
...
 
unmanaged code has no application domain.
Code and data are safely isolated using the boundary provided by the AppDomain.
if two AppDomains want to communicate with each other pass objects, .NET techniques such as Remoting or Web Services should be used.
 
See here for the full story:
 
Link to post
Share on other sites
Link to post
Share on other sites

Image example
In the PowerShell code in this post by junkew there are two examples. One is a NotePad example and the other is about generating an image. The example here is about generating an image.

PSScripts.au3: 

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=Y

Opt( "MustDeclareVars", 1 )

#include <GDIPlus.au3>
#include <GUIConstantsEx.au3>

#include "DotNetAll.au3"

Example()

Func Example()
  Local $oNetCode = DotNet_LoadCScode( FileRead( "PSScripts.cs" ), "System.Core.dll | C:\Windows\Microsoft.NET\assembly\GAC_MSIL\System.Management.Automation\v4.0_3.0.0.0__31bf3856ad364e35\System.Management.Automation.dll" )
  Local $oPSClass = DotNet_CreateObject( $oNetCode, "PSClass" )

  Local $pBitmap = $oPSClass.RunScript( FileRead( "tst00.ps1" ) )

  ; Create GUI
  Local $hGui = GUICreate( "GDI+ Through .NET", 320, 240 )
  GUISetState( @SW_SHOW )

  ; Initialize GDI+
  _GDIPlus_Startup()

  ; Draw bitmap to GUI
  Local $hBitmap = _GDIPlus_BitmapCreateFromHBITMAP( $pBitmap )
  Local $hGraphic = _GDIPlus_GraphicsCreateFromHWND( $hGui )
  _GDIPlus_GraphicsDrawImage( $hGraphic, $hBitmap, 0, 0 )

  ; Clean up resources
  _GDIPlus_GraphicsDispose( $hGraphic )
  _GDIPlus_BitmapDispose( $hBitmap )
  _WinAPI_DeleteObject( $pBitmap )

  ; Shut down GDI+
  _GDIPlus_Shutdown()

  ; Loop
  Do
  Until GUIGetMsg() = $GUI_EVENT_CLOSE
EndFunc

PSScripts.cs: 

using System.Management.Automation;
using System.Management.Automation.Runspaces;
using System.Collections.ObjectModel;

public class PSClass {
  public string RunScript( string script ) {
    Runspace runspace = RunspaceFactory.CreateRunspace();
    runspace.Open();

    Pipeline pipeline = runspace.CreatePipeline();
    pipeline.Commands.AddScript( script );
    pipeline.Commands.Add( "Out-String" );

    Collection<PSObject> results = pipeline.Invoke();

    runspace.Close();
    return results[0].ToString();
  }
}

tst00.ps1: 

add-type -assemblyname System
add-type -assemblyname System.Drawing

Class TestClass {
  [intptr]Test() {

   $bmp = New-Object System.Drawing.Bitmap(320, 240)

    for ($i = 0; $i -lt 100; $i++) {
       for ($j = 0; $j -lt 100; $j += 2) {
         $bmp.SetPixel($i, $j, 'Red')
         $bmp.SetPixel($i, $j + 1, 150)
       }
    }

    #$bmp.Save("test.bmp")

    $newintptr = New-Object system.Intptr 
    $newIntPtr = $bmp.GetHbitmap()
    return $newintptr
    }
}

$bmp1=new-object testclass
echo $bmp1.test()

Run PSScripts.au3. Only tested on Windows 10.

PowerShell.7z

Edited by LarsJ
Link to post
Share on other sites
Link to post
Share on other sites

You have to do it this way: 

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=y

Opt( "MustDeclareVars", 1 )

#include "DotNetAll.au3"

Example()

Func Example()
  Local $oNetCode = DotNet_LoadCScode( FileRead( "PSScripts.cs" ), "System.Core.dll | C:\Windows\Microsoft.NET\assembly\GAC_MSIL\System.Management.Automation\v4.0_3.0.0.0__31bf3856ad364e35\System.Management.Automation.dll" )
  Local $oPSClass = DotNet_CreateObject( $oNetCode, "PSClass" ), $sOutput

  $sOutput = $oPSClass.RunScript( "Get-Process" )
  ConsoleWrite( $sOutput )
EndFunc

All of these examples (the example here, the example in the post above, and the examples in this post) work like this: C# code executes a PowerShell script, and returns output back to AutoIt. If the output is textual information, you can display the information in SciTE console. If the output is an image bitmap handle you can display the image in a GUI.

Link to post
Share on other sites

NotePad example
In the PowerShell code in this post by junkew there are two examples. One is a NotePad example and the other is about generating an image.The example here is about the NotePad example.

PSScripts.au3: 

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=y

Opt( "MustDeclareVars", 1 )

#include "DotNetAll.au3"

Example()

Func Example()
  Local $oNetCode = DotNet_LoadCScode( FileRead( "PSScripts.cs" ), "System.Core.dll | C:\Windows\Microsoft.NET\assembly\GAC_MSIL\System.Management.Automation\v4.0_3.0.0.0__31bf3856ad364e35\System.Management.Automation.dll" )
  Local $oPSClass = DotNet_CreateObject( $oNetCode, "PSClass" ), $sOutput

  $sOutput = $oPSClass.RunScript( FileRead( "tst01.ps1" ) )
EndFunc

tst01.ps1: 

$winApi = add-type  -name user32 -passThru `
  -memberDefinition '

  [DllImport("user32.dll")] public static extern IntPtr FindWindowEx(IntPtr hwndParent, IntPtr hwndChildAfter,String lpszClass,String lpszWindow);

  [DllImport("User32.dll")] public static extern IntPtr SendMessage(IntPtr hWnd,int uMsg, int wParam, string lParam);'

$notepad_1 = start-process -passThru notepad

start-sleep 1

$hwndNotepad_1 = $notepad_1.MainWindowHandle

$hwndEdit_1    = $winApi::FindWindowEx($hwndNotepad_1, [IntPtr]::Zero, "Edit", $null)
$null          = $winApi::SendMessage($hwndEdit_1, 0x000C, 0, "This is the first notepad window.")

PSScripts.cs is exactly the same as in the image example.
 

Error handling
Errors in PSScripts.cs are sent to SciTE console. But since PSScripts.cs is copied directly from this 2 year old example without any changes at all, we already know that the C#-code is error free.

Errors in PowerShell scripts don't appear in SciTE console.  To catch these errors, use a COM error handler as shown in the example of the ObjEvent() function in the help file.

When I first ran all junkew's code at once, the AutoIt script failed: 

"...\PSScripts.au3" (15) : ==> The requested action with this object has failed.:
$sOutput = $oPSClass.RunScript( FileRead( "tst00.ps1" ) )
$sOutput = $oPSClass^ ERROR

After adding a COM error handler, this error was detected in the PowerShell code: 

PSScripts.au3 (17) : ==> COM Error intercepted !
  err.number is:        0x80020009
  err.windescription:   Exception occurred.
  err.description is:   At line:27 char:37
+          $bmp.SetPixel($i, $j + 1, [System.Drawing.Color]::FromArgb(0 ...
+                                     ~~~~~~~~~~~~~~~~~~~~
Unable to find type [System.Drawing.Color].
  err.source is:        System.Management.Automation
  err.helpfile is:   
  err.helpcontext is:   0
  err.lastdllerror is:  0
  err.scriptline is:    17
  err.retcode is:       0x80131501

After replacing "[System.Drawing.Color]::FromArgb(0, 100, 200)" with the numeric value 150, the code was executed flawlessly. 

All PowerShell code cannot simply be executed in exactly the same way with this technique as executed directly in the PowerShell window.
 

Recapitulation
The posts above are about running PowerShell scripts in AutoIt using C#-code. The C#-code in PSScripts.cs is the essential code that makes it all work. PSScripts.cs is copied directly from this 2 year old example without any changes at all: 

using System.Management.Automation;
using System.Management.Automation.Runspaces;
using System.Collections.ObjectModel;

public class PSClass {
  public string RunScript( string script ) {
    Runspace runspace = RunspaceFactory.CreateRunspace();
    runspace.Open();

    Pipeline pipeline = runspace.CreatePipeline();
    pipeline.Commands.AddScript( script );
    pipeline.Commands.Add( "Out-String" );

    Collection<PSObject> results = pipeline.Invoke();

    runspace.Close();
    return results[0].ToString();
  }
}

The code in RunScript() method takes a PowerShell script as input in the string variable "script". The PowerShell code is executed and output, which is normally sent to the PowerShell window/console, is stored in the "results" variable, which is an array of PSObjects. When the PowerShell script exits, the output (in "results") is returned as a string. This string is returned to the AutoIt code.

On 11/4/2021 at 8:32 PM, ptrex said:

And example would be the copy console output from a PS script to Au3, or the other way around.

These examples are precisely about copy console output from a PS script to Au3.

Edited by LarsJ
Link to post
Share on other sites

Hi Larsj,

Thanks for the example running the PS Script.

It is indeed returning the output back to consolewrite as you metioned ! 👌

I must study it a bit more why on prior attempts it was not possible to get anything return back to AutoIT ?

 

.Net Core and PowerShell Core are really powerfull stuff since they are cross platform running 1 source code on Windows / Linux and Mac.

I just funished a series of posts about this. One of them is how to use PS on Linux.

https://audministrator.wordpress.com/2021/10/11/raspberry-pi-installing-ms-powershell-core

In the meantime I created a PowerShell host app in C# that runs on Linux as well.

 

So interacting with .Net and PowerShell is definitely here to stay ! 🙂

Link to post
Share on other sites

C# Pointers
One difference between C# and VB.NET is that C# offers more advanced and specialized functionality. In C# it's possible to use pointers to further speed up the code. This isn't possible in VB.NET. This post is about performance optimizing C# code using pointers.

C# pointers in this thread are interesting because the IntPtr technique makes it possible to copy even very large amounts of data very fast. Therefore, it's also interesting with very fast code to process these large amounts of data.

C# and VB.NET code perform pretty much as well as C/C++ code. C# code based on pointers performs as well as C/C++ code based on pointers.
 

Unsafe code
The part of C# code that uses pointers must be marked as unsafe code. When compiling the code, use the /unsafe compiler option.

In DotNetAll.au3, DotNet_LoadCScode() is used to load and compile C# code: 

DotNet_LoadCScode( $sCode, $sReferences = "", $oAppDomain = 0, $sFileName = "", $sCompilerOptions = "" )

The function is used this way to load and compile C# code with pointers: 

Local $oNetCode = DotNet_LoadCScode( FileRead( "Example01.cs" ), "System.dll", 0, "", "/unsafe" )

 

Simple example
Example01.au3 and Example01.cs in \Examples\4) C# Pointers\ is a simple example that demonstrates the use of pointers: 

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

#AutoIt3Wrapper_UseX64=Y

Opt( "MustDeclareVars", 1 )

#include "..\..\Includes\DotNetAll.au3"

Example()

Func Example()
  Local $oNetCode = DotNet_LoadCScode( FileRead( "Example01.cs" ), "System.dll", 0, "", "/unsafe" )
  Local $oTestClass = DotNet_CreateObject( $oNetCode, "TestClass" )
  $oTestClass.Test()
EndFunc

 

using System;

class TestClass {
  public void Test() {
    int i;         // Integer variable i.
    unsafe {       // Unsafe code block with pointers.
      int *p = &i; // Integer type pointer p is initialized with the memory address of i.
      *p = 123;    // The memory address to which pointer p refers is set to the value 123.
    }              // Since pointer p refers to the memory address of i, the value of i is
    Console.WriteLine(i); // actually set to 123.
  }
}

 

Image example
The other examples are translations of the image manipulation example at the end of first post. The example is translated from VB.NET code to first C# code without pointers and then C# code with pointers.

Example02.au3 and Example02.vb is the original example with VB.NET code: 

Imports System
Imports System.Drawing
Imports System.Drawing.Imaging
Imports System.Runtime.InteropServices

Class TestClass
  Public Function Test() As IntPtr
    'Create bitmap
    Dim bmp As New Bitmap( "Test.bmp" )

    'Lock the bits in the bitmap
    Dim rect As New Rectangle( 0, 0, bmp.Width, bmp.Height )
    Dim bmpData As BitmapData = bmp.LockBits( rect, ImageLockMode.ReadWrite, bmp.PixelFormat )

    'Get the address of the first line
    Dim ptr As IntPtr = bmpData.Scan0

    'Declare an array to hold the bytes of the bitmap
    'This code is specific to a bitmap with 24 bits per pixel (bpp)
    Dim iBytes As Integer = Math.Abs( bmpData.Stride ) * bmp.Height
    Dim aRGBValues(iBytes-1) As Byte

    'Copy the RGB values into the array
    Marshal.Copy( ptr, aRGBValues, 0, iBytes )

    'Set every third value to 255, a 24bpp image will look red
    For i As Integer = 2 To aRGBValues.Length - 1 Step 3
      aRGBValues(i) = 255
    Next

    'Copy the RGB values back to the bitmap
    Marshal.Copy( aRGBValues, 0, ptr, iBytes )

    'Unlock the bits
    bmp.UnlockBits( bmpData )

    'Create GDI bitmap
    Dim hBitmap As IntPtr = bmp.GetHbitmap()

    'Return GDI bitmap
    Return hBitmap
  End Function
End Class

Example03.au3 and Example03.cs is a translation of the VB.NET code into C# code: 

using System;
using System.Drawing;
using System.Drawing.Imaging;
using System.Runtime.InteropServices;

class TestClass {
  public IntPtr Test() {
    // Create bitmap
    Bitmap bmp = new Bitmap( "Test.bmp" );

    // Lock the bits in the bitmap
    Rectangle rect = new Rectangle( 0, 0, bmp.Width, bmp.Height );
    BitmapData bmpData = bmp.LockBits( rect, ImageLockMode.ReadWrite, bmp.PixelFormat );

    // Get the address of the first line
    IntPtr ptr = bmpData.Scan0;

    // Declare an array to hold the bytes of the bitmap
    // This code is specific to a bitmap with 24 bits per pixel (bpp)
    int iBytes = Math.Abs( bmpData.Stride ) * bmp.Height;
    byte[] aRGBValues = new byte[iBytes];

    // Copy the RGB values into the array
    Marshal.Copy( ptr, aRGBValues, 0, iBytes );

    // Set every third value to 255, a 24bpp image will look red
    for ( int i = 2; i < aRGBValues.Length; i += 3 ) aRGBValues[i] = 255;

    // Copy the RGB values back to the bitmap
    Marshal.Copy( aRGBValues, 0, ptr, iBytes );

    // Unlock the bits
    bmp.UnlockBits( bmpData );

    // Create GDI bitmap
    IntPtr hBitmap = bmp.GetHbitmap();

    // Return GDI bitmap
    return hBitmap;
  }
}

Example04.au3 and Example04.cs is an implementation of the C# code using pointers: 

using System;
using System.Drawing;
using System.Drawing.Imaging;
using System.Runtime.InteropServices;

class TestClass {
  public unsafe IntPtr Test() { // Unsafe function
    // Create bitmap
    Bitmap bmp = new Bitmap( "Test.bmp" );

    // Lock the bits in the bitmap
    Rectangle rect = new Rectangle( 0, 0, bmp.Width, bmp.Height );
    BitmapData bmpData = bmp.LockBits( rect, ImageLockMode.ReadWrite, bmp.PixelFormat );

    // Get the address of the first line
    byte* ptr = (byte*)bmpData.Scan0.ToPointer();

    // Declare an array to hold the bytes of the bitmap
    // This code is specific to a bitmap with 24 bits per pixel (bpp)
    int iBytes = Math.Abs( bmpData.Stride ) * bmp.Height;
    byte[] aRGBValues = new byte[iBytes]; // Normal pointer to an object

    // Copy the RGB values into the array
    Marshal.Copy( (IntPtr)ptr, aRGBValues, 0, iBytes );

    // Set every third value to 255, a 24bpp image will look red
    // Must pin object on heap so that it doesn't move while using pointers
    fixed ( byte* p = &aRGBValues[2] ) { // aRGBValues[2] = First red color value
      // p is pinned as well as object, so create another pointer to increment it
      for ( byte* p2 = p; p2 < p + aRGBValues.Length; p2 += 3 ) *p2 = 255;
    }

    // Copy the RGB values back to the bitmap
    Marshal.Copy( aRGBValues, 0, (IntPtr)ptr, iBytes );

    // Unlock the bits
    bmp.UnlockBits( bmpData );

    // Create GDI bitmap
    IntPtr hBitmap = bmp.GetHbitmap();

    // Return GDI bitmap
    return hBitmap;
  }
}

 

Performance
It makes no sense to performance test the three examples above because the test image is a small image and the image manipulation is simple and can be performed in a very tight fast loop. They'll pretty much all three be equally fast.
 

7z-file
New 7z-file at bottom of first post.

Edited by LarsJ
Link to post
Share on other sites

Struct Array
In Example03 above, many instances of a larger structure are copied from .NET to AutoIt. The total time spent both generating and copying data is measured. In order to measure the time used exclusively for copying data, it's necessary to split the code into two functions: A function to generate data and a function to copy data. This can be done by storing all instances of the structure in an array of structures.

To copy instances of the structure from AutoIt back to .NET, it's convenient to store the structures in an array in the .NET code.

Example03 below is a new version where it's possible to measure structure generation times and transfer times separately.

Similarly, Example02 is a new version where it's possible to measure array generation times and transfer times separately.

Note that code to generate duplicates is included in Rand2dArrayCreate() and RandStructsCreate() functions.
 

Example02

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

#AutoIt3Wrapper_UseX64=Y

Opt( "MustDeclareVars", 1 )

#include <Array.au3>
#include "Random2dArray.au3"

Example(     100, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
Example(   50000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
Example(  100000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
Example(  250000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
;Example(  750000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
;Example( 2000000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )

Func Example( _
  $iRows, _       ; Number of rows in the array
  $aColumns, _    ; Number of columns and data types in the array, see 1) in docu
  $sSource = "" ) ; Character source for string columns, random strings are created from these characters

  ; Load .NET code, create object
  Local Static $oNetCode = 0, $oRand2dArrayClass
  If Not $oNetCode Then
    $oNetCode = DotNet_LoadVBcode( FileRead( "Example02.vb" ), "System.dll" )
    $oRand2dArrayClass = DotNet_CreateObject( $oNetCode, "Rand2dArrayClass" )
  EndIf

  ; Check $aColumns array
  Local $iError = CheckColumnsArray( $aColumns )
  If $iError Then Return SetError($iError,0,0)

  ; Define $aColTypes array
  Local $iColumns = UBound( $aColumns )
  Local $aColTypes[6][$iColumns+1]

  ; Update $aColTypes and $aColumns
  For $i = 0 To $iColumns - 1
    If CalcColTypes( $i, $aColumns, $aColTypes, $iRows, $sSource ) Then ContinueLoop
    $aColumns[$i][2] += 1 ; Because of .NET Random class
    ; Convert duplicates as a percentage to number of unique rows
    $aColumns[$i][3] = $iRows - Int( $iRows * $aColumns[$i][3] / 100 )
    If $aColumns[$i][3] And $aColumns[$i][3] < $iRows Then $aColumns[$i][3] -= 1
  Next

  ; Visual code verification
  If $iRows = 100 Then
    $oRand2dArrayClass.Rand2dArrayCreate( $iRows, $aColumns, $aColTypes, $sSource )
    Local $aArray100 = $oRand2dArrayClass.Rand2dArrayToAutoIt()
    $oRand2dArrayClass.Rand2dArrayDelete()
    _ArrayDisplay( $aArray100 )
    Return
  EndIf

  ; Number of rows
  ConsoleWrite( "Array rows     = " & $iRows & @CRLF )

  ; Create array
  Local $hTimer = TimerInit()
  $oRand2dArrayClass.Rand2dArrayCreate( $iRows, $aColumns, $aColTypes, $sSource )
  ConsoleWrite( "Create array   = " & TimerDiff( $hTimer ) & @CRLF )

  ; Pass array to AutoIt
  $hTimer = TimerInit()
  Local $aArray = $oRand2dArrayClass.Rand2dArrayToAutoIt()
  ConsoleWrite( "Pass to AutoIt = " & TimerDiff( $hTimer ) & @CRLF )

  ; Delete .NET array
  $oRand2dArrayClass.Rand2dArrayDelete()

  ; Pass array to .NET
  $hTimer = TimerInit()
  $oRand2dArrayClass.Rand2dArrayToDotNet( $aArray )
  ConsoleWrite( "Pass to .NET   = " & TimerDiff( $hTimer ) & @CRLF & @CRLF )

  ; Delete .NET array
  $oRand2dArrayClass.Rand2dArrayDelete()
EndFunc

#cs
Array rows     = 50000
Create array   = 161.3358744048
Pass to AutoIt = 53.1796500402687
Pass to .NET   = 128.998994597868

Array rows     = 100000
Create array   = 276.623278397762
Pass to AutoIt = 108.432993923537
Pass to .NET   = 258.063372194661

Array rows     = 250000
Create array   = 898.781576733183
Pass to AutoIt = 264.890631696736
Pass to .NET   = 631.123803539603

Array rows     = 750000
Create array   = 2717.34763337093
Pass to AutoIt = 837.568960363219
Pass to .NET   = 2094.36373615079

Array rows     = 2000000
Create array   = 7424.69637769875
Pass to AutoIt = 2081.94096635518
Pass to .NET   = 5806.68962274841
#ce

 

Imports System

Class Rand2dArrayClass
  'Global in class
  Dim aObjects(,)

  Public Sub Rand2dArrayCreate( iRows As Integer, aColumns As Object(,), aColTypes As Object(,), sSource As String )
    Dim iColumns As Integer = aColumns.GetUpperBound(1) + 1
    Dim aSource As Char() = sSource.ToCharArray()
    Dim iSourceLen As Integer = sSource.Length
    ReDim aObjects(iColumns-1,iRows-1)
    Dim oRnd As New Random()

    Dim sStr As String = ""
    Dim k, y, m, dMax As Integer

    For i As Integer = 0 To iRows - 1
      'String columns
      For j As Integer = 1 To aColTypes(0,0)
        k = aColTypes(j,0)
        If i <= aColumns(3,k) Then
          For l As Integer = 1 To oRnd.Next( aColumns(1,k), aColumns(2,k) )
            sStr &= aSource(oRnd.Next( iSourceLen ))
          Next
          aObjects(k,i) = sStr
          sStr = ""
        Else
          aObjects(k,i) = aObjects(k,oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Integer columns
      For j As Integer = 1 To aColTypes(0,1)
        k = aColTypes(j,1)
        If i <= aColumns(3,k) Then
          aObjects(k,i) = oRnd.Next( aColumns(1,k), aColumns(2,k) )
        Else
          aObjects(k,i) = aObjects(k,oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Float columns
      For j As Integer = 1 To aColTypes(0,2)
        k = aColTypes(j,2)
        If i <= aColumns(3,k) Then
          aObjects(k,i) = aColumns(1,k) + ( aColumns(2,k) - aColumns(1,k) ) * oRnd.NextDouble()
        Else
          aObjects(k,i) = aObjects(k,oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Date columns
      For j As Integer = 1 To aColTypes(0,3)
        k = aColTypes(j,3)
        If i <= aColumns(3,k) Then
          y = oRnd.Next( aColumns(1,k), aColumns(2,k) )
          m = oRnd.Next( 1, 13 )
          Select m
            Case 1, 3, 5, 7, 8, 10, 12
              dMax = 31
            Case 4, 6, 9, 11
              dMax = 30
            Case Else '2
              dMax = If( y Mod 4 <> 0, 28, If( y Mod 400 = 0, 29, If( y Mod 100 = 0, 28, 29 ) ) )
          End Select
          aObjects(k,i) = y * 10000 + m * 100 + oRnd.Next( 1, dMax + 1 )
        Else
          aObjects(k,i) = aObjects(k,oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Time columns
      For j As Integer = 1 To aColTypes(0,4)
        k = aColTypes(j,4)
        If i <= aColumns(3,k) Then
          aObjects(k,i) = oRnd.Next( aColumns(1,k), aColumns(2,k) ) * 10000 + oRnd.Next( 0, 60 ) * 100 + oRnd.Next( 0, 60 )
        Else
          aObjects(k,i) = aObjects(k,oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Row/col columns
      For j As Integer = 1 To aColTypes(0,5)
        k = aColTypes(j,5)
        aObjects(k,i) = i & "/" & k
      Next
    Next
  End Sub

  Public Function Rand2dArrayToAutoIt() As Object(,)
    Return aObjects
  End Function

  Public Sub Rand2dArrayToDotNet( aArray As Object(,) )
    aObjects = aArray
  End Sub

  Public Sub Rand2dArrayDelete()
    ReDim aObjects(0,0)
  End Sub
End Class


Example03

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

#AutoIt3Wrapper_UseX64=Y

Opt( "MustDeclareVars", 1 )

#include "Random2dArray.au3"

Example(     100, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
Example(   50000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
Example(  100000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
Example(  250000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
;Example(  750000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )
;Example( 2000000, "sifdtr", "abcdefghijklmnopqrstuvwxyz" )

Func Example( _
  $iRows, _       ; Number of rows in the array
  $aColumns, _    ; Number of columns and data types in the array, see 1) in docu
  $sSource = "" ) ; Character source for string columns, random strings are created from these characters

  ; Load .NET code, create object
  Local Static $oNetCode = 0, $oRandStructsClass
  If Not $oNetCode Then
    $oNetCode = DotNet_LoadVBcode( FileRead( "Example03.vb" ), "System.dll" )
    $oRandStructsClass = DotNet_CreateObject( $oNetCode, "RandStructsClass" )
  EndIf

  ; Check $aColumns array
  Local $iError = CheckColumnsArray( $aColumns )
  If $iError Then Return SetError($iError,0,0)

  ; Define $aColTypes array
  Local $iColumns = UBound( $aColumns )
  Local $aColTypes[6][$iColumns+1]

  ; Update $aColTypes and $aColumns
  For $i = 0 To $iColumns - 1
    If CalcColTypes( $i, $aColumns, $aColTypes, $iRows, $sSource ) Then ContinueLoop
    $aColumns[$i][2] += 1 ; Because of .NET Random class
    ; Convert duplicates as a percentage to number of unique rows
    $aColumns[$i][3] = $iRows - Int( $iRows * $aColumns[$i][3] / 100 )
    If $aColumns[$i][3] And $aColumns[$i][3] < $iRows Then $aColumns[$i][3] -= 1
  Next

  ; Visual code verification
  If $iRows = 100 Then
    $oRandStructsClass.RandStructsCreate( $iRows, $aColumns, $aColTypes, $sSource )
    Local $pPtr100 = $oRandStructsClass.RandStructsToAutoIt( $iRows ), $tStruct, $tString, $iSize = @AutoItX64 ? 56 : 40
    ConsoleWrite( "$pPtr100 = " & Ptr( $pPtr100 ) & @CRLF & @CRLF )

    ; Display first structure
    ConsoleWrite( "First structure:" & @CRLF )
    $tStruct = DllStructCreate( "int iString;ptr pString;int iInteger;double dDouble;int iDate;int iTime;int iRowCol;ptr pRowCol", $pPtr100 )
    ConsoleWrite( "$tStruct.iString  = " &      $tStruct.iString   & @CRLF )
    ConsoleWrite( "$tStruct.pString  = " & Ptr( $tStruct.pString ) & @CRLF )
    $tString = DllStructCreate( "char sString[" & $tStruct.iString & "]", $tStruct.pString )
    ConsoleWrite( "$tString.sString  = " &      $tString.sString   & @CRLF )
    ConsoleWrite( "$tStruct.iInteger = " &      $tStruct.iInteger  & @CRLF )
    ConsoleWrite( "$tStruct.dDouble  = " &      $tStruct.dDouble   & @CRLF )
    ConsoleWrite( "$tStruct.iDate    = " &      $tStruct.iDate     & @CRLF )
    ConsoleWrite( "$tStruct.iTime    = " &      $tStruct.iTime     & @CRLF )
    ConsoleWrite( "$tStruct.iRowCol  = " &      $tStruct.iRowCol   & @CRLF )
    ConsoleWrite( "$tStruct.pRowCol  = " & Ptr( $tStruct.pRowCol ) & @CRLF )
    $tString = DllStructCreate( "char sString[" & $tStruct.iRowCol & "]", $tStruct.pRowCol )
    ConsoleWrite( "$tString.sString  = " &      $tString.sString   & @CRLF & @CRLF )

    ; Display last structure
    ConsoleWrite( "Last structure:" & @CRLF )
    $tStruct = DllStructCreate( "int iString;ptr pString;int iInteger;double dDouble;int iDate;int iTime;int iRowCol;ptr pRowCol", $pPtr100 + $iSize * ( $iRows - 1 ) )
    ConsoleWrite( "$tStruct.iString  = " &      $tStruct.iString   & @CRLF )
    ConsoleWrite( "$tStruct.pString  = " & Ptr( $tStruct.pString ) & @CRLF )
    $tString = DllStructCreate( "char sString[" & $tStruct.iString & "]", $tStruct.pString )
    ConsoleWrite( "$tString.sString  = " &      $tString.sString   & @CRLF )
    ConsoleWrite( "$tStruct.iInteger = " &      $tStruct.iInteger  & @CRLF )
    ConsoleWrite( "$tStruct.dDouble  = " &      $tStruct.dDouble   & @CRLF )
    ConsoleWrite( "$tStruct.iDate    = " &      $tStruct.iDate     & @CRLF )
    ConsoleWrite( "$tStruct.iTime    = " &      $tStruct.iTime     & @CRLF )
    ConsoleWrite( "$tStruct.iRowCol  = " &      $tStruct.iRowCol   & @CRLF )
    ConsoleWrite( "$tStruct.pRowCol  = " & Ptr( $tStruct.pRowCol ) & @CRLF )
    $tString = DllStructCreate( "char sString[" & $tStruct.iRowCol & "]", $tStruct.pRowCol )
    ConsoleWrite( "$tString.sString  = " &      $tString.sString   & @CRLF & @CRLF )

    ; Delete .NET structs
    $oRandStructsClass.RandStructsDelete()

    ; Free structure memory
    $oRandStructsClass.FreeStructure()
    Return
  EndIf

  ; Number of rows
  ConsoleWrite( "Struct rows    = " & $iRows & @CRLF )

  ; Create structs
  Local $hTimer = TimerInit()
  $oRandStructsClass.RandStructsCreate( $iRows, $aColumns, $aColTypes, $sSource )
  ConsoleWrite( "Create structs = " & TimerDiff( $hTimer ) & @CRLF )

  ; Copy structs to AutoIt
  $hTimer = TimerInit()
  Local $pPtr = $oRandStructsClass.RandStructsToAutoIt( $iRows )
  ConsoleWrite( "Copy to AutoIt = " & TimerDiff( $hTimer ) & @CRLF )

  ; Delete .NET structs
  $oRandStructsClass.RandStructsDelete()

  ; Copy structs to .NET
  $hTimer = TimerInit()
  $oRandStructsClass.RandStructsToDotNet( $iRows )
  ConsoleWrite( "Copy to .NET   = " & TimerDiff( $hTimer ) & @CRLF & @CRLF )

  ; Delete .NET structs
  $oRandStructsClass.RandStructsDelete()

  ; Free structure memory
  $oRandStructsClass.FreeStructure()

  #forceref $pPtr
EndFunc

#cs
Struct rows    = 50000
Create structs = 118.687319937022
Copy to AutoIt = 14.4374527116075
Copy to .NET   = 27.3954765492515

Struct rows    = 100000
Create structs = 270.28500615459
Copy to AutoIt = 30.5258261460518
Copy to .NET   = 71.3821661221316

Struct rows    = 250000
Create structs = 654.394887271121
Copy to AutoIt = 70.4205373540553
Copy to .NET   = 144.230739927763

Struct rows    = 750000
Create structs = 1731.17973924928
Copy to AutoIt = 203.711537966051
Copy to .NET   = 437.325547215529

Struct rows    = 2000000
Create structs = 4556.50490192412
Copy to AutoIt = 547.36291803216
Copy to .NET   = 1646.96449730863
#ce

 

Imports System
Imports System.Runtime.InteropServices

Public Structure Test
  Public iString  As Integer 'String length
  Public sString  As String
  Public iInteger As Integer
  Public dDouble  As Double
  Public iDate    As Integer
  Public iTime    As Integer
  Public iRowCol  As Integer 'String length
  Public sRowCol  As String
End Structure

Class RandStructsClass
  'Globals in class
  Dim tTest As Test, iSize As Integer = Marshal.SizeOf( tTest )
  Dim aTest() As Test
  Dim pnt As IntPtr

  Public Sub RandStructsCreate( iRows As Integer, aColumns As Object(,), aColTypes As Object(,), sSource As String )
    Dim iColumns As Integer = aColumns.GetUpperBound(1) + 1
    Dim aSource As Char() = sSource.ToCharArray()
    Dim iSourceLen As Integer = sSource.Length
    Dim oRnd As New Random()
    ReDim aTest(iRows-1)
    Dim tTest As Test

    Dim sStr As String = ""
    Dim k, y, m, dMax As Integer

    For i As Integer = 0 To iRows - 1
      'String columns
      For j As Integer = 1 To aColTypes(0,0)
        k = aColTypes(j,0)
        If i <= aColumns(3,k) Then
          For l As Integer = 1 To oRnd.Next( aColumns(1,k), aColumns(2,k) )
            sStr &= aSource(oRnd.Next( iSourceLen ))
          Next
          tTest.iString = sStr.Length + 1
          tTest.sString = sStr
          sStr = ""
        Else
          aTest(i) = aTest(oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Integer columns
      For j As Integer = 1 To aColTypes(0,1)
        k = aColTypes(j,1)
        If i <= aColumns(3,k) Then
          tTest.iInteger = oRnd.Next( aColumns(1,k), aColumns(2,k) )
        Else
          aTest(i) = aTest(oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Float columns
      For j As Integer = 1 To aColTypes(0,2)
        k = aColTypes(j,2)
        If i <= aColumns(3,k) Then
          tTest.dDouble = aColumns(1,k) + ( aColumns(2,k) - aColumns(1,k) ) * oRnd.NextDouble()
        Else
          aTest(i) = aTest(oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Date columns
      For j As Integer = 1 To aColTypes(0,3)
        k = aColTypes(j,3)
        If i <= aColumns(3,k) Then
          y = oRnd.Next( aColumns(1,k), aColumns(2,k) )
          m = oRnd.Next( 1, 13 )
          Select m
            Case 1, 3, 5, 7, 8, 10, 12
              dMax = 31
            Case 4, 6, 9, 11
              dMax = 30
            Case Else '2
              dMax = If( y Mod 4 <> 0, 28, If( y Mod 400 = 0, 29, If( y Mod 100 = 0, 28, 29 ) ) )
          End Select
          tTest.iDate = y * 10000 + m * 100 + oRnd.Next( 1, dMax + 1 )
        Else
          aTest(i) = aTest(oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Time columns
      For j As Integer = 1 To aColTypes(0,4)
        k = aColTypes(j,4)
        If i <= aColumns(3,k) Then
          tTest.iTime = oRnd.Next( aColumns(1,k), aColumns(2,k) ) * 10000 + oRnd.Next( 0, 60 ) * 100 + oRnd.Next( 0, 60 )
        Else
          aTest(i) = aTest(oRnd.Next(0,aColumns(3,k)+1))
        End If
      Next

      'Row/col columns
      For j As Integer = 1 To aColTypes(0,5)
        k = aColTypes(j,5)
        tTest.sRowCol = i & "/" & k
        tTest.iRowCol = tTest.sRowCol.Length + 1
      Next

      aTest(i) = tTest
    Next
  End Sub

  Public Function RandStructsToAutoIt( iRows As Integer ) As String
    pnt = Marshal.AllocHGlobal( iRows * iSize )
    'Console.WriteLine( "Marshal.SizeOf( tTest ) = {0}", iSize )

    'Copy structures to unmanaged memory
    For i As Integer = 0 To iRows - 1
      Marshal.StructureToPtr( aTest(i), pnt + i * iSize, False )
    Next

    Return pnt.ToString()
  End Function

  Public Sub RandStructsToDotNet( iRows As Integer )
    ReDim aTest(iRows-1)
    For i As Integer = 0 To iRows - 1
      aTest(i) = CType( Marshal.PtrToStructure( pnt + i * iSize, GetType( Test ) ), Test )
    Next
  End Sub

  Public Sub RandStructsDelete()
    ReDim aTest(0)
  End Sub

  Public Sub FreeStructure()
    Marshal.FreeHGlobal( pnt )
  End Sub
End Class


Performance

Example02                              Example03
---------------------------------      ---------------------------------
Array rows     = 50000                 Struct rows    = 50000
Create array   = 161.3358744048        Create structs = 118.687319937022
Pass to AutoIt = 53.1796500402687      Copy to AutoIt = 14.4374527116075
Pass to .NET   = 128.998994597868      Copy to .NET   = 27.3954765492515

Array rows     = 100000                Struct rows    = 100000
Create array   = 276.623278397762      Create structs = 270.28500615459
Pass to AutoIt = 108.432993923537      Copy to AutoIt = 30.5258261460518
Pass to .NET   = 258.063372194661      Copy to .NET   = 71.3821661221316

Array rows     = 250000                Struct rows    = 250000
Create array   = 898.781576733183      Create structs = 654.394887271121
Pass to AutoIt = 264.890631696736      Copy to AutoIt = 70.4205373540553
Pass to .NET   = 631.123803539603      Copy to .NET   = 144.230739927763

Array rows     = 750000                Struct rows    = 750000
Create array   = 2717.34763337093      Create structs = 1731.17973924928
Pass to AutoIt = 837.568960363219      Copy to AutoIt = 203.711537966051
Pass to .NET   = 2094.36373615079      Copy to .NET   = 437.325547215529

Array rows     = 2000000               Struct rows    = 2000000
Create array   = 7424.69637769875      Create structs = 4556.50490192412
Pass to AutoIt = 2081.94096635518      Copy to AutoIt = 547.36291803216
Pass to .NET   = 5806.68962274841      Copy to .NET   = 1646.96449730863

Data creation rates are higher for structures than for arrays. It takes about 30 - 100 percent longer to create arrays than structures. This is probably due to the fact that the 2d arrays are arrays of objects, where the more computationally heavy objects are needed to handle data of different types (strings, integers and doubles). While the 1d array of structures consists only of one data type (structures). The code to create random data is the same for both arrays and structures. Therefore, it can almost only be the difference between arrays and structures that results in a higher speed for structures. 

Data transfer rates are significantly higher for structures than for arrays. The speeds are about 3-4 times higher for structures. And this applies to transfers in both directions from .NET to AutoIt and from AutoIt to .NET. Transfers from .NET to AutoIt are 2-3 times faster than the other direction.

With the Marshal Class used in this thread, it's apparently not possible to transfer a 1d array of structures as a single data block. But it cannot be ruled out that it's possible through custom marshalling. And that, of course, would be very interesting.


7z-file
New 7z-file at bottom of first post.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...