Jump to content

Learning WinRt from AutoIt goal to read OCR (and understand the logic)

junkew

By junkew
in AutoIt General Help and Support

 Share

Recommended Posts

junkew Universalist

junkew

Posted
Posted (edited)

Any feedback on the code is appreciated.

  • It compiles and runs so from that perspective
  • no issues but still complex to understand when to use vtable or just hack by knowing the function numbers.
  • no tesseract or other OCR libraries needed, just plain AutoIt and Windows libraries (on windows 11)


So once in a while I still use AutoIt

and now found out with Powershell 5.1 the WinRt

on how to do stuff with the nice OCR gettext (thats also in Win11 Capture tool) and was wondering how to get this working in AutoIt

Jumping back and forward in threads of this forum on AutoItObject, ObjCreateInterface, Com/Com++, Embedding .NET in AutoIt, CuiAutomation I already understand WinRt is another animal to get into AutoIt.

I didnt want to hijack: 


Learning step 1

  • Understanding combase.dll and RoGetActivationFactory
  • Made a very nice script that goes from
    • Learning WinRt
    • To understand the WinRt OCR
    • To highlighting aspects of the rectangles found
  • So much easier with AI then when I wrote the IUIAutomation library (took me months, now only several hours to understand the WinRt base as extension on traditional COM)
      
; =====================================================================================
; WINRT OCR SCRIPT - THE BULLETPROOF HYBRID METHOD
; =====================================================================================
#include "ScreenCapture.au3"

#region --- GLOBALS & DLLS ---
Global $hDLLComBase = DllOpen("combase.dll")
Global $hDLLOle32 = DllOpen("ole32.dll")
Global $hDLLShcore = DllOpen("shcore.dll")
#endregion

#region --- GUIDS & BLUEPRINTS ---
Global Const $sIInspectable = "{AF86E2E0-B12D-4C6A-9C5A-D7AA65101E90}"
Global Const $sIAsyncInfoGUID = "{00000036-0000-0000-C000-000000000046}"
Global Const $sIRandomAccessStreamGUID = "{905A0FE1-BC53-11DF-8C49-001E4FC686DA}"
Global Const $sIBitmapDecoderStaticsGUID = "{438CCB26-BCEF-4E95-BAD6-23A822E58D01}"
Global Const $sIBitmapFrameWithSoftwareBitmapGUID = "{FE287C9A-420C-4963-87AD-691436E08383}"
Global Const $sIOcrEngineStaticsGUID = "{5BFFA85A-3384-3540-9940-699120D428A8}"
Global Const $sIOcrEngineGUID = "{5A14BC41-5B76-3140-B680-8825562683AC}"

Global $tagIInspectable = _
    "GetIids hresult(ulong*;ptr*);" & _
    "GetRuntimeClassName hresult(ptr*);" & _
    "GetTrustLevel hresult(int*);"

Global $tagIAsyncInfo = $tagIInspectable & _
    "get_Id hresult(uint*);" & _
    "get_Status hresult(int*);" & _
    "get_ErrorCode hresult(long*);" & _
    "Cancel hresult();" & _
    "Close hresult();"

Global $tagIBitmapDecoderStatics = $tagIInspectable & _
    "get_BmpDecoderId hresult(ptr*);" & _
    "get_JpegDecoderId hresult(ptr*);" & _
    "get_PngDecoderId hresult(ptr*);" & _
    "get_TiffDecoderId hresult(ptr*);" & _
    "get_GifDecoderId hresult(ptr*);" & _
    "get_JpegXRDecoderId hresult(ptr*);" & _
    "get_IcoDecoderId hresult(ptr*);" & _
    "GetDecoderInformationEnumerator hresult(ptr*);" & _
    "CreateAsync hresult(ptr;ptr*);"

Global $tagIBitmapFrameWithSoftwareBitmap = $tagIInspectable & _
    "GetSoftwareBitmapAsync hresult(ptr*);"

Global $tagIOcrEngineStatics = $tagIInspectable & _
    "get_MaxImageDimension hresult(uint*);" & _
    "get_AvailableRecognizerLanguages hresult(ptr*);" & _
    "IsLanguageSupported hresult(ptr;bool*);" & _
    "TryCreateFromLanguage hresult(ptr;ptr*);" & _
    "TryCreateFromUserProfileLanguages hresult(ptr*);"

Global $tagIOcrEngine = $tagIInspectable & _
    "RecognizeAsync hresult(ptr;ptr*);" & _
    "get_RecognizerLanguage hresult(ptr*);"
#endregion

; --- OCR RESULT BLUEPRINTS ---
Global $tagIOcrResult = $tagIInspectable & _
    "get_Lines hresult(ptr*);" & _
    "get_TextAngle hresult(ptr*);" & _
    "get_Text hresult(ptr*);"

Global $tagIVectorView = $tagIInspectable & _
    "GetAt hresult(uint;ptr*);" & _
    "get_Size hresult(uint*);"

Global $tagIOcrLine = $tagIInspectable & _
    "get_Words hresult(ptr*);" & _
    "get_Text hresult(ptr*);"

Global $tagIOcrWord = $tagIInspectable & _
    "get_BoundingRect hresult(ptr);" & _
    "get_Text hresult(ptr*);"

#region --- STEP 1 & 2: INIT & CAPTURE ---
ConsoleWrite(">>> [Step 1] Initializing WinRT..." & @CRLF)
DllCall($hDLLComBase, "long", "RoInitialize", "int", 1)

Global $sTempImage = @TempDir & "\ocr_snapshot.png"
ConsoleWrite(">>> [Step 2] Capturing screen to: " & $sTempImage & @CRLF)
_ScreenCapture_Capture($sTempImage, 0, 0, 500, 500)
Sleep(300) ; Geef de hardeschijf genoeg tijd om de foto te flushen
#endregion

#region --- STEP 3: CREATE WINRT STREAM ---
ConsoleWrite(">>> [Step 3] Opening file as a WinRT Stream..." & @CRLF)
Global $pStream = 0
Global $tIID_Stream = _WinRT_CreateGUID($sIRandomAccessStreamGUID)
Global $aResStream = DllCall($hDLLShcore, "long", "CreateRandomAccessStreamOnFile", "wstr", $sTempImage, "dword", 0, "struct*", $tIID_Stream, "ptr*", 0)

If Not @error And $aResStream[0] = 0 Then
    $pStream = $aResStream[4]
    ConsoleWrite("    + Success! Stream pointer: " & $pStream & @CRLF)
Else
    ConsoleWrite("    ! Error: Failed to create Stream." & @CRLF)
    Exit
EndIf
#endregion

#region --- STEP 4: BITMAP DECODER ---
ConsoleWrite(">>> [Step 4] Creating BitmapDecoder..." & @CRLF)
Global $pDecoderStatics = _WinRT_RoGetActivationFactory("Windows.Graphics.Imaging.BitmapDecoder", $sIBitmapDecoderStaticsGUID)
Global $oDecoderStatics = ObjCreateInterface($pDecoderStatics, $sIBitmapDecoderStaticsGUID, $tagIBitmapDecoderStatics, True)

Global $pTicket1 = 0
$oDecoderStatics.CreateAsync($pStream, $pTicket1)

; De Helper functie verzorgt het wachten en de VTable hack voor GetResults!
Global $pBitmapDecoder = _WinRT_WaitAsync($pTicket1, "Decoder")

If $pBitmapDecoder Then
    ConsoleWrite("    + Success! Decoder retrieved: " & $pBitmapDecoder & @CRLF)
Else
    Exit
EndIf
#endregion

#region --- STEP 5: SOFTWARE BITMAP ---
ConsoleWrite(">>> [Step 5] Converting Decoder to SoftwareBitmap..." & @CRLF)
; We dwingen WinRT om de Decoder pointer te voorzien van de SoftwareBitmap lens
Global $oBitmapFrame = ObjCreateInterface($pBitmapDecoder, $sIBitmapFrameWithSoftwareBitmapGUID, $tagIBitmapFrameWithSoftwareBitmap, True)

Global $pTicket2 = 0
$oBitmapFrame.GetSoftwareBitmapAsync($pTicket2)

Global $pSoftwareBitmap = _WinRT_WaitAsync($pTicket2, "SoftwareBitmap")

If $pSoftwareBitmap Then
    ConsoleWrite("    + Success! SoftwareBitmap retrieved: " & $pSoftwareBitmap & @CRLF)
Else
    Exit
EndIf
#endregion

#region --- STEP 6: OCR ENGINE RECOGNIZE ---
ConsoleWrite(">>> [Step 6] Feeding image to OCR Engine..." & @CRLF)
Global $pOcrStatics = _WinRT_RoGetActivationFactory("Windows.Media.Ocr.OcrEngine", $sIOcrEngineStaticsGUID)
Global $oOcrStatics = ObjCreateInterface($pOcrStatics, $sIOcrEngineStaticsGUID, $tagIOcrEngineStatics, True)

Global $pOcrEnginePtr = 0
$oOcrStatics.TryCreateFromUserProfileLanguages($pOcrEnginePtr)
Global $oOcrEngine = ObjCreateInterface($pOcrEnginePtr, $sIOcrEngineGUID, $tagIOcrEngine, True)

ConsoleWrite("    -> Calling RecognizeAsync..." & @CRLF)
Global $pTicket3 = 0
$oOcrEngine.RecognizeAsync($pSoftwareBitmap, $pTicket3)

Global $pOcrResult = _WinRT_WaitAsync($pTicket3, "OCR_Engine")

If $pOcrResult Then
    ConsoleWrite("    + ===========================================================" & @CRLF)
    ConsoleWrite("    + MEGA SUCCESS! OCR Result object retrieved: " & $pOcrResult & @CRLF)
    ConsoleWrite("    + THE IMAGE HAS BEEN SUCCESSFULLY READ BY WINDOWS!" & @CRLF)
    ConsoleWrite("    + ===========================================================" & @CRLF)
Else
    ConsoleWrite("    ! Error: OCR Recognition failed." & @CRLF)
EndIf
#endregion

#region --- STEP 7: EXTRACT TEXT, POSITIONS & VISUAL HUD DEMO ---
ConsoleWrite(">>> [Step 7] Extracting OCR Text and Highlighting Bounding Boxes..." & @CRLF)

Global $pVTable_Res = DllStructGetData(DllStructCreate("ptr", $pOcrResult), 1)

; 1. Complete tekst uitlezen
Global $pGetText_Res = DllStructGetData(DllStructCreate("ptr", $pVTable_Res + (8 * (@AutoItX64 ? 8 : 4))), 1)
Global $aCallText = DllCallAddress("long", $pGetText_Res, "ptr", $pOcrResult, "ptr*", 0)

If Not @error And $aCallText[0] = 0 Then
    Global $hFullText = $aCallText[2]
    ConsoleWrite("    [VOLLEDIGE TEKST] ===================================" & @CRLF)
    ConsoleWrite("    " & _WinRT_GetHStringText($hFullText) & @CRLF)
    ConsoleWrite("    =====================================================" & @CRLF)
    _WinRT_DeleteHString($hFullText)
EndIf

; 2. Lijnen en Woorden ophalen en live highlighten op het scherm
Global $pLinesVector = 0
Global $pGetLines = DllStructGetData(DllStructCreate("ptr", $pVTable_Res + (6 * (@AutoItX64 ? 8 : 4))), 1)
Global $aCallLines = DllCallAddress("long", $pGetLines, "ptr", $pOcrResult, "ptr*", 0)

If Not @error And $aCallLines[0] = 0 Then $pLinesVector = $aCallLines[2]

If $pLinesVector Then
    Global $pVTable_Lines = DllStructGetData(DllStructCreate("ptr", $pLinesVector), 1)
    Global $pGetAt_Lines = DllStructGetData(DllStructCreate("ptr", $pVTable_Lines + (6 * (@AutoItX64 ? 8 : 4))), 1)
    Global $pGetSize_Lines = DllStructGetData(DllStructCreate("ptr", $pVTable_Lines + (7 * (@AutoItX64 ? 8 : 4))), 1)

    Global $iLineCount = 0
    Global $aCallSize = DllCallAddress("long", $pGetSize_Lines, "ptr", $pLinesVector, "uint*", 0)
    If Not @error And $aCallSize[0] = 0 Then $iLineCount = $aCallSize[2]

    ConsoleWrite("    -> Aantal tekstregels gevonden: " & $iLineCount & @CRLF)

    For $i = 0 To $iLineCount - 1
        Global $pLine = 0
        Global $aCallAt = DllCallAddress("long", $pGetAt_Lines, "ptr", $pLinesVector, "uint", $i, "ptr*", 0)
        If Not @error And $aCallAt[0] = 0 Then $pLine = $aCallAt[3]

        If $pLine Then
            Global $pVTable_Line = DllStructGetData(DllStructCreate("ptr", $pLine), 1)
            Global $pGetWords = DllStructGetData(DllStructCreate("ptr", $pVTable_Line + (6 * (@AutoItX64 ? 8 : 4))), 1)

            Global $pWordsVector = 0
            Global $aCallWords = DllCallAddress("long", $pGetWords, "ptr", $pLine, "ptr*", 0)
            If Not @error And $aCallWords[0] = 0 Then $pWordsVector = $aCallWords[2]

            If $pWordsVector Then
                Global $pVTable_Words = DllStructGetData(DllStructCreate("ptr", $pWordsVector), 1)
                Global $pGetAt_Words = DllStructGetData(DllStructCreate("ptr", $pVTable_Words + (6 * (@AutoItX64 ? 8 : 4))), 1)
                Global $pGetSize_Words = DllStructGetData(DllStructCreate("ptr", $pVTable_Words + (7 * (@AutoItX64 ? 8 : 4))), 1)

                Global $iWordCount = 0
                Global $aCallWSize = DllCallAddress("long", $pGetSize_Words, "ptr", $pWordsVector, "uint*", 0)
                If Not @error And $aCallWSize[0] = 0 Then $iWordCount = $aCallWSize[2]

                For $j = 0 To $iWordCount - 1
                    Global $pWord = 0
                    Global $aCallWAt = DllCallAddress("long", $pGetAt_Words, "ptr", $pWordsVector, "uint", $j, "ptr*", 0)
                    If Not @error And $aCallWAt[0] = 0 Then $pWord = $aCallWAt[3]

                    If $pWord Then
                        Global $pVTable_Word = DllStructGetData(DllStructCreate("ptr", $pWord), 1)
                        Global $pGetRect = DllStructGetData(DllStructCreate("ptr", $pVTable_Word + (6 * (@AutoItX64 ? 8 : 4))), 1)
                        Global $pGetText_Word = DllStructGetData(DllStructCreate("ptr", $pVTable_Word + (7 * (@AutoItX64 ? 8 : 4))), 1)

                        ; Tekst ophalen
                        Global $hWordText = 0
                        Global $aCallWText = DllCallAddress("long", $pGetText_Word, "ptr", $pWord, "ptr*", 0)
                        If Not @error And $aCallWText[0] = 0 Then $hWordText = $aCallWText[2]

                        Global $sWordText = _WinRT_GetHStringText($hWordText)
                        _WinRT_DeleteHString($hWordText)

                        ; Coördinaten ophalen (Rect)
                        Global $tRect = DllStructCreate("float X; float Y; float Width; float Height")
                        DllCallAddress("long", $pGetRect, "ptr", $pWord, "ptr", DllStructGetPtr($tRect))

                        Local $fX = DllStructGetData($tRect, "X")
                        Local $fY = DllStructGetData($tRect, "Y")
                        Local $fW = DllStructGetData($tRect, "Width")
                        Local $fH = DllStructGetData($tRect, "Height")

                        ; --- DE DEMO MAGIE: RECHTHOEK TEKENEN OP HET SCHERM ---
                        ; Parameters: Left, Right, Top, Bottom (volgens jouw functie-definitie)
                        _UIA_DrawRect(Int($fX), Int($fX + $fW), Int($fY), Int($fY + $fH), 0x0000FF, 2)

                        ; Kleine pauze zodat je de scanner live over het scherm ziet trekken (optioneel, zet op 0 voor instant)
                        Sleep(15)

                        ConsoleWrite("       Regel " & $i & " | Woord: '" & $sWordText & "' | Box: X=" & $fX & ", Y=" & $fY & ", W=" & $fW & ", H=" & $fH & @CRLF)
                    EndIf
                Next
            EndIf
        EndIf
    Next
EndIf
#endregion

#region --- CLEANUP ---
FileDelete($sTempImage)
DllCall($hDLLComBase, "none", "RoUninitialize")
DllClose($hDLLComBase)
DllClose($hDLLOle32)
DllClose($hDLLShcore)
#endregion

#region --- HELPER FUNCTIONS ---
; De ultieme hybride Async Helper
Func _WinRT_WaitAsync($pAsyncTicket, $sName = "Async")
    If $pAsyncTicket = 0 Then Return 0

    ; 1. SAFE POLLING: We dwingen QueryInterface af via ObjCreateInterface
    Local $oAsyncInfo = ObjCreateInterface($pAsyncTicket, $sIAsyncInfoGUID, $tagIAsyncInfo, True)
    If Not IsObj($oAsyncInfo) Then Return 0

    Local $iStatus = 0
    While True
        $oAsyncInfo.get_Status($iStatus)
        If $iStatus >= 1 Then ExitLoop
        Sleep(10)
    WEnd

    If $iStatus <> 1 Then
        ConsoleWrite("       ! [" & $sName & "] Failed with status: " & $iStatus & @CRLF)
        Return 0
    EndIf

    ; 2. RAW VTABLE HACK VOOR RESULTS: We weten dat GetResults altijd Index 8 is!
    Local $pVTable = DllStructGetData(DllStructCreate("ptr", $pAsyncTicket), 1)
    Local $pGetResultsFunc = DllStructGetData(DllStructCreate("ptr", $pVTable + (8 * (@AutoItX64 ? 8 : 4))), 1)

    Local $aCall = DllCallAddress("long", $pGetResultsFunc, "ptr", $pAsyncTicket, "ptr*", 0)

    If @error Or $aCall[0] <> 0 Then Return 0
    Return $aCall[2] ; Het finale pointer resultaat!
EndFunc

Func _WinRT_RoGetActivationFactory($sClassID, $sIID)
    Local $hsClassID = _WinRT_CreateHString($sClassID)
    Local $tIID = _WinRT_CreateGUID($sIID)
    Local $aRes = DllCall($hDLLComBase, "long", "RoGetActivationFactory", "ptr", $hsClassID, "ptr", DllStructGetPtr($tIID), "ptr*", 0)
    _WinRT_DeleteHString($hsClassID)
    If @error Or $aRes[0] < 0 Then Return 0
    Return $aRes[3]
EndFunc

Func _WinRT_CreateHString($sString)
    Local $aRes = DllCall($hDLLComBase, "long", "WindowsCreateString", "wstr", $sString, "uint", StringLen($sString), "ptr*", 0)
    If @error Or $aRes[0] < 0 Then Return 0
    Return $aRes[3]
EndFunc

Func _WinRT_DeleteHString(ByRef $hString)
    If $hString = 0 Then Return
    DllCall($hDLLComBase, "long", "WindowsDeleteString", "ptr", $hString)
    $hString = 0
EndFunc
Func _WinRT_GetHStringText($hString)
    If $hString = 0 Then Return ""
    Local $aRes = DllCall($hDLLComBase, "ptr", "WindowsGetStringRawBuffer", "ptr", $hString, "uint*", 0)
    If @error Or $aRes[0] = 0 Then Return ""
    Local $iLength = $aRes[2]
    If $iLength = 0 Then Return ""
    Local $tString = DllStructCreate("wchar[" & ($iLength + 1) & "]", $aRes[0])
    Return DllStructGetData($tString, 1)
EndFunc

Func _WinRT_CreateGUID($sGUID)
    Local $tGUID = DllStructCreate("dword Data1;word Data2;word Data3;byte Data4[8]")
    DllCall($hDLLOle32, "long", "CLSIDFromString", "wstr", $sGUID, "struct*", $tGUID)
    Return $tGUID
EndFunc

; Draw rectangle on screen.
Func _UIA_DrawRect($tLeft, $tRight, $tTop, $tBottom, $color = 0xFF, $PenWidth = 4)
    Local $hDC, $hPen, $obj_orig, $x1, $x2, $y1, $y2
    $x1 = $tLeft
    $x2 = $tRight
    $y1 = $tTop
    $y2 = $tBottom
    $hDC = _WinAPI_GetWindowDC(0) ; DC of entire screen (desktop)
    $hPen = _WinAPI_CreatePen(0, $PenWidth, $color) ; $PS_SOLID = 0
    $obj_orig = _WinAPI_SelectObject($hDC, $hPen)

    _WinAPI_DrawLine($hDC, $x1, $y1, $x2, $y1) ; horizontal to right
    _WinAPI_DrawLine($hDC, $x2, $y1, $x2, $y2) ; vertical down on right
    _WinAPI_DrawLine($hDC, $x2, $y2, $x1, $y2) ; horizontal to left right
    _WinAPI_DrawLine($hDC, $x1, $y2, $x1, $y1) ; vertical up on left

    ; clear resources
    _WinAPI_SelectObject($hDC, $obj_orig)
    _WinAPI_DeleteObject($hPen)
    _WinAPI_ReleaseDC(0, $hDC)
EndFunc   ;==>_UIA_DrawRect
#endregion

 

Edited by junkew

FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets

MattyD Universalist

MattyD

Posted
Posted (edited)

well done :),

Quick tip - There are a few exceptions, but generally you should release any object that you "aquire".

You won't need to worry about any $pObject object "converted" to $oObject by ObjCreateInterface though. 
ObjCreateInterface does a QI/Release call on the ptr, so the refcount doesn't change at that point.  But an $oObject.Release() is automatically called when $oObject goes out of scope. So  that cancels out the original $pObject reference. 

But the collections etc. that you're passing directly to DllCallAddress() should probably be released at some point!

Edited by MattyD

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...