Microsoft Edge - WebView2, embed web code in your native application

[Gianni]

Hi @UEZ
your _WinAPI_SHCreateStreamOnFileEx function is awesome!
I just put that function in my draft script and changed the parameters of the $dtag_ICoreWebView2 variable a bit and the CapturePreview method started working! Thanks so much for that 🙂
However it seems that the generated screenshot has some problems.
It seems that the generated screenshot doesn't capture the whole page but only a part;
also maybe you need somehow to close the stream after taking the screenshot? it appears that the file is only "flushed to disk" when the script is closed.
However this seems to be the right way even if still to be improved....

I've updated the two draft scripts (which are now starting to work) in my previous post

See You UEZ and thanks again
PS
If anyone has any improvements or new features to add to this material, they are welcome

Edited August 27, 2023 by Gianni

[UEZ]

When I add _WinAPI_ReleaseStream($hStream) then it seems to work properly:

...
            Case $idWebScreenShot ; CapturePreview
                ; create a stream on a file [By the great UEZ! :) ]
                FileDelete($sScrrenShot_Path)
                $hStream = _WinAPI_SHCreateStreamOnFileEx($sScrrenShot_Path, BitOR($STGM_READWRITE, $STGM_CREATE), $FILE_ATTRIBUTE_NORMAL, True)

                ConsoleWrite("-Debug stream --: " & @error & @TAB & VarGetType($hStream) & @TAB & $hStream & @TAB & VarGetType($oCoreWebView2) & @CRLF)
                ; load a web page
                ; $oCoreWebView2.Navigate("https://www.autoitscript.com")
                ; MsgBox(0, '', 'Pause to give time to load page')
                ConsoleWrite("------------------" & @CRLF)

                $oCoreWebView2.CapturePreview($COREWEBVIEW2_CAPTURE_PREVIEW_IMAGE_FORMAT_PNG, $hStream, $pICoreWebView2CapturePreviewCompletedHandler)
                ; above call Forces in turn CoreWebView2CapturePreviewCompletedHandler_Invoke() callback function below to be executed after screenshot is taken
                ; MsgBox(0, '', 'Pause post screenshot')
                _WinAPI_ReleaseStream($hStream)

            Case $GUI_EVENT_CLOSE
...

Any idea how to get the image as a handle in the memory instead of writing it to a file?

Here the ChatGPT suggestion:

// Define a global or class-level buffer to hold the screenshot data
std::vector<BYTE> screenshotBuffer;

// Show the user a dialog and capture a screenshot of the WebView to memory
void FileComponent::CaptureScreenshotToMemory()
{
    // Clear the previous screenshot buffer
    screenshotBuffer.clear();

    CHECK_FAILURE(m_webView->CapturePreview(
        COREWEBVIEW2_CAPTURE_PREVIEW_IMAGE_FORMAT_PNG,
        nullptr,
        Callback<ICoreWebView2CapturePreviewCompletedHandler>(
            [this](HRESULT error_code, IStream* preview_stream) -> HRESULT {
                CHECK_FAILURE(error_code);

                // Read the screenshot data from the stream into the memory buffer
                STATSTG stat;
                ULONG bytesRead = 0;
                preview_stream->Stat(&stat, STATFLAG_NONAME);
                screenshotBuffer.resize(stat.cbSize.LowPart);
                preview_stream->Read(&screenshotBuffer[0], stat.cbSize.LowPart, &bytesRead);

                // Handle the captured screenshot in memory (e.g., display or process it)
                // ...

                return S_OK;
            })
            .Get()));
}

I don't know how to init preview_stream as IStream. Maybe solution can be found here:

 

Btw, I made some small changes to the  _WinAPI_SHCreateStreamOnFileEx function.

Edited August 28, 2023 by UEZ

[Gianni]

double post deleted

Edited August 29, 2023 by Gianni

[Gianni]

  On 8/28/2023 at 2:52 PM, UEZ said:

Any idea how to get the image as a handle in the memory instead of writing it to a file?

Expand  

_WinAPI_CreateStreamOnHGlobal() seems to work.
However, it is important to wait for the screenshot to be completed before reading the stream (I inserted a loop that waits for $GlobalFlag to be True. It is set to True by the callback function when the screenshot is completed). Below snippet shows the captured screenshot in a temporary GUI.....

Case $idWebScreenShot ; CapturePreview
    Local $hStream = _WinAPI_CreateStreamOnHGlobal()
    $GlobalFlag = False
    $oCoreWebView2.CapturePreview($COREWEBVIEW2_CAPTURE_PREVIEW_IMAGE_FORMAT_PNG, $hStream, $pICoreWebView2CapturePreviewCompletedHandler)
    ; above call Forces in turn CoreWebView2CapturePreviewCompletedHandler_Invoke() callback function below to be executed after screenshot is taken
                
    Do ; wait until screenshot is completw
        Sleep(200)
    Until $GlobalFlag ; is set to True in the CoreWebView2CapturePreviewCompletedHandler_Invoke() below callback function
                
    Local $hBitmapFromStream = _GDIPlus_BitmapCreateFromStream($hStream) ;create bitmap from a stream
    WinAPI_ReleaseStream($hStream)
    Local $iX = _GDIPlus_ImageGetWidth($hBitmapFromStream)
    Local $iY = _GDIPlus_ImageGetHeight($hBitmapFromStream)
    Local $hTempGui = GUICreate('', $iX, $iY, 5, 5) ; to show the screenshot
    GUISetState(@SW_SHOW,$hTempGui)
    Local $hGraphics = _GDIPlus_GraphicsCreateFromHWND($hTempGui) ;create a graphics object from a window handle
    _GDIPlus_GraphicsDrawImage($hGraphics, $hBitmapFromStream, 0, 0) ;display streamed image
    MsgBox(0, '', 'pause')
    ;cleanup resources
    _GDIPlus_GraphicsDispose($hGraphics)
    _GDIPlus_BitmapDispose($hBitmapFromStream)
    GUIDelete($hTempGui)
Case $GUI_EVENT_CLOSE

 or something like that as suggested in this post ... 

Local $hStream = _WinAPI_CreateStreamOnHGlobal()
    Local $hMemory = _WinAPI_GetHGlobalFromStream($hStream)
    Local $iMemSize = _MemGlobalSize($hMemory)
    Local $pMem = _MemGlobalLock($hMemory)
    Local $tData = DllStructCreate("byte[" & $iMemSize & "]", $pMem)
    Local $bData = DllStructGetData($tData, 1)
    _WinAPI_ReleaseStream($hStream)
    _MemGlobalFree($hMemory)
    Return $bData

 

Edited August 29, 2023 by Gianni

[UEZ]

@Gianni I already tested all that stuff but it didn't work because _GDIPlus_BitmapCreateFromStream always returned 0 / error. 😉

If it is working for you then probably I made some mistakes. 

 

Edit:

"However, it is important to wait for the screenshot to be completed before reading the stream" - that was the problem why it didn't work - thx.

Edited August 29, 2023 by UEZ

[UEZ]

  On 8/29/2023 at 12:32 AM, Gianni said:

(I inserted a loop that waits for $GlobalFlag to be True. It is set to True by the callback function when the screenshot is completed).

Expand  

$GlobalFlag is always false and I couldn't find a definition of this variable in any of the files. I used Sleep(1000) for this example.

Where is it?

 

[Gianni]

Hi @UEZ

sorry for the mess of my snippets

I declared the Global variable $GlobalFlag at the beginning of one of my test drafts but I didn't post it, I only posted the core piece of code...

Global $GlobalFlag = False

Also, since the CapturePreview function seems to be asynchronous, I've set the $GlobalFlag setting inside the callback function for the CapturePreview so that it's only set to True when the capture is complete.

This functio is executed automatically only when CapturePreview has ended

Func CoreWebView2CapturePreviewCompletedHandler_Invoke($pSelf, $iErrorCode)   ; Ret: dword
    ConsoleWrite("CoreWebView2CapturePreviewCompletedHandler_Invoke()" & @CRLF)
    ConsoleWrite("CapturePreview returned Error code: " & $iErrorCode & @CRLF)
    $GlobalFlag = True
    Return 0     ; For AddRef/Release
    #forceref $pSelf, $iErrorCode
EndFunc   ;==>CoreWebView2CapturePreviewCompletedHandler_Invoke

in this way the "wait" loop is exceeded only when the $GlobalFlag becomes True in the CallBack function, i.e. when the screenshot capture is finished.

Sorry again for the mess... as soon as I can I try to group all the positive "discoveries" into a single script.

[UEZ]

Thx, got it working now.

[KaFu]

How did you do it? My _GDIPlus_BitmapCreateFromStream is still 0.

Case $idWebScreenShot ; CapturePreview

                $GlobalFlag = False

                Local $hStream = _WinAPI_CreateStreamOnHGlobal()

                $oCoreWebView2.CapturePreview($COREWEBVIEW2_CAPTURE_PREVIEW_IMAGE_FORMAT_PNG, $hStream, $pICoreWebView2CapturePreviewCompletedHandler)
                ; above call Forces in turn CoreWebView2CapturePreviewCompletedHandler_Invoke() callback function below to be executed after screenshot is taken

                Do ; wait until screenshot is completw
                    Sleep(10)
                Until $GlobalFlag ; is set to True in the CoreWebView2CapturePreviewCompletedHandler_Invoke() below callback function

                Local $hBitmapFromStream = _GDIPlus_BitmapCreateFromStream($hStream) ;create bitmap from a stream
                MsgBox(0, "", $hBitmapFromStream)

                _WinAPI_ReleaseStream($hStream)
                
                Local $iX = _GDIPlus_ImageGetWidth($hBitmapFromStream)
                Local $iY = _GDIPlus_ImageGetHeight($hBitmapFromStream)

                Local $hTempGui = GUICreate('', $iX, $iY, 5, 5) ; to show the screenshot
                GUISetState(@SW_SHOW, $hTempGui)

                Local $hGraphics = _GDIPlus_GraphicsCreateFromHWND($hTempGui) ;create a graphics object from a window handle
                _GDIPlus_GraphicsDrawImage($hGraphics, $hBitmapFromStream, 0, 0) ;display streamed image

                MsgBox(0, '', 'pause')
                ;cleanup resources
                _GDIPlus_GraphicsDispose($hGraphics)
                _GDIPlus_BitmapDispose($hBitmapFromStream)
                GUIDelete($hTempGui)

 

Func CoreWebView2CapturePreviewCompletedHandler_Invoke($pSelf, $iErrorCode)   ; Ret: dword
    #forceref $pSelf, $iErrorCode
    ConsoleWrite("CoreWebView2CapturePreviewCompletedHandler_Invoke()" & @CRLF)
    ConsoleWrite("CapturePreview returned Error code: " & $iErrorCode & @CRLF)
    $GlobalFlag = True
    Return 0     ; For AddRef/Release
EndFunc   ;==>CoreWebView2CapturePreviewCompletedHandler_Invoke

 

[UEZ]

Did you add?

_GDIPlus_Startup()
WebView2()
_GDIPlus_Shutdown()

 

[KaFu]

Thanks, _GDIPlus_Startup() was missing 🙄🙈.

[UEZ]

I'm just struggling with this to take a screenshot of a given website without showing the site in a gui. Is it somehow possible? Any suggestions?

 

Thx.

[KaFu]

Not directly, but I guess it might work if you create the process on a different desktop (_WinAPI_CreateDesktop) without switching to it.

[mLipok]

  On 9/6/2023 at 9:54 AM, KaFu said:

Not directly, but I guess it might work if you create the process on a different desktop (_WinAPI_CreateDesktop) without switching to it.

Expand  

I'm wondering if the second DESKTOP needs to be active to transmit signals, or it will be the same as with a minimized RDP session when AutoIt GUI based programs don't work.

[KaFu]

I use this technique in SSD to switch the Windows Sound dialog on the invisible secondary desktop with controlsend().

Here a dirty PoC, works fine for me.

_WinAPI_CreateDesktop_Wrapper.zipFetching info...

[gmmg]

Hello, is there an example to fill a webform with user and password?

Greetings,

gmmg

[UEZ]

  On 9/6/2023 at 10:48 AM, KaFu said:

I use this technique in SSD to switch the Windows Sound dialog on the invisible secondary desktop with controlsend().

Here a dirty PoC, works fine for me.

_WinAPI_CreateDesktop_Wrapper.zip 793.5 kB · 11 downloads

Expand  

Thanks KaFu, but I would like to take a screenshot of the entire site, not just the GUI size. Further, your code returns a hStream handle but hStream doesn't contain any bitmap data because it is not filled up by $oCoreWebView2.CapturePreview. Reason is that the website is not rendered. You must add some Sleep() to give some time for rendering.

[KaFu]

  On 9/6/2023 at 7:36 PM, UEZ said:

Thanks KaFu, but I would like to take a screenshot of the entire site, not just the GUI size. Further, your code returns a hStream handle but hStream doesn't contain any bitmap data because it is not filled up by $oCoreWebView2.CapturePreview. Reason is that the website is not rendered. You must add some Sleep() to give some time for rendering.

Expand  

That was not your question 😉. I've play around with the function last week and never could render more than the "visible" client size (even if the window is larger), seems the function is lazy.

There's a sleep(1000) in the script, maybe that's not enough on your system, for me it worked fine.

For a full-size preview of a website maybe go back to this "Shell.Explorer.2" solution?

 

[UEZ]

Your are right, my question was not precise enough but the web screenshot from Ward or any other IE solution doesn't work anymore when IE is not active anymore, for example on my Win11 notebook.

[KaFu]

When I remove the GUI style and use a GUI like this:

$hGui = GUICreate("WebView2 Sample", 100,100)

I can resize it to any arbitrary size after the GUISetState() like this

WinMove($hGui,"",0,0,1080,8000)

Performing this on a different desktop allows for saving the full website invisible.

Now a challenge is to determine the full size of the webpage in px. Maybe a function to check when a single color starts to repeat to fill all following lines, and then cut it off?

Edit:

A solution also might be to cycle through increasing screenshots until the scrollbar on the right disappears.

Sleep() time depends on the responsiveness of the webpage (obviously), for how long it takes to fully render.

Edit2:

Do you assume that "Shell.Explorer.2" will not work on Win11, or did you try it? I'm still on Win10, but search results indicate it still should work,

Edit3:

The sleep() call should be replace with an event listener for "NavigationCompleted"

Edited September 7, 2023 by KaFu