_GUICtrlRichEdit_SetCharBkColor performance help please

[orbs]

this little script demonstrate the issue: using  _GUICtrlRichEdit_SetCharBkColor()to "highlight" selection in RichEdit takes too long. that's over 5 seconds for ~4000 characters (and much longer when it comes to real-life figures). i even tried to adapt the said function to eliminate some validation checks - this is _GUICtrlRichEdit_SetCharBkColorEx() in the script - no effect.

this is a simple demo that just highlights every character in a random color - but that is very similar to the real application, which is highlight searched text. just run it and see for yourself. also, during the operation, the focus (obviously) keeps returning to the not-yet-shown GUI, so it's practically impossible to work with anything else...

using _GUICtrlRichEdit_PauseRedraw() and _GUICtrlRichEdit_ResumeRedraw() has very little effect. you can comment them out and move the GUISetState() upfront, so you can actually see the entire process in action. very nice, slow display.

how can i improve the performance of this operation? any hints are most welcome!

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7
#include <GuiRichEdit.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>

; build GUI
Global Const $nGUI_MaxW = 500, $nGUI_MaxH = 400
Global $nGUI_W = 720, $nGUI_H = 500, $nRichEditOffset = 10, $nToolbarOffset = 35
Global $hGui = GUICreate('_GUICtrlRichEdit_SetCharBkColor performance test', $nGUI_W, $nGUI_H, -1, -1, BitOR($WS_MINIMIZEBOX, $WS_MAXIMIZEBOX, $WS_SIZEBOX, $WS_SYSMENU))
GUISetBkColor(0xCCDDEE)

Global $hRichEdit = _GUICtrlRichEdit_Create($hGui, FileRead(@ScriptFullPath) & @LF, $nRichEditOffset, $nRichEditOffset + $nToolbarOffset, $nGUI_W - $nRichEditOffset * 2, $nGUI_H - $nRichEditOffset * 2 - $nToolbarOffset - 24, BitOR($ES_MULTILINE, $ES_READONLY, $WS_HSCROLL, $WS_VSCROLL))
_GUICtrlRichEdit_SetSel($hRichEdit, 0, -1, True)
_GUICtrlRichEdit_SetFont($hRichEdit, Default, 'Courier New')

GUICtrlCreateLabel('time before highlight: ' & @TAB & @HOUR & ':' & @MIN & ':' & @SEC & ',' & @MSEC, 15, 5, $nGUI_W - 30)
GUICtrlSetFont(-1, Default, Default, Default, 'Courier New')
GUICtrlSetResizing(-1, BitOR($GUI_DOCKLEFT, $GUI_DOCKTOP, $GUI_DOCKSIZE))

_GUICtrlRichEdit_PauseRedraw($hRichEdit)

For $i = 1 To StringLen(_GUICtrlRichEdit_GetText($hRichEdit))
    _GUICtrlRichEdit_SetSel($hRichEdit, $i, $i + 2)
    _GUICtrlRichEdit_SetCharBkColor($hRichEdit, Random(253545, 304050, 1))
Next

_GUICtrlRichEdit_ResumeRedraw($hRichEdit)

GUICtrlCreateLabel('time after highlight: ' & @TAB & @HOUR & ':' & @MIN & ':' & @SEC & ',' & @MSEC, 15, 25, $nGUI_W - 30)
GUICtrlSetFont(-1, Default, Default, Default, 'Courier New')
GUICtrlSetResizing(-1, BitOR($GUI_DOCKLEFT, $GUI_DOCKTOP, $GUI_DOCKSIZE))

GUISetState()
GUIRegisterMsg($WM_GETMINMAXINFO, "_WM_GETMINMAXINFO")
GUIRegisterMsg($WM_SIZE, "_WM_SIZE")

Global $iMsg = 0
While True
    $iMsg = GUIGetMsg()
    Switch $iMsg
        Case $GUI_EVENT_CLOSE
            _GUICtrlRichEdit_Destroy($hRichEdit)
            Exit
    EndSwitch
WEnd

#region resizing funcs
Func _WM_GETMINMAXINFO($hWnd, $iMsg, $wParam, $lParam)
    #forceref $hWnd, $iMsg, $wParam, $lParam
    Local $tagMaxinfo
    If $hWnd = $hGui Then
        $tagMaxinfo = DllStructCreate("int;int;int;int;int;int;int;int;int;int", $lParam)
        DllStructSetData($tagMaxinfo, 7, $nGUI_MaxW) ; min X
        DllStructSetData($tagMaxinfo, 8, $nGUI_MaxH) ; min Y
        Return 0
    EndIf
EndFunc   ;==>_WM_GETMINMAXINFO

Func _WM_SIZE($hWnd, $iMsg, $wParam, $lParam)
    #forceref $hWnd, $iMsg, $wParam, $lParam
    ; ref: http://www.autoitscript.com/forum/topic/123904-solved-auto-resizing-richedit-control/#entry860464
    If $hWnd = $hGui Then
        Local $iWidth = _WinAPI_LoWord($lParam) ; the entire GUI internal width
        Local $iHeight = _WinAPI_HiWord($lParam) ; the entire GUI internal height
        _WinAPI_MoveWindow($hRichEdit, $nRichEditOffset, $nRichEditOffset + $nToolbarOffset, $iWidth - $nRichEditOffset * 2, $iHeight - $nRichEditOffset * 2 - $nToolbarOffset)
    EndIf
EndFunc   ;==>_WM_SIZE
#endregion resizing funcs

Func _GUICtrlRichEdit_SetCharBkColorEx($hWnd, $iBkColor = Default) ; this was adapted from the UDF to skip some validation checks that are irrelvant for current purpose
    ;If Not _WinAPI_IsClassName($hWnd, $_GRE_sRTFClassName) Then Return SetError(101, 0, False)

    Local $tCharFormat = DllStructCreate($tagCHARFORMAT2)
    DllStructSetData($tCharFormat, 1, DllStructGetSize($tCharFormat))
    If IsKeyword($iBkColor) Then
        DllStructSetData($tCharFormat, 3, $CFE_AUTOBACKCOLOR)
        $iBkColor = 0
    Else
        If BitAND($iBkColor, 0xff000000) Then Return SetError(1022, 0, False)
    EndIf

    DllStructSetData($tCharFormat, 2, $CFM_BACKCOLOR)
    DllStructSetData($tCharFormat, 12, $iBkColor)
    ;Local $ai = _GUICtrlRichEdit_GetSel($hWnd)
    ;If $ai[0] = $ai[1] Then
        ;Return _SendMessage($hWnd, $EM_SETCHARFORMAT, $SCF_ALL, $tCharFormat, 0, "wparam", "struct*") <> 0
    ;Else
        Return _SendMessage($hWnd, $EM_SETCHARFORMAT, $SCF_SELECTION, $tCharFormat, 0, "wparam", "struct*") <> 0
    ;EndIf
EndFunc   ;==>_GUICtrlRichEdit_SetCharBkColorEx

[BrewManNH]

That script ran in less than a second for me, approximately 900 ms. Windows 7 x64.Running it as a 32 bit process takes about 1.2 seconds.

[mLipok]

Win7 Pro x64   IntelCore2Duo P8700 2,53 GHz

Run as 32Bit - 2,7 s

[orbs]

thank for your feedback! this is only a 4KB sample. now try it on a real-life size, >100KB

P.S.

CPU: i5-2500 @ 3.3GHz

RAM: DDR3 16GB @ 666MHz

GPU: [integrated] Intel HD Graphics 1000 (GT1) @ 850MHz

running as 32-bit or 64-bit - no difference

Edited February 14, 2015 by orbs

[BrewManNH]

On my less than stellar laptop at home the script in the OP runs in 2.5 seconds x64 and 3.8 seconds running 32 bits.

 

Intel® Core2 Duo CPU T7100  @ 1.80GHz, 1800 Mhz

As you can see this computer isn't anything special and is REALLY slow.

[kylomas]

AMD Athlon II 620 QuadCore

2.5 seconds on my turtle...

[LarsJ]

orbs, Why do you think this is a performance issue, and why do you expect it to go faster? Even a C++ program would probably be rather slow, if it should use the Rich Edit DLL for such a task. Simply because the Rich Edit DLL is not designed to set a different color for each character in a large string in a fast way.

[orbs]

the starting point for this entire thing is WinMerge (and other comparison tools, for that matter). when it needs to highlight differences between files, it shows both files side-by-side with highlights, and does it blazingly fast.

i was under the impression it uses RichEdit for that purpose (because regular Edit control does not supports coloring), but now i examined it with the AutoIt Window Info tool, and found it seems to use its own custom class for this.

anyway, for me i'll stick to RichEdit, but i think i thought of an optimization technique. i'm building the RichEdit after it was analyzed for the text that should be highlighted, so:

currently, i highlight all instances of the text, and then the user is given control.

i'm about to check this alternative: build the RichEdit with the text only (no highlights at all), then give the user the control, and highlight every instance of the text only when that instance becomes visible.

so basically, i need to:

(1) determine what are the first and last characters visible.

(2) detect changes to the visible text area.

for (1) i'm already using _GUICtrlRichEdit_GetCharPosFromXY() with decent success; for (2), that depends on scrolling, keyboard arrow keys moving the caret, window resize, and probably more. i'm going to try and trap it with GUIRegisterMsg(), but if there is any simpler way, i'll be happy to know...

 

EDIT: will the event $EN_CHANGE work for change in visible text, or is it only for change in text? are there RichEdit-specific events i can trap that will be useful?

Edited February 15, 2015 by orbs

[LarsJ]

You can find all information including lists of messages and notifications here.

[orbs]

this is what i did eventually:

i need the first and last visible chars. for that, i'm using these:

Func _GUICtrlRichEdit_VisibleChar_GetFirst($hRichEdit)
    Return _GUICtrlRichEdit_GetCharPosFromXY($hRichEdit, 1, 1)
EndFunc   ;==>_GUICtrlRichEdit_VisibleChar_GetFirst

Func _GUICtrlRichEdit_VisibleChar_GetLast($hRichEdit, $nRE_W, $nRE_H)
    Local $iResult = 0
    Local $iDiag = 0
    Do
        $iDiag += 1
        $iResult = _GUICtrlRichEdit_GetCharPosFromXY($hRichEdit, $nRE_W - $iDiag, $nRE_H - $iDiag)
    Until $iResult > 0
    Return $iResult
EndFunc   ;==>_GUICtrlRichEdit_VisibleChar_GetLast

(the reason i'm scanning from lower-right corner diagonally inwards, is because i don't know if any scrollbar is visible, or what is the  scrolllbar width).

now, i need to know when there was a change in visible text. that change simply should be an indication to re-check the first and last chars. so, instead of using messages and notifications (which i did not find anything simple that will tell me about change in visible text), i'm constantly re-checking the first and last chars (in the main loop). if any of them changed, i need to process what's in the middle. it's a crude method, but it works... kind of. i'm still trying to optimize it.

[LarsJ]

Take a look at the EN_MSGFILTER notification.

[orbs]

did, and several others. it's simply that there are so many ways in which visible text can change: scrollbars click, scrollbars drag, arrow keys, font change, font size change, window resize, internal commands - which i'm using a lot, like _GUICtrlRichEdit_SetSel(), _GUICtrlRichEdit_GotoCharPos() etc. - and probably others i did not list here. EN_MSGFILTER can handle mouse & keyboard events; fine. but foreseeing all possible events requires combining notifications from several sources, which quickly got a lot messier than i'd prefer.

[Melba23]

orbs,

You could try adding the highlighting manually by extracting, modifying and reloading the RTF stream - something like this:

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <GuiRichEdit.au3>

Global Const $nGUI_MaxW = 500, $nGUI_MaxH = 400
Global $nGUI_W = 720, $nGUI_H = 500, $nRichEditOffset = 10, $nToolbarOffset = 35

Global $hGui = GUICreate('_GUICtrlRichEdit_SetCharBkColor performance test', $nGUI_W, $nGUI_H)
GUISetBkColor(0xCCDDEE)

Global $hRichEdit = _GUICtrlRichEdit_Create($hGui, FileRead(@ScriptFullPath) & @LF, $nRichEditOffset, $nRichEditOffset + $nToolbarOffset, $nGUI_W - $nRichEditOffset * 2, $nGUI_H - $nRichEditOffset * 2 - $nToolbarOffset - 24, BitOR($ES_MULTILINE, $ES_READONLY, $WS_HSCROLL, $WS_VSCROLL))
_GUICtrlRichEdit_SetFont($hRichEdit, Default, 'Courier New')

$nBegin = TimerInit()

; Make sure there is no selection
_GUICtrlRichEdit_Deselect($hRichEdit)
; Read the RTF stream
$sContent = _GUICtrlRichEdit_StreamToVar($hRichEdit)

; Define the highlight colour
$sColDef = "{\colortbl ;\red255\green128\blue128;}" ; This is \highlight1, you can add as many others as you want
; And add it as line 2
$sContent = StringReplace($sContent, ";}}", ";}}" & @CRLF & $sColDef)

; Search for RichEdit
$sSearch = "RichEdit"
; And highlight all occurences
$sContent = StringRegExpReplace($sContent, $sSearch, "\\highlight1 " & $sSearch & "\\highlight0 ") ; highlight0 is the base colour

; Now replace the current text with thhighlighted version
_GUICtrlRichEdit_StreamFromVar($hRichEdit, $sContent)

ConsoleWrite(TimerDiff($nBegin) & @CRLF)

GUISetState()

While 1
    Switch GUIGetMsg()
        Case $GUI_EVENT_CLOSE
            _GUICtrlRichEdit_Destroy($hRichEdit)
            Exit
    EndSwitch
WEnd

That runs lightning fast on my system.

M23

[orbs]

never in my entire life have i been so glad to ditch an honest days work.

this looks very potent! however i'm having a small difficulty:

the data of substrings to highlight is formatted as characters index, not as contents of substrings. for example, i have this list:

...

22,27,1

54,88,2

...

meaning: highlight chars 22 to 27 in predefined color #1, highlight chars 54 to 88 in predefined color #2, etc.

when i'm working with the raw text, locating the substrings is no problem; but the stream has much additional data in it.

i'm having trouble locating the position of the substrings in the stream. is there a built-in method to do this?

[Melba23]

orbs,

is there a built-in method to do this?

No - but I am sure we can come up with something.

M23

[orbs]

i figured out a way to detect the first char in the stream - and knowing that the stream origin is plain text, and that each @CRLF is now accompanied by "par", i think i can manage to locate substrings. this is how i do it:

this gives me the content of the first line:

Global $sContentSource=FileRead(@ScriptFullPath)
Global $sContentSourceLine1=StringLeft($sContentSource,StringInStr($sContentSource,@CRLF)-1)

and the first occurrence in the stream of that line, suffixed by "par", is the starting of the text. this is given by:

StringInStr($sContent,$sContentSourceLine1&'\par')

i added dump to temp file the content of the string, before and after the insertion of the highlights. opening it in Notepad++ i can see that the location is correct. this is the complete demo:

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <GuiRichEdit.au3>

Global Const $nGUI_MaxW = 500, $nGUI_MaxH = 400
Global $nGUI_W = 720, $nGUI_H = 500, $nRichEditOffset = 10, $nToolbarOffset = 35

Global $hGui = GUICreate('_GUICtrlRichEdit_SetCharBkColor performance test', $nGUI_W, $nGUI_H)
GUISetBkColor(0xCCDDEE)

Global $sContentSource=FileRead(@ScriptFullPath) ; reading it to var because will work with this twice
Global $sContentSourceLine1=StringLeft($sContentSource,StringInStr($sContentSource,@CRLF)-1) ; this is the text of the first line

Global $hRichEdit = _GUICtrlRichEdit_Create($hGui, $sContentSource, $nRichEditOffset, $nRichEditOffset + $nToolbarOffset, $nGUI_W - $nRichEditOffset * 2, $nGUI_H - $nRichEditOffset * 2 - $nToolbarOffset - 24, BitOR($ES_MULTILINE, $ES_READONLY, $WS_HSCROLL, $WS_VSCROLL))
_GUICtrlRichEdit_SetFont($hRichEdit, Default, 'Courier New')

$nBegin = TimerInit()

; Make sure there is no selection
_GUICtrlRichEdit_Deselect($hRichEdit)
; Read the RTF stream
$sContent = _GUICtrlRichEdit_StreamToVar($hRichEdit)

; unmodified stream - write to file for later check
FileDelete(@TempDir&'\rtf-stream.txt')
FileWrite(@TempDir&'\rtf-stream.txt',$sContent)

; Define the highlight colour
$sColDef = "{\colortbl ;\red255\green128\blue128;}" ; This is \highlight1, you can add as many others as you want
; And add it as line 2
$sContent = StringReplace($sContent, ";}}", ";}}" & @CRLF & $sColDef)

; find first char position
ConsoleWrite('1st char position: '&StringInStr($sContent,$sContentSourceLine1&'\par') & @LF)

; Search for RichEdit
$sSearch = "RichEdit"
; And highlight all occurences
$sContent = StringRegExpReplace($sContent, $sSearch, "\\highlight1 " & $sSearch & "\\highlight0 ") ; highlight0 is the base colour

; modified stream - write to file for later check
FileDelete(@TempDir&'\rtf-stream-after.txt')
FileWrite(@TempDir&'\rtf-stream-after.txt',$sContent)

; Now replace the current text with thhighlighted version
_GUICtrlRichEdit_StreamFromVar($hRichEdit, $sContent)

ConsoleWrite(TimerDiff($nBegin) & @CRLF)

GUISetState()

While 1
    Switch GUIGetMsg()
        Case $GUI_EVENT_CLOSE
            _GUICtrlRichEdit_Destroy($hRichEdit)
            Exit
    EndSwitch
WEnd

now i'm struggling with locating substrings in next lines...

[Melba23]

orbs,

Do the index values you have for the highlighting ignore the par EOL? Or do we have to take them into account?

M23

[orbs]

yes, ignoring "par", because it's origin is plain text and not aware of the RTF stream.

the real trouble i'm facing is that i can't determine if it's ignoring @CRLF or just @LF, or not at all.

[Melba23]

orbs,

 

i can't determine if it's ignoring @CRLF or just @LF, or not at all.

Ah, that makes it a bit tricky. >

However, I am currently working on splitting the stream into separate "lines", so if you can determine how the EOLs are dealt with it should be relatively simple to add either 0, 1 or 2 characters to the count every time we move to a new line. I will keep working on the code as a "proof of concept" and we can revisit this EOL problem later.

M23

[orbs]

splitting the stream into separate "lines",

 

brilliant! should be rather easy (given stream origin is plain text, of course) - trim the part before the first char, then StringSplit by "par". i'll see what i can make of this.

EDIT: oops, forgot about the ending... the last line, which in the source text is simply WEnd, in the stream looks like this:

WEnd\lang1037\f1\rtlch\par
}

with @CRLF after that "par" and after the closing curly brackets at the following line.

EDIT #2: but this can be ignored for now, because it should have no effect on the rest of the text and on the highlights indices.

Edited February 15, 2015 by orbs