Sign in to follow this  
Followers 0

Detect lines of black text on screen (not OCR)

2 posts in this topic

#1 ·  Posted (edited)

Hello forums,

I'd like to know if anyone has better idea than what I have. I'm creating a program that will detect (not OCR!) blocks of words (black text) on a white background displayed on the screen. (If anyone is curious about its purpose, this code will be part of a program that will randomly select a word on a PDF file and have the user guess that word. I'll be using it to quiz myself with my PDF notes. )

What I created so far: Using FastFind.au3, I was able to make it detect lines of black text (font Lucida Console) on a single column. It basically considers a line a separate one if bounded above and below by horizontal lines that do not contain black pixels. Next step, which I haven't done yet, will be to go through each line and detect the blocks of words.

The program works fine so far with Lucida Console font but it doesn't when tested on TNR font (see attached sample image). The former can be tested by setting $iOption to "text" while the latter by setting it to "select".

So does anyone have a better idea on going about this? Thank you!

Sorry if my code's a mess. I tried to make it cleaner before posting it but this is the best I can manage.

#include <FastFind.au3>
#include <array.au3>
HotKeySet('{ESC}', '__Exit')

Local $iImagePos[4], $iXa, $iXb, $iYa, $iYb
$iOption = 'text'
Switch $iOption
  Case 'text'   ; Creates sample text file for detection
    Local Const $sSampleTextFileName = 'sample.txt'
    FileWrite($sSampleTextFileName, '')
    FileOpen($sSampleTextFileName, 2)
    FileWrite($sSampleTextFileName, 'First line.' & @CRLF & 'Second line.' & @CRLF & @CRLF & 'Third line.')
    Local Const $sSampleTextFileTitle = 'sample - Notepad'
    WinMove($sSampleTextFileTitle, '', 0, 0, @DesktopWidth / 2, @DesktopHeight / 2)
    $iImagePos = __ControlGetPos($sSampleTextFileTitle, '[CLASS:Edit; INSTANCE:1]') ; Gets the absolute coordinates of the edit control.
    Local $iXa = $iImagePos[0], _
        $iXb = $iImagePos[0] + $iImagePos[2], _
        $iYa = $iImagePos[1], _
        $iYb = $iImagePos[1] + $iImagePos[3]
  Case 'select'
    MsgBox(0, 'Select', 'Select first')
    $iXa = MouseGetPos(0)
    $iYa = MouseGetPos(1)
    MsgBox(0, 'Select', 'Select second')
    $iXb = MouseGetPos(0)
    $iYb = MouseGetPos(1)

Local $array[1][2] ; Will be used for saving coordinates of lines of text
Local $iBlackPixelCount
Local $bHasNoPreviousZero = True ; Will be used as a toggle for skipping succeeding lines that have no black pixels.
Local $iStep = 1
For $y = $iYa To $iYb Step $iStep
  FFSnapShot($iXa, $y, $iXb, $y)
  $iBlackPixelCount = FFColorCount(0x000000, 50, False)
  ; $iBlackPixelPercent = Round(100 * (FFColorCount(0x000000, 0, False) / $iImagePos[2]), 2) ; Computes for percentage of black pixels on horizontal line.
  ; If $iBlackPixelCount > 0 Or $bHasNoPreviousZero Then
  ; $bHasNoPreviousZero = True
  _ArrayAdd($array, $iBlackPixelCount)
  $array[UBound($array) - 1][1] = $y
  ; If $iBlackPixelCount = 0 Then $bHasNoPreviousZero = False
  ; EndIf

; To check detected lines. This will move mouse to the lower-left corner of the text line.
For $i = 0 To UBound($array) - 1 Step +1
  If $array[$i][0] > 0 Then
    If $i + 1 <= UBound($array) - 1 Then
      If $array[$i + 1][0] = 0 Then
        MouseMove($iXa, $array[$i][1])


Func __ControlGetPos($hWnd, $controlID, $bAbsolute = Default)
  If $bAbsolute = Default Then $bAbsolute = True
  Local $controlPos
  Switch $bAbsolute
    Case True
      Local Const $hWnd_Control = ControlGetHandle($hWnd, "", $controlID)
      $controlPos = WinGetPos($hWnd_Control)
    Case False
      $controlPos = ControlGetPos($hWnd, "", $controlID)
  If Not IsArray($controlPos) Then SetError(1)
  Return $controlPos
EndFunc   ;==>__ControlGetPos

Func __Exit()
EndFunc   ;==>__Exit





sample image.bmp

Edited by Mingre

Share this post

Link to post
Share on other sites

Hello instead of checking for black pixels, I changed it to check white pixels and it now works on TNR font. Thanks everyone! :) 

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • ronaldo97
      By ronaldo97
      Hello How can I make the GUI is compatible with all screen sizes ??  
      #include <GUIConstantsEx.au3> #include <StaticConstants.au3> #include <WindowsConstants.au3> #Region ### START Koda GUI section ### Form= $Form1 = GUICreate("Form1", 623, 445, 192, 124, BitOR($GUI_SS_DEFAULT_GUI,$WS_MAXIMIZE)) GUISetBkColor(0x1E1E1E) $Label1 = GUICtrlCreateLabel("Welcome Back !!", 424, 64, 161, 29) GUICtrlSetFont(-1, 16, 400, 0, "Tahoma") GUICtrlSetColor(-1, 0x800000) GUISetState(@SW_SHOW) #EndRegion ### END Koda GUI section ### While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit EndSwitch WEnd How do I make Gui interface compatible with all screen sizes ?? 800*600
    • marcoauto
      By marcoauto
      I need to scale an outlined GDI text in a GUI. The problem is that I can't do this. The outline is often out of the GUI
      I started from this script:
      that fit a string to windows and I changed
       $aBounds = _GDIPlus_PathGetWorldBounds($hPath) ;Get bounding rectangle of the path
      $aBounds = _GDIPlus_PathGetWorldBounds($hPath) ;Get bounding rectangle of the path with 
      $aBounds = _GDIPlus_PathGetWorldBounds($hPath,$hMatrix,$hPen) ;Get bounding rectangle of the path with pen size but doesn't work
      so, I tried to change other thins, but nothing... Sometimes works, but if I change the string often result change
      This is my script:
      #include <GUIConstantsEx.au3> #include <GDIPlus.au3> #include <array.au3> Example() Func Example() Local $iW, $iH, $hGUI, $hGraphic, $hBrush, $hPen, $hPath, $hFamily, $tLayout, $hMatrix, $aBounds ; Create GUI $iW = 650 $iH = 300 $hGUI = GUICreate("GDI+", $iW, $iH) GUISetState(@SW_SHOW) ; Draw a string using path _GDIPlus_Startup() $hGraphic = _GDIPlus_GraphicsCreateFromHWND($hGUI) ;Create a graphics object from a window handle _GDIPlus_GraphicsSetSmoothingMode($hGraphic, $GDIP_SMOOTHINGMODE_HIGHQUALITY) ;Sets the graphics object rendering quality (antialiasing) ;~ _GDIPlus_GraphicsClear($hGraphic, 0xFF000000) $MergedImageBackgroundColor = 0x00000000 _GDIPlus_GraphicsClear($hGraphic, $MergedImageBackgroundColor) $hBrush = _GDIPlus_BrushCreateSolid(0xFFDD2200) $stroke_size = 15 $hPen = _GDIPlus_PenCreate(0xFFFFBB00, $stroke_size) ;stroke size in pixel $hPen2 = _GDIPlus_PenCreate(0xFFFFff00, 2) ;stroke size in pixel _GDIPlus_PenSetLineJoin($hPen,$GDIP_PENSETLINEJOIN_ROUND) $hPath = _GDIPlus_PathCreate() ;Create new path object ;~ $fSize = 20 $hFamily = _GDIPlus_FontFamilyCreate("Myriad Pro") ;Create font family object $tLayout = _GDIPlus_RectFCreate() ;Create string bounding rectangle X=0, Y=0 Local $hFormat = _GDIPlus_StringFormatCreate() _GDIPlus_PathAddString($hPath, "autoitForeverLollipop", $tLayout, $hFamily);, 0, $fSize, $hFormat) ;Add the outline of the string to the path ;~ _GDIPlus_PathAddString($hPath, $sText, $tLayout, $hFamily, 0, $fSize, $hFormat) ; Tranform Path to fit to window $aBounds = _GDIPlus_PathGetWorldBounds($hPath) ;Get bounding rectangle of the path $aPoints = _GDIPlus_PathGetPoints($hPath) ConsoleWrite($aBounds[0] & @CRLF) ConsoleWrite($aBounds[1] & @CRLF) ConsoleWrite($aBounds[2] & @CRLF) ConsoleWrite($aBounds[3] & @CRLF) ConsoleWrite($aPoints[$aPoints[0][0]][0] & " - " & $aPoints[1][0] & " + 2*" & $stroke_size & " = " & $aPoints[$aPoints[0][0]][0]-$aPoints[1][0]+2*$stroke_size & @CRLF) ;~ $aBounds[0] = $aBounds[0] $aBoundsNp = _GDIPlus_PathGetWorldBounds($hPath,0,$hPen) ;Get bounding rectangle of the path with pen ConsoleWrite($aBoundsNp[0] & @CRLF) ConsoleWrite($aBoundsNp[1] & @CRLF) ConsoleWrite($aBoundsNp[2] & @CRLF) ConsoleWrite($aBoundsNp[3] & @CRLF) ConsoleWrite($iW & " / 2 = " & $iW / 2 & @CRLF) ConsoleWrite("$ih / 2 = " & $iH / 2 & @CRLF) ;~ ConsoleWrite($iW & " $aBounds[2] = " & $iW / $aBounds[2] & @CRLF) ConsoleWrite($iW & " $aBoundsNp[2] = " & $iW / $aBoundsNp[2] & @CRLF) ConsoleWrite($iH & " $aBoundsNp[3] = " & $iH / $aBoundsNp[3] & @CRLF) ;~ _GDIPlus_GraphicsDrawPath($hGraphic, $hPath, $hPen) ;Draw path to graphics handle (GUI) ;~ _GDIPlus_GraphicsFillPath($hGraphic, $hPath, $hBrush) ;Fill path to graphics handle (GUI) $hMatrix = _GDIPlus_MatrixCreate() $iW = $iW - $stroke_size ;+ $stroke_size $iH = $iH - $stroke_size ConsoleWrite(" -$aBounds[0]+1*$stroke_size/72 = " & -$aBounds[0]+1*$stroke_size/72 & @CRLF) ConsoleWrite(" -1.8+1*$stroke_size/72 = " & -1.8+1*$stroke_size/72 & @CRLF) _GDIPlus_MatrixTranslate($hMatrix, -$aBounds[0]+2*$stroke_size/72, -$aBounds[1]+1*$stroke_size/72) ;Translate Matrix to the offset of the bounding rectangle ;~ _GDIPlus_MatrixTranslate($hMatrix, -1.59166666666667+0*($stroke_size/72), -$aBounds[1]+1*$stroke_size/72) ;Translate Matrix to the offset of the bounding rectangle ;~ _GDIPlus_MatrixScale($hMatrix, $iW / $aBounds[2], $iH / $aBounds[3], True) ;Scale Matrix _GDIPlus_MatrixScale($hMatrix, $iW / $aBounds[2], $iH / $aBounds[3], True) ;Scale Matrix _GDIPlus_PathTransform($hPath, $hMatrix) ;Translate and Scale Path $aBoundsNp = _GDIPlus_PathGetWorldBounds($hPath,0,$hPen) ;Get bounding rectangle of the path with pen ConsoleWrite($aBoundsNp[0] & @CRLF) ConsoleWrite($aBoundsNp[1] & @CRLF) ConsoleWrite($aBoundsNp[2] & @CRLF) ConsoleWrite($aBoundsNp[3] & @CRLF) ;~ _GDIPlus_GraphicsFillPath($hGraphic, $hPath, $hBrush) ;Fill path to graphics handle (GUI) _GDIPlus_GraphicsDrawPath($hGraphic, $hPath, $hPen) ;Draw path to graphics handle (GUI) _GDIPlus_GraphicsFillPath($hGraphic, $hPath, $hBrush) ;Fill path to graphics handle (GUI) ; Loop until the user exits. Do Until GUIGetMsg() = $GUI_EVENT_CLOSE ; Clean up resources _GDIPlus_MatrixDispose($hMatrix) _GDIPlus_FontFamilyDispose($hFamily) _GDIPlus_PathDispose($hPath) _GDIPlus_PenDispose($hPen) _GDIPlus_BrushDispose($hBrush) _GDIPlus_GraphicsDispose($hGraphic) _GDIPlus_Shutdown() EndFunc ;==>Example if you change the line 
      _GDIPlus_PathAddString($hPath, "autoitForeverLollipop", $tLayout, $hFamily);, 0, $fSize, $hFormat) ;Add the outline of the string to the path with a smaller string
      _GDIPlus_PathAddString($hPath, "autoitFitted", $tLayout, $hFamily);, 0, $fSize, $hFormat) ;Add the outline of the string to the path the result is ok
      Can someone can help me?
    • marcoauto
      By marcoauto
      Is there a function to create a stroke text?
      I have found the solution to fill the text with a texture, to make a shadow and to make a text with glow , but I can't reproduce the stoke effect.)
      I have attached an example image of what I would like to reproduce
      I have tried to adapt this script of Malkey but with no result
      The Malkey script is this (I have edit some line) and the original post is here: 
      #include <GDIPlus.au3> #include <WinAPI.au3> #include <GuiConstants.au3> #include <WindowsConstants.au3> Global $nPI = 3.1415926535897932384626433832795 Global $ghGDIPDll $hWnd = GUICreate("GDI+ Example", 800, 150) $iGlow = 2; <== Amount of Glow min = 1 Max about 5 _GDIPlus_Startup() $hGraphicGUI = _GDIPlus_GraphicsCreateFromHWND($hWnd) $hBMPBuff = _GDIPlus_BitmapCreateFromGraphics(800, 150, $hGraphicGUI) $hGraphics = _GDIPlus_ImageGetGraphicsContext($hBMPBuff) _GDIPlus_GraphicsClear($hGraphics, 0xff003000) ; Black-green background ;~ _GDIPlus_GraphicsClear($hGraphics, 0xffffffff) ; White Background _AntiAlias($hGraphics, 4) $hFamily = _GDIPlus_FontFamilyCreate("Arial Black") $hFont = _GDIPlus_FontCreate($hFamily, 80) $hLayout = _GDIPlus_RectFCreate(0, 0, 800, 150) $hStringFormat = _GDIPlus_StringFormatCreate() _GDIPlus_StringFormatSetAlign($hStringFormat, 1) For $i = 1 To 15 Step 3 $hBrush3 = _GDIPlus_BrushCreateSolid("0x" & hex($iGlow,2) & "00ff00"); <====== Glow (Transparency) For $x = 0 To 360 Step 3 DllStructSetData($hLayout, "Y", ((18 - $i)) * Sin($x * $nPI / 180)) DllStructSetData($hLayout, "X", ((18 - $i)) * Cos($x * $nPI / 180)) $hStringFormat = _GDIPlus_StringFormatCreate() _GDIPlus_GraphicsDrawStringEx($hGraphics, "AutoIt Rocks", $hFont, $hLayout, $hStringFormat, $hBrush3) Next Next DllStructSetData($hLayout, "Y", 0) DllStructSetData($hLayout, "X", 0) $hStringFormat = _GDIPlus_StringFormatCreate() $hBrush3 = _GDIPlus_BrushCreateSolid("0xfF00FF00") _GDIPlus_GraphicsDrawStringEx($hGraphics, "AutoIt Rocks", $hFont, $hLayout, $hStringFormat, $hBrush3) GUISetState() GUIRegisterMsg(0xF, "MY_PAINT"); Register PAINT-Event 0x000F = $WM_PAINT (WindowsConstants.au3) GUIRegisterMsg(0x85, "MY_PAINT") ; $WM_NCPAINT = 0x0085 (WindowsConstants.au3)Restore after Minimize. _GDIPlus_GraphicsDrawImage($hGraphicGUI, $hBMPBuff, 0, 0) Do $msg = GUIGetMsg() Until $msg = $GUI_EVENT_CLOSE _GDIPlus_BrushDispose($hBrush3) _GDIPlus_FontFamilyDispose($hFamily) _GDIPlus_FontDispose($hFont) _GDIPlus_StringFormatDispose($hStringFormat) _GDIPlus_GraphicsDispose($hGraphics) _GDIPlus_GraphicsDispose($hGraphicGUI) _WinAPI_DeleteObject($hBMPBuff) _GDIPlus_Shutdown() Func _AntiAlias($hGraphics, $iMode) Local $aResult $aResult = DllCall($ghGDIPDll, "int", "GdipSetSmoothingMode", "hwnd", $hGraphics, "int", $iMode) If @error Then Return SetError(@error, @extended, False) Return SetError($aResult[0], 0, $aResult[0] = 0) EndFunc ;==>_AntiAlias ;Func to redraw on PAINT MSG Func MY_PAINT($hWnd, $msg, $wParam, $lParam) _GDIPlus_GraphicsDrawImage($hGraphicGUI, $hBMPBuff, 0, 0) _WinAPI_RedrawWindow($hWnd, "", "", BitOR($RDW_INVALIDATE, $RDW_UPDATENOW, $RDW_FRAME)) ; , $RDW_ALLCHILDREN Return $GUI_RUNDEFMSG EndFunc ;==>MY_PAINT Thank you very much,

    • Blueman
      By Blueman
      Hey Guys,
      I am Struggling with a issue that i cannot seem to fix.
      I hope that you guys can help me with this.
      I have a GUI with some text attributes, for all i use the following code;
      $Label_ts_Header = GUICtrlCreateLabel("Manual Adress", 60, 10, 500, 50) GUICtrlSetBkColor($Label_ts_Header, $COLOR_WHITE) GUICtrlSetFont ($Label_ts_Header, 26, 800, 0, "Calibri") This is all working like it should, but when i activate the GUI on another pc the text size is bigger and the GUI size is somewhat smaller.
      The problem is that it isn't on all computers, i can't seem to find the problem,..
      If i make it fit on one pc it is to small on another or to big,..
      What am i doing wrong?
      Is there some kind of attribute that i need to add to the top of my script to normalize text sizes in the script?
      EDIT: Found the problem guys, some computers has changed the text size to 125% // Can i force 100% for my GUI somehow ?
      EDIT2: Fixed it guys! // I used the Function Below to get the DPI Setting and used that to determine the text size (example below)
      Func _GDIPlus_GraphicsGetDPIRatio($iDPIDef = 96) _GDIPlus_Startup() Local $hGfx = _GDIPlus_GraphicsCreateFromHWND(0) If @error Then Return SetError(1, @extended, 0) Local $aResult #forcedef $__g_hGDIPDll, $ghGDIPDll $aResult = DllCall($__g_hGDIPDll, "int", "GdipGetDpiX", "handle", $hGfx, "float*", 0) If @error Then Return SetError(2, @extended, 0) Local $iDPI = $aResult[2] Local $aresults[2] = [$iDPIDef / $iDPI, $iDPI / $iDPIDef] _GDIPlus_GraphicsDispose($hGfx) _GDIPlus_Shutdown() Return $aresults EndFunc ;==>_GDIPlus_GraphicsGetDPIRatio  
      $Label_S2 = GUICtrlCreateLabel("Data from The Base", 10, 300, 400, 20) GUICtrlSetBkColor($Label_S2, $COLOR_WHITE) GUICtrlSetFont ($Label_S2, 13 * _GDIPlus_GraphicsGetDPIRatio()[0])  
    • mazz3d
      By mazz3d
      hello  everyone,
      I'm new with AutoIT and just looking for a way to open Firefox with random window size
      i have tried too many different samples from the web, some how it works but i can't make it works with Firefox 
      MozRpel and ff.au3 installed 
      right now i can open Firefox with this code, but i still can't control the window of Firefox
      ; open firefox #include <ff.au3> _FFStart() ; open firefox with random size