I had to change the ImageFile parameter when calling the DLL function to get transparency to work. It was missing the correct prefix for the transparency option.
For reference, the Image file options are here: https://autohotkey.com/docs/commands/ImageSearch.htm
*TransN: This option makes it easier to find a match by specifying one color within the image that will match any color on the screen.
This is my updated _ImageSearchArea function with the transparency option fixed. Notice the inclusion of the *Trans text in the ImageFile parameter parameters.
If $transparency <> 0 Then $ImageFileOptions = "*Trans" & Hex($transparency) & " " & $ImageFileOptions
Func _ImageSearchArea($findImage, $resultPosition, $x1, $y1, $right, $bottom, ByRef $x, ByRef $y, $tolerance = 0, $transparency = 0);Credits to Sven for the Transparency addition
If Not FileExists($findImage) Then Return "Image File not found"
If $tolerance < 0 Or $tolerance > 255 Then $tolerance = 0
If $h_ImageSearchDLL = -1 Then _ImageSearchStartup()
Local $ImageFileOptions = ""
If $tolerance > 0 Then $findImage = "*" & $tolerance & " " & $ImageFileOptions
If $transparency <> 0 Then $ImageFileOptions = "*Trans" & Hex($transparency) & " " & $ImageFileOptions
$ImageFileOptions = $ImageFileOptions & $findImage
$result = DllCall($h_ImageSearchDLL, "str", "ImageSearch", "int", $x1, "int", $y1, "int", $right, "int", $bottom, "str", $ImageFileOptions)
If @error Then Return "DllCall Error=" & @error
If $result = "0" Or Not IsArray($result) Or $result[0] = "0" Then Return False
$array = StringSplit($result[0], "|")
If (UBound($array) >= 4) Then
$x = Int(Number($array[2])); Get the x,y location of the match
$y = Int(Number($array[3]))
If $resultPosition = 1 Then
$x = $x + Int(Number($array[4]) / 2); Account for the size of the image to compute the centre of search
$y = $y + Int(Number($array[5]) / 2)
EndIf
Return True
EndIf
EndFunc ;==>_ImageSearchArea