Sign in to follow this  
Followers 0
Rnde

Trouble with regex

12 posts in this topic

#1 ·  Posted (edited)

Hello,

I'm trying to make my own custom script obfuscator, mostly to learn more about regular expression, however i have run into a generic problem that i've been stuck with for a few days even though it looks fairly easy.

I have this string:

Global $a = 1, $b
$c = $a + 1
 
I want to get all the variable names from lines starting with "Global", which is "$b" and "$a" in this case, in a single StringRegExp call.
 
I have tried this:
$sString = "Global $a, $b" & @CRLF & "$c = $a + 1" ;Given string
$aVariables = StringRegExp($sString, "\b(?U)Global (?:.*)\$(\w+)", 3)
 

And this:

$sString = "Global $a, $b" & @CRLF & "$c = $a + 1" ;Given string
$aVariables = StringRegExp($sString, "\b(?U)(?<=Global).*\$(\w+)", 3)

But so far it only captures the first variable with (?U) quantifier and the last variable without (?U).

Anyone knows what i'm doing wrong ? This will help me a lot of the future since i have come across problems similar like this many times before.

Thanks in advance.

Edited by Rnde

Share this post


Link to post
Share on other sites



You will always get only one variable if you don't put an alternation in the lookbehind

Try this

#include <Array.au3>
$sString = 'Global $a, $b = 0, $c = "something"' & @CRLF & '$c = $a + 1'    ;Given string
$aVariables = StringRegExp($sString, "(?<=Global|,)\h*(\$\w+)", 3)
_ArrayDisplay($aVariables)

Share this post


Link to post
Share on other sites

I will just say that you should temporarily remove comments and strings before using the above regular expression. I did alot of work for PreExpand on this subject.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Hey

Thank you both for the replies.

@mikell: That's a brilliant and interesting take on the problem, i have been looking at it the wrong way the whole time.

But let's say now i have this string:

I have an eraser and 2 pencils
Jane has a ruler and a stapler
 
What if i want to get the name of the items that i have ?
The method you used for finding globally declared vars will apparently not work in this case. However i will give it a try and apply mikell's method, pardon if i'm being inept and cant come up with anyway more creative:
 
#include <array.au3>
$sString = 'I have 2 pencils and an eraser' & @CRLF & 'Jane has a ruler and a stapler'
$aMyItems = StringRegExp($sString, "(?<=I have|and) (?:a|an|\d+) (\w+)", 3)
_ArrayDisplay($aMyItems)
 
Apprently the above script will return both jane's and my items, i'm looking for a way to match a given string/expression at the absolute beginning of a line (or somewhere before the capturing group) if it's possible.
 

@guinness: Yes i actually process my target script with tidy.exe and au3stripper.exe before starting the obfuscation sequence.

 
Edited by Rnde

Share this post


Link to post
Share on other sites

Alas, regex is not magic and can't solve all cases, and is a delicate thing which can easily fail

Sometimes the best way (as guinness warned) is to add some steps to make the code more secure

#include <array.au3>
$sString = 'I have 2 pencils and an eraser' & @CRLF & 'Jane has a ruler and a stapler' & @CRLF & 'I have also a brain'

Local $aMyItems[0]
$lines = StringSplit($sString, @crlf, 1)
For $i = 1 to $lines[0]
  If StringInStr($lines[$i], "I have") Then 
      $tmp = StringRegExp($lines[$i], "\h+(?:a|an|\d+)\h+(\w+)", 3)
      If IsArray($tmp) Then _ArrayAdd($aMyItems, $tmp)
  EndIf
Next  
_ArrayDisplay($aMyItems)

Share this post


Link to post
Share on other sites

 

@guinness: Yes i actually process my target script with tidy.exe and au3stripper.exe before starting the obfuscation sequence.

 

 

Well this doesn't solve the problem if you have a string that contains "Global $aArray[] = [1, 2, 3, 4, 5]"


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

Well this doesn't solve the problem if you have a string that contains "Global $aArray[] = [1, 2, 3, 4, 5]"

I had to step backwards in the code ( remove single and double quotes, continuation lines, comments, etc ) with EncodeIt obfuscator (before most of your time here).

I actually started writing another one 2 years ago just before I quit programming all together.

Anyway, It's funny, but just this morning, I had to do the single/double quote replacement for a pseudo function:

Take a look at $aDQ and $aSQ regex's.  Would probably use a larger number than %03d on stringformat, for a whole script though.


[center]Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.[/center]

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Oh cool. I came up with this a couple of years ago.

Edited by guinness

_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

F.Y.I.

This 

global _
            $a = 1, _
            $b = 2, _
            $c = 3

and many other variations are valid syntax...


Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites

F.Y.I.

This 

global _
            $a = 1, _
            $b = 2, _
            $c = 3

and many other variations are valid syntax...

Exactly, which is what we're kind of talking about :).

If you see my comment about what needs to be done:

store double quotes, replace them with temp val

store single quotes, replace them with temp val

remove comments: #comment/#ce/;

strip and remove continuation lines

Then you can start working on vars and funcs

Being sure to sort them from largest to smallest before replacing with obfuscated vars/func names

There's more, but you get the gist


[center]Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.[/center]

Share this post


Link to post
Share on other sites

#11 ·  Posted (edited)

F.Y.I.

This 

global _
            $a = 1, _
            $b = 2, _
            $c = 3

and many other variations are valid syntax...

Might want to check my link above.

Whoops, I was meant to post it: 

Edit: These functions might of be some use too: 

Edited by guinness

_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

Thanks all for the replies, those were some helpful info.
 
@guinness: i know what you mean now, i also looked at the referred threads, processing literal strings first is indeed the correct way, i have tried the method that grabs all the strings and replace them one by one but this method's performance was getting proportionally slow the more literal strings the script contains.
 
So, i came up with a way to do all the string processing in 1 call, this also works on strings that contain both single and double quote for me so far:
 
$sTargetDir = "SomeScript_stripped.au3" ;Only works on tidied and stripped scripts
$sScriptContent = FileRead($sTargetDir)
$sScriptContent = Execute('"' & StringRegExpReplace($sScriptContent, "( |, |\x28|\R|\x2C|\x5B)(?<quote>\x22|\x27)(.*?)(\k<quote>)(\R|\x2C|\x26|\x29| |\x3B|\x5D)", '$1" & _ObfConvertString($2$3$4) & "$5') & '"')

ConsoleWrite(@CRLF & $sScriptContent & @CRLF)

Func _ObfConvertString($sfString)
    $sBinaryString = StringToBinary($sfString, 4)
    $sfResult = 'BinaryToString' & '("' & $sBinaryString & '")'
    Return $sfResult
EndFunc

Main part of the above script:

$sScriptContent = Execute('"' & StringRegExpReplace($sScriptContent, "( |, |\x28|\R|\x2C|\x5B)(?<quote>\x22|\x27)(.*?)(\k<quote>)(\R|\x2C|\x26|\x29| |\x3B|\x5D)", '$1" & _ObfConvertString($2$3$4) & "$5') & '"')

 

@mikell: And i agree with you on the delicacies part, working with string regex is like building the Eiffel tower with toothpicks and duct tape. But regex is pure magic to me
Edited by Rnde

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0