Sign in to follow this  
Followers 0
guinness

Regular expression challenge for stripping single comments.

44 posts in this topic

#1 ·  Posted (edited)

My aim is to improve stripping single line comments in a script and even though the regular expression works, when stripping a large file (such as AutoItWrapper) it crashes AutoIt.

So SRE gurus, if you like a challenge this one is for you.

#include <Constants.au3> ; Should be MsgBoxConstants for all those rocking v3.3.10.0.

; Example()

; New line comment.
Func Example() ; This is some comment after a function.
    Local $sText = "; This is a commment as a string and shouldn't be removed." ; This is a comment to explain the; string. Woah! inception.
EndFunc   ;==>Example

; How I currently strip single line comments.
Local $sData = FileRead(@ScriptFullPath)
_StripCommentLines($sData)
MsgBox($MB_SYSTEMMODAL, 'Comment stripping', $sData)


Func _StripCommentLines(ByRef $sData)
    $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"]).*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. http://www.autoitscript.com/forum/topic/108728-stripcomments/#entry765712
EndFunc   ;==>_StripCommentLines
Edited by guinness

_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites



I don't have time to look closely but I strongly suspect the issue is the heavy backtracking implied by the pattern.

For me, latest AutoIt3Wrapper.au3 doesn't crash with either regexp:

$sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"]).*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. http://www.autoitscript.com/forum/topic/108728-stripcomments/#entry765712
    $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"])[^\r\n]*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. http://www.autoitscript.com/forum/topic/108728-stripcomments/#entry765712
    $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"])\N*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. http://www.autoitscript.com/forum/topic/108728-stripcomments/#entry765712

Win7 x64, run in x86. It remains to check that the output is correct.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Even with a regex shortened a bit, this makes no significative difference (on the last release 3.3.10.0)

Local $sData = FileRead("G:\Program Files\AutoIt3\Include\GDIPlus.au3")
_StripCommentLines($sData)
FileWrite("test.txt", $sData)

Func _StripCommentLines(ByRef $sData)
      $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"]).*?\2)*[^;]*);?.*$', '$1')
;~   $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"]).*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. 
EndFunc   ;==>_StripCommentLine

Share this post


Link to post
Share on other sites

jchd,

Thanks. I am not getting any crash now, but will look at your amendments and discuss with wraithdu who seems to be getting the crashes. I don't suck at regular expressions that much as evident in the MVP section.

mikell,

Cheers for your amendments and time as well. I see you changed the last non-capturing group.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

Yes, for readability only...

Grouping this part is not necessary and with (?m) the dot can be used as it doesn't match newlines

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

Hi guinness, in 99.9% of cases is always the fault of Pattern, will have to be so ("K" is really nice and helpful ;) )

#include <Constants.au3> ; Should be MsgBoxConstants for all those rocking v3.3.10.0.

; Example()

; New line comment.
Func Example() ; This is some comment after a function.
    Local $sText = "; This is a commment as a string and shouldn't be removed." ; This is a comment to explain the; string. Woah! inception.
EndFunc   ;==>Example

; How I currently strip single line comments.
Local $sData = FileRead(@ScriptFullPath)
_StripCommentLines($sData)
MsgBox($MB_SYSTEMMODAL, 'Comment stripping', $sData)


Func _StripCommentLines(ByRef $sData)
    ;;$sData = StringRegExpReplace($sData, '(?m)^(?:[^;"' & "']|'[^']*'|" & '"[^"]*")*\K;.*', "") ;; Pattern (?m)^(?:[^;"']|'[^']*'|"[^"]*")*\K;.*
    $sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', "")  ;; Pattern (?m)^[^;"']*(?:[^;"']|'[^']*'|"[^"]*")*\K;.*
EndFunc   ;==>_StripCommentLine



#cs    ;ConsoleWrite($sData & @LF)
 #include <Constants.au3>




Func Example()
    Local $sText = "; This is a commment as a string and shouldn't be removed."
EndFunc   


Local $sData = FileRead(@ScriptFullPath)
_StripCommentLines($sData)
ConsoleWrite($sData & @LF)


Func _StripCommentLines(ByRef $sData)
    $sData = StringRegExpReplace($sData, '(?m)^(?:[^;"' & "']|'[^']*'|" & '"[^"]*")*\K;.*', "")
EndFunc   
#CE
#include <Constants.au3>

Local $sData = FileRead(@DesktopDir & "\test.au3") ;55 MB, 1.122.000 Line
local $fTimerDiff =TimerInit()
_StripCommentLines($sData)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @LF)


Func _StripCommentLines(ByRef $sData)
    $sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', "")
EndFunc   ;==>_StripCommentLine

#cs
 >"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\DXRW4E\Desktop\New AutoIt v3 Script (2).au3" /UserParams    
+>14:42:18 Starting AutoIt3Wrapper v.2.1.3.3 SciTE v.3.3.7.0 ;  Keyboard:00000409  OS:WIN_8/  CPU:X64 OS:X64    Environment(Language:0409  Keyboard:00000409  OS:WIN_8/  CPU:X64 OS:X64)
>Running AU3Check (3.3.10.0)  from:C:\Program Files (x86)\AutoIt3
+>14:42:18 AU3Check ended.rc:0
>Running:(3.3.10.0):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\DXRW4E\Desktop\New AutoIt v3 Script (2).au3"    
--> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
1751.5439052816
+>14:42:22 AutoIt3.exe ended.rc:0
>Exit code: 0    Time: 4.351
#CE

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

Nice !  and no backreferencing providing max lightness   :)

$sData = StringRegExpReplace($sData, '(?m)^(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', "")

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Hi mikell, how to say is the ABC of RegEx & match string in quotes '(?:[^"]|"[^"]")' servesmust (for more savecompatibility) to matchcheck all the string, not start directly from '"' ect ect

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

If you guys have time could you explain in more detail or break the patterns down for us light-weights?  Especially why DWRW4E's pattern is better/different.

Thanks and Merry Christmas!

kylomas

edit:spelling

Edited by kylomas

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites

#10 ·  Posted (edited)

as mentioned in the post above, first use the correct way to readcheck string in quotes (so match correctly ";" or "," ect ect), in this case use of "K" tired much less the regex, as you know Regex lose time when capture Group ect ect, instead with the "K" all is more simple regex does not need to capture that group so normally uses less memory and CPU ect ect, and also more faster, so need to do everything to reduce the options in pattern, because there are the limits of the pointer or do not know in windows hmmmmm, so when there are many MB of data to be check, regex in complicated patterns performingchecking thousands and thousands of options uses much more memory and cpu and is slower ect ect and  passes certain limits of windows for this goes to error ect ect, however does not exist basic rules regard to the patterns, all depends on the situation from what you want to do

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

Oh wow DXRW4E. I have to have a day for what you just wrote to sink in. Thanks for that. I will probably PM you in the next couple of days if that's OK?


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

OK, I have isolated the crashing I was talking about.

; Compile AutoIt3Wrapper.au3 and the run the regular expression against the Obfuscated code.
Local $sData = FileRead("C:\Program Files\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper_Obfuscated.au3")
$sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', '')
Exit

This is something I can't work my head around.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

I will probably PM you in the next couple of days if that's OK?

OK

 

OK, I have isolated the crashing I was talking about.

; Compile AutoIt3Wrapper.au3 and the run the regular expression against the Obfuscated code.
Local $sData = FileRead("C:\Program Files\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper_Obfuscated.au3")
$sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', '')
Exit
This is something I can't work my head around.

in my script i personally never use the (?m) , I always use the 'n' or any point of reference, so everything is more faster ect ect and much less stressful for regex

 

this will have to be Ok

#include <array.au3>
; Compile AutoIt3Wrapper.au3 and the run the regular expression against the Obfuscated code.
Local $sData = FileRead("C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper_Obfuscated.au3")
;;$sData = $aFileData = StringRegExpReplace(@LF & $sData, '\r\n?', @LF)
$sData = StringRegExpReplace(@LF & $sData, '\r(?!\n)', @CRLF)
;$sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*?', '')
$sData = StringTrimLeft(StringRegExpReplace($sData, '\n[^;"''\r\n]*(?:[^;"''\r\n]|''[^''\r\n]*''|"[^"\r\n]*")*\K;[^\r\n]*', ''), 1)
Exit
or ignore #Region comments

#include <Constants.au3> ; Should be MsgBoxConstants for all those rocking v3.3.10.0.

#Region ;aaaaaaa
;bbbbbbbbbbbbbbbb
#EndRegion ;ccccccccccc

; Example()

; New line comment.
Func Example() ; This is some comment after a function.
    Local $sText = "; This is a commment as a string and shouldn't be removed." ; This is a comment to explain the; string. Woah! inception.
EndFunc   ;==>Example

; How I currently strip single line comments.
Local $sData = StringRegExpReplace(FileRead(@ScriptFullPath), '\r(?!\n)', @CRLF)
_StripCommentLines($sData)
MsgBox($MB_SYSTEMMODAL, 'Comment stripping', $sData)


Func _StripCommentLines(ByRef $sData)
    ;; ignore #Region comments
    $sData = StringTrimLeft(StringRegExpReplace(@LF & $sData, '\n\h*(?:#(?:EndRegion|Region)(*SKIP)[^\r\n]*$\f)?[^;"''\r\n]*(?:[^;"''\r\n]|''[^''\r\n]*''|"[^"\r\n]*")*\K;[^\r\n]*', ''), 1)
EndFunc   ;==>_StripCommentLines
Ciao. Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

Ah, awesome. Thanks for your help.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

About your update with the regions, I will keep that in my notes, but I am stripping #EndRegion and #Region in my scripts as well.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

everything was to show the usefulness of (*SKIP) is very useful in certain situations, only move offset in the end of the line, so in large files will not lose anything in performance ect ect, really useful

Ciao.


apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

That's something new to me as well as the usage of K. I know I have a lot left to learn with these things. Thanks for your time.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

example (I was writing an INF function, and there really is a challenge, a really big mess) translate StrKey in Inf file, so %% = % And %StrKey2% ([strings] section, StrKey2="ect ect ect")
 

#include <Constants.au3>

$sSectionData = 'aaa%bbb%ccc%ddd%%eee%fff%%ggg'
$sSectionData = StringRegExpReplace($sSectionData, '\r\n?', @LF) ;;@CR serves to use as a delimiter ect ect
$sSectionData = StringRegExpReplace($sSectionData, "%(?:%|[^\n%]+%(*SKIP)$\f)", "%" & @CR) ;;I need "%" & @CR, because during untraslate - StringRegExpReplace($sSectionData, "%\r", "%%") 
;;$aStrKey = StringRegExp($sSectionData, "%([^\r\n%]+)%", 3)
;;ect ect ect
MsgBox($MB_SYSTEMMODAL, 'Comment stripping', $sSectionData)

I wanted to say that options such as (*SKIP) they are really useful

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

#19 ·  Posted (edited)

Hi guinness, another useful thing that will have to get your attention is the (DEFINE) ( how to say byref func or recursive func), example (is not 100% Ok, but so to give an idea)

#include <array.au3>
Global Const $sRefVDSubPattern = '(?(DEFINE)(?<VD>' & '\h*\w+\((?:[^;\(\)"''\r\n]|''[^''\r\n]*''|"[^"\r\n]*"|(?&VD)|\h*\_\h*(?:;[^\r\n]*)*\r?\n\h*)*\)\h*' & '))'

Local $aTest, $sTest = "$Var = StringRegExpReplace($ScriptFullPath & FileRead(@ScriptFullPath), $ScriptFullPath, @CRLF), $Var2 = 'aaaaaaaa', $Var3, $Var4 = FuncName(FuncName(FuncName($Var3, 'aaaa'))), $Var5"


$aTest = StringRegExp($sTest, $sRefVDSubPattern & '(\$\w+)(?>\h*=\h*(?:''[^''\r\n]*''|"[^"\r\n]*")+|\h*=(?&VD)|\h+\_\h*(?:;[^\r\n]*)*\r?\n\h*)*\h*', 3)
;;$aTest = StringRegExp($sTest, $sRefVDSubPattern & '(\$\w+(?>\h*=\h*(?:''[^''\r\n]*''|"[^"\r\n]*")+|\h*=(?&VD)|\h+\_\h*(?:;[^\r\n]*)*\r?\n\h*)*\h*)', 3)

_ArrayDisplay($aTest)

so you can calluse the '(?&VD)' where you want and when you want ect ect, very very useful in complicated situations

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

DXRW4E (An aside, off topic, FYI)

I could not help being distracted by your constant mentioning of electroconvulsive therapy (ECT), formerly known as electroshock. I realise you must be refering to "et cetera, et cetera, et cetera", which was made famous from the movie, "The King and I", and from the book, "Anna and the King of Siam".

An accepted abbreviation of "et cetera" is "etc." or "&c.", where "et" means "and", and "cetera" means "the rest", or "other things".   Normally one "et cetera" is all that is needed to convey "and the rest".   And, "etc." normally follows a coma, &c.


salut, adios, aloha, etc.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0