Ghost21

SubinACL - In need of Speed!

6 posts in this topic

#1 ·  Posted

Well here is the issue I have written a conversion tool for SubinACL. Essentially what I'm doing is exporting all them groups/Members then reading the export file doing some filtering etc. and writing the file back to the clean file. Now in doing this some of the file exports are 1.5GB in size and take many hours to run. Can someone please look at the code and see if you can find anything glaring that I could clean up to improve performance.

 

#include <FileConstants.au3>
#include <MsgBoxConstants.au3>
#include <Array.au3>
#include <File.au3>
#include <String.au3>

Global $LS = InputBox("Local Server Name", "What server are you migrating from local groups:" & @CRLF & "eg. contosofilesvr", "")
If $LS = "" Then Exit

Global $Domain = InputBox("Domain Name", "What domain are you migrating permissions to:" & @CRLF & "eg. contoso", "")
If $Domain = "" Then Exit

Global $SEF = FileOpenDialog("SubinACL Export File", @ScriptDir, "Text files (*.txt)", "", "")

If FileExists($SEF) Then
    MsgBox(0, "SubinACL Export File.", "Confirmed. " & $SEF)
Else
    MsgBox(0, "SubinACL Export File is Missing or not created.", "Please correct this issue. Exiting")
    Exit
EndIf

Global $SIF = FileOpenDialog("Subinacl Import File", @ScriptDir, "Text files (*.txt)", "", "")

If FileExists($SIF) Then
    MsgBox(0, "SubinACL Import File.", "Confirmed. " & $SIF)
Else
    MsgBox(0, "SubinACL Import File is missing or not created.", "Please correct this issue. Exiting")
    Exit
EndIf

Global Const $MF = @ScriptDir & "\" & "Members.csv"

If FileExists($MF) Then
    MsgBox(0, "Local Groups Export File.", "Confirmed. " & $MF)
Else
    MsgBox(0, "Local Groups Export File is missing or not created.", "Please correct this issue. Exiting")
    Exit
EndIf

$timer = TimerInit()

FindACL()

ConsoleWrite(TimerDiff($timer) & @CRLF)

Func FindACL()
    $file = FileOpen($SEF, 512)
    While 1
        $line = FileReadLine($file)
        If @error = -1 Then ExitLoop
        $SplitString = StringSplit($line, " ")
        For $i = 1 To $SplitString[0]
            $SearchString = StringSplit($SplitString[$i], "\")
            For $k = 1 To $SearchString[0]
                If $SearchString[$k] = "=" & $LS Then
                    $lgss = $SearchString[$k + 1]
                    LocalGroups($lgss, $line)
                EndIf
            Next
        Next
        If StringLeft($line, 9) <> "/pace =S-" Then
            If StringLeft($line, 15) <> "/pace =everyone" Then
                If StringLeft($line, 20) <> "/pace =builtin\users" Then
                    If StringLeft($line, 16) <> "/pace =" & $LS Then
                        WriteSameData($line)
                    EndIf
                EndIf
            EndIf
        EndIf
    WEnd
    FileClose($file)
EndFunc   ;==>FindACL

Func LocalGroups($lgss, $line)
    $aInput = ""
    _FileReadToArray($MF, $aInput)
    For $i = 2 To UBound($aInput) - 1
        $output = StringSplit($aInput[$i], ",")
        $LG = StringTrimRight(StringTrimLeft($output[2], 1), 1)
        If $LG = $lgss Then
            $M = StringTrimRight(StringTrimLeft($output[3], 1), 1)
            $M = StringSplit($M, ";")
            For $C = 1 To $M[0]
                $FM = $M[$C]
                If $FM <> "" Then
                    If StringLeft($FM, 9) <> "S-1-5-21-" Then
                        $newdata = StringReplace($line, "=" & $LS & "\" & $lgss, "=" & $Domain & "\" & $FM)
                        WriteData($newdata)
                    EndIf
                EndIf
            Next
        EndIf
    Next
EndFunc   ;==>LocalGroups

Func WriteData($newdata)
    FileWriteLine($SIF, $newdata)
EndFunc   ;==>WriteData

Func WriteSameData($line)
    FileWriteLine($SIF, $line)
EndFunc   ;==>WriteSameData

 

Share this post


Link to post
Share on other sites



#2 ·  Posted

There is nothing obvious to greatly improve execution speed.
As functions WriteData and WriteSameData only contain a single line of code I would replace this function calls with teh FileWriteLine statement to drop th overhead of calling those functions many, many times.
Another idea: Use a file handle for the output file as well. Now you open and close $SIF every time you write a single line to this file.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

You also reread and reparse the members file ($MF) at every step. You should parse it only once and use a more suitable data structure to find matches to $lgss. I'd recommend using a Map datatype (use the beta to make them available) of $LG entries where you create another Map containing $M values.

Can you post limited samples of anonymized inputs so I can hopefully provide an example code? Disclaimer: I've only little idea of the input format but I strongly suspect you can greatly improve run time.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#4 ·  Posted

Thank you both for the insight. In changing the $SIF to open the file and append I went from 3hrs of run time on a 200MB file to 2.75mins of run time.. I appreciate all the help.

Share this post


Link to post
Share on other sites

#5 ·  Posted

To further enhance execution speed I would (as jchd already suggested) as a first and easiest step move the _FileReadToArray($MF) statement to the top of your script.
Now you read the whole file every time you call function LocalGroups.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

Water and JCHD,

With both of your recommendations has resulted in a 200MB file to be completed in under 2mins..

This will be a great time saver of like weeks of manual labor.

Edited by Ghost21

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now