SubinACL - In need of Speed!

[Ghost21]

Well here is the issue I have written a conversion tool for SubinACL. Essentially what I'm doing is exporting all them groups/Members then reading the export file doing some filtering etc. and writing the file back to the clean file. Now in doing this some of the file exports are 1.5GB in size and take many hours to run. Can someone please look at the code and see if you can find anything glaring that I could clean up to improve performance.

 

expandcollapse popup

#include <FileConstants.au3>
#include <MsgBoxConstants.au3>
#include <Array.au3>
#include <File.au3>
#include <String.au3>

Global $LS = InputBox("Local Server Name", "What server are you migrating from local groups:" & @CRLF & "eg. contosofilesvr", "")
If $LS = "" Then Exit

Global $Domain = InputBox("Domain Name", "What domain are you migrating permissions to:" & @CRLF & "eg. contoso", "")
If $Domain = "" Then Exit

Global $SEF = FileOpenDialog("SubinACL Export File", @ScriptDir, "Text files (*.txt)", "", "")

If FileExists($SEF) Then
    MsgBox(0, "SubinACL Export File.", "Confirmed. " & $SEF)
Else
    MsgBox(0, "SubinACL Export File is Missing or not created.", "Please correct this issue. Exiting")
    Exit
EndIf

Global $SIF = FileOpenDialog("Subinacl Import File", @ScriptDir, "Text files (*.txt)", "", "")

If FileExists($SIF) Then
    MsgBox(0, "SubinACL Import File.", "Confirmed. " & $SIF)
Else
    MsgBox(0, "SubinACL Import File is missing or not created.", "Please correct this issue. Exiting")
    Exit
EndIf

Global Const $MF = @ScriptDir & "\" & "Members.csv"

If FileExists($MF) Then
    MsgBox(0, "Local Groups Export File.", "Confirmed. " & $MF)
Else
    MsgBox(0, "Local Groups Export File is missing or not created.", "Please correct this issue. Exiting")
    Exit
EndIf

$timer = TimerInit()

FindACL()

ConsoleWrite(TimerDiff($timer) & @CRLF)

Func FindACL()
    $file = FileOpen($SEF, 512)
    While 1
        $line = FileReadLine($file)
        If @error = -1 Then ExitLoop
        $SplitString = StringSplit($line, " ")
        For $i = 1 To $SplitString[0]
            $SearchString = StringSplit($SplitString[$i], "\")
            For $k = 1 To $SearchString[0]
                If $SearchString[$k] = "=" & $LS Then
                    $lgss = $SearchString[$k + 1]
                    LocalGroups($lgss, $line)
                EndIf
            Next
        Next
        If StringLeft($line, 9) <> "/pace =S-" Then
            If StringLeft($line, 15) <> "/pace =everyone" Then
                If StringLeft($line, 20) <> "/pace =builtin\users" Then
                    If StringLeft($line, 16) <> "/pace =" & $LS Then
                        WriteSameData($line)
                    EndIf
                EndIf
            EndIf
        EndIf
    WEnd
    FileClose($file)
EndFunc   ;==>FindACL

Func LocalGroups($lgss, $line)
    $aInput = ""
    _FileReadToArray($MF, $aInput)
    For $i = 2 To UBound($aInput) - 1
        $output = StringSplit($aInput[$i], ",")
        $LG = StringTrimRight(StringTrimLeft($output[2], 1), 1)
        If $LG = $lgss Then
            $M = StringTrimRight(StringTrimLeft($output[3], 1), 1)
            $M = StringSplit($M, ";")
            For $C = 1 To $M[0]
                $FM = $M[$C]
                If $FM <> "" Then
                    If StringLeft($FM, 9) <> "S-1-5-21-" Then
                        $newdata = StringReplace($line, "=" & $LS & "\" & $lgss, "=" & $Domain & "\" & $FM)
                        WriteData($newdata)
                    EndIf
                EndIf
            Next
        EndIf
    Next
EndFunc   ;==>LocalGroups

Func WriteData($newdata)
    FileWriteLine($SIF, $newdata)
EndFunc   ;==>WriteData

Func WriteSameData($line)
    FileWriteLine($SIF, $line)
EndFunc   ;==>WriteSameData

 

[water]

There is nothing obvious to greatly improve execution speed.
As functions WriteData and WriteSameData only contain a single line of code I would replace this function calls with teh FileWriteLine statement to drop th overhead of calling those functions many, many times.
Another idea: Use a file handle for the output file as well. Now you open and close $SIF every time you write a single line to this file.

[jchd]

You also reread and reparse the members file ($MF) at every step. You should parse it only once and use a more suitable data structure to find matches to $lgss. I'd recommend using a Map datatype (use the beta to make them available) of $LG entries where you create another Map containing $M values.

Can you post limited samples of anonymized inputs so I can hopefully provide an example code? Disclaimer: I've only little idea of the input format but I strongly suspect you can greatly improve run time.

Edited May 21, 2017 by jchd

[Ghost21]

Thank you both for the insight. In changing the $SIF to open the file and append I went from 3hrs of run time on a 200MB file to 2.75mins of run time.. I appreciate all the help.

[water]

To further enhance execution speed I would (as jchd already suggested) as a first and easiest step move the _FileReadToArray($MF) statement to the top of your script.
Now you read the whole file every time you call function LocalGroups.

[Ghost21]

Water and JCHD,

With both of your recommendations has resulted in a 200MB file to be completed in under 2mins..

This will be a great time saver of like weeks of manual labor.

Edited May 22, 2017 by Ghost21