Sign in to follow this  
Followers 0
Terenz

Folder size exclude an ext

22 posts in this topic

Proprably impossible but i'll try to ask.

There is an API, WMI or any other things that give me a folder size but with a filter, so excluding an extension of a file? Like take this folder ( and his subfolder ) give me the size without any .word file.

I know i can search all the file, exclude the .word, loop all the file with FileGetSize and finally sum the total size but if exist a faster way please tell me.

Thank you


Nothing is so strong as gentleness. Nothing is so gentle as real strength

 

Share this post


Link to post
Share on other sites



Afaik, it is not possible to do what you want with a simple command or function.

You have to read the folder recursively and add the size of each file by exluding the extension you want.


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

If your environment is equipped with powershell, you may want to go with a system call to a powershell oneliner like:

(Get-ChildItem c:tmp -recurse | Where-Object {$_.Name -match ".*au3"} | Measure-Object -property length -sum).sum

Put your base directory in the blue part, and any regular expression filter in the argument of the -match switch to filter on any filespec you want. (/edit and if you don't like regex, use any other Get-ChildItem filter, run help Get-ChildItem in powershell for more information, google for specifics.)

It's probably much faster than reading through a directory from within an AutoIt script, but I have not benchmarked it.

Good luck!

Edited by SadBunny

Roses are FF0000, violets are 0000FF... All my base are belong to you.

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

You have an one-liner but you have to understand what's going on behind the scene of PS.

I bet that AutoIt isn't slower than PS in this case.

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

You have an one-liner but you have to understand what's going on behind the scene of PS.

I bet that AutoIt isn't slower than PS in this case.

 

I am a novice powersheller, don't know much about its under-the-hood mechanics... All I know is that I tried this on a directory with 90.000 files (my entire steam library, with regex .*) and it had the sum within a second. /edit: Now I must say that this is a set of triple raid-0 ssd's, so that may help. I did it again and timed it - it's actually 0.92 seconds :) It's good to live in the future.

And whether it's faster or slower - I wrote the oneliner faster and with less effort than writing an autoitscript would have taken me :) Anyway, it's just a suggestion for TS.

Edited by SadBunny

Roses are FF0000, violets are 0000FF... All my base are belong to you.

Share this post


Link to post
Share on other sites

Maybe you can try with my _DirGetSizeByExtension (in my signature), but it will be slower than PS

Local $iSizeAll = DirGetSize("c:\tmp")
Local $iSizeDoc = _DirGetSizeByExtension("c:\tmp", "doc")
Local $iSizeExludeDoc = $iSizeAll - $iSizeDoc

ConsoleWrite($iSizeAll & " - " & $iSizeDoc & " = " & $iSizeExludeDoc)

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

I compared this PS script:

$sw = [Diagnostics.Stopwatch]::StartNew()
(Get-ChildItem "c:\A_DIR\" -recurse | Where-Object {$_.Name -notmatch ".+\.au3"} | Measure-Object -property length -sum).sum
$sw.Stop()
$sw.Elapsed
with this script:

#include <File.au3>

Global $fTimer = TimerInit()
ConsoleWrite("Overall file size: " & FilesGetSize("C:\A_DIR\") &  " bytes / amount of files: "  & @extended & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF)

Func FilesGetSize($sDir, $sExcludeExt = ".au3", $bRecursiv = True)
    Local $hSearch = FileFindFirstFile($sDir & "\*"), $sFullFileName, $sFAttrib
    If $hSearch = -1 Then Return SetError(1, 0, 0)  ;no files found
    Local Static $iFSize = 0, $iIncludedFiles = 0, $iAllFilesFolders = 0, $iFolders = 0
    While 1
        $sFileName = FileFindNextFile($hSearch)
        If @error Then ExitLoop         ; If there is no more file matching the search.
        $sFullFileName = $sDir & "\" & $sFileName
        $sFAttrib = FileGetAttrib($sFullFileName)
        If  StringInStr($sFAttrib, "D") And $bRecursiv Then
            $iFolders += 1
            FilesGetSize($sFullFileName, $sExcludeExt, $bRecursiv)
        EndIf
        If StringRight($sFileName, StringLen($sExcludeExt)) <> $sExcludeExt Then
            $iFSize += FileGetSize($sFullFileName)
            $iIncludedFiles += 1
        EndIf
        ;$iAllFilesFolders += 1
    WEnd
    FileClose($hSearch)
    Return SetExtended($iIncludedFiles - $iFolders , $iFSize)
EndFunc
My test folder: 50573 files and 5285 folder

 

PS: ~10662,5321 ms

AU3: ~9790.24117279014 ms

 

As I suggested both are more or less in the same range.

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

My test folder: 50573 files and 5285 folder

PS: ~10662,5321 ms

AU3: ~9790.24117279014 ms

As I suggested both are more or less in the same range.

 

Sweet :) Didn't expect that. Tried your script, seems the speeds of the PS way are very constant for me at about .90-1.00 while the AU3 fluctuates between .80 and 1.20. Don't know why that is. But yeah, doesn't seem to make a noticeable difference. Guess it really comes down to preference then :) /edit: ah, got it, I was tarballing some very large files on another drive. Yes, .80 vs. .90. Au3 beats PS :)

I used to use AutoIt for all these kinds of things, but I've taken more and more of a liking to powershell for Windows sysadmin-related automation. It's insanely useful once you kind of surrender to it. Takes a lot of googling to get the hang of it, but once you do, in my opinion at least, it runs circles around AutoIt for sysadmin tasks like these in terms of development efficiency.

Edited by SadBunny

Roses are FF0000, violets are 0000FF... All my base are belong to you.

Share this post


Link to post
Share on other sites

Here is my code, using my suggest in #6 :

$sDir = @SystemDir

Global $fTimer = TimerInit()
ConsoleWrite("Overall file size: " & FilesGetSize($sDir, ".exe") &  " bytes / amount of files: "  & @extended & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF & @CRLF)

Global $fTimer = TimerInit()
Local $aSizeAll = DirGetSize($sDir, 1)
Local $aSizeDoc = _DirGetSizeByExtension($sDir, "exe", 1)
ConsoleWrite("Overall file size: " & $aSizeAll[0] - $aSizeDoc[0] &  " bytes / amount of files: "  & $aSizeAll[1] - $aSizeDoc[1] & " / amount of folders: " & $aSizeAll[2] & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF)




Func _DirGetSizeByExtension($sDir, $sExt = "*", $iFlag = 0)
    If NOT FileExists($sDir) Then Return SetError(1, 0, -1)
    If NOT StringInStr(FileGetAttrib($sDir), "D") Then Return SetError(1, 0, -1)

    If $sExt = "*" Then $sExt = "?.*"

    Local $aDirs[1] = [ StringRegExpReplace($sDir, "\\$", "") ]
    Local $iCountDir = 0, $iCountFile = 0, $n = 0
    Local $hSearch, $sFileName
    Local $iSize = 0

    While 1
        $hSearch = FileFindFirstFile( $aDirs[$n]  & "\*.*"  )
        If $hSearch <> -1 Then
            While 1
                $sFileName = FileFindNextFile($hSearch)
                If @error Then ExitLoop

                If @Extended Then
                    $iCountDir += 1
                    If $iCountDir >= UBound($aDirs) Then Redim $aDirs[ UBound($aDirs) * 2]
                    $aDirs[$iCountDir] = $aDirs[$n] & "\" & $sFileName
                Else
                    If StringRegExp($sFileName, "(?i)\." & $sExt & "$") Then
                        $iSize += FileGetSize($aDirs[$n] &  "\" & $sFileName)
                        $iCountFile += 1
                    EndIf
                EndIf
            WEnd
        EndIf
        FileClose($hSearch)

        If $n = $iCountDir Then ExitLoop
        $n += 1
    WEnd

    If NOT $iFlag Then Return $iSize
    Local $aResult[3] = [ $iSize, $iCountFile, $iCountDir ]
    Return $aResult
EndFunc






Func FilesGetSize($sDir, $sExcludeExt = ".au3", $bRecursiv = True)
    Local $hSearch = FileFindFirstFile($sDir & "\*"), $sFullFileName, $sFAttrib
    If $hSearch = -1 Then Return SetError(1, 0, 0)  ;no files found
    Local Static $iFSize = 0, $iCounter = 0
    While 1
        $sFileName = FileFindNextFile($hSearch)
        If @error Then ExitLoop         ; If there is no more file matching the search.
        $sFullFileName = $sDir & "\" & $sFileName
        $sFAttrib = FileGetAttrib($sFullFileName)
        If  StringInStr($sFAttrib, "D") And $bRecursiv Then
            FilesGetSize($sFullFileName, $sExcludeExt, $bRecursiv)
        EndIf
        If StringRight($sFileName, StringLen($sExcludeExt)) <> $sExcludeExt Then
            $iFSize += FileGetSize($sFullFileName)
            $iCounter += 1
        EndIf
    WEnd
    FileClose($hSearch)
    Return SetExtended($iCounter, $iFSize)
EndFunc

PS : 2193,45 ms

UEZ : 720.19 ms

jguinch : 128.77 ms

Both AU3 codes return the same size, but PS returns a diffenrent size...

1 person likes this

Share this post


Link to post
Share on other sites

I made a mistake in the calulcation for the files without the excluded ones -> I updated the code post #7.

Your code is approx. 4 times faster than my recursive version.  :thumbsup:


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

My post of yesterday has disappeard?

jguinch,

Plese you can add to your code an optional parameter for make the search not recursive and stop at parent folder? I don't know for DirGetSize if we can use a paramenter like 2+1 or BitOr or only 1 or 2 but in any case i'm intrested only in the total size

Thanks for the help to all


Nothing is so strong as gentleness. Nothing is so gentle as real strength

 

Share this post


Link to post
Share on other sites

Terenz,

the DB has crashed and the forum was restored but unfortunately the posts from 6am - 12pm (GMT) were lost. -> '?do=embed' frameborder='0' data-embedContent>>


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

Oh, Ok. So maybe other post was missed from this thread, like the answer to my question. Thanks


Nothing is so strong as gentleness. Nothing is so gentle as real strength

 

Share this post


Link to post
Share on other sites

@Terenz : please, try this new version, which allows the same filter possibilities than _FileListToArrayRec. Also, like the original  DirGetSize, you can set the $iflag parameter to 2 or 3 to make the function non-recursive.

Local $aSize = _DirGetSizeEx("c:\tmp", "*.xls;*.doc", 1)

ConsoleWrite("Size :" & $aSize[0] & @CRLF)
ConsoleWrite("Files count :" & $aSize[1]  & @CRLF)
ConsoleWrite("Folders count  :" & $aSize[2] & @CRLF)


; #FUNCTION#====================================================================================================================
; Name ..........: _DirGetSizeEx ; Description ...: Returns the size in bytes of a file list by extension.
; Syntax ........: _DirGetSizeEx($sDir[, $sMask = "*"[, $iFlag = 0]])
; Parameters ....: $sDir                - The directory to search in.
;                  $sMask               - [optional] Filter for results. Default is "*" (all).
;                                         Filter for result. Multiple filters must be separated by ";"
;                                         Use "|" to separate 3 possible sets of filters: "Include|Exclude|Exclude_Folders"
;                                             Include = Files/Folders to include (default = "*" [all])
;                                             Exclude = Files/Folders to exclude (default = "" [none])
;                                             Exclude_Folders = only used if $iRecur = 1 AND $iReturn <> 2 to exclude defined folders (default = "" [none])
;                  $iFlag               - [optional] 0 (default) = Returns the size
;                                                    1 = Extended mode is On -> returns an array that contains extended information (see Remarks).
;                                                    2 = Don't get the size of files in subdirectories (recursive mode is Off) ; Return values .: Success = The size in bytes, or a single dimension array
;                  Failure = -1 and sets the @error flag to 1 if the path doesn't exist.
; Author ........: jguinch
; Remarks .......: If you use the extended mode then the array returned from this function is a single dimension array containing the followingelements:
;                    $aArray[0] = Size
;                    $aArray[1] = Files count
;                    $aArray[2] = Dirs Count
;===============================================================================================================================
Func _DirGetSizeEx($sDir, $sMask = "*", $iFlag = 0); OK
    If NOT FileExists($sDir) Then Return SetError(1, 0, -1)
    If NOT StringInStr(FileGetAttrib($sDir), "D") Then Return SetError(1, 0, -1)

    Local $iExtMode = BitAND($iFlag, 1) > 0
    Local $iRecMode = NOT BitAND($iFlag, 2) > 0
    Local $aDirs[1] = [ StringRegExpReplace($sDir, "\\$", "") ]
    Local $iCountDir = 0, $iCountFile = 0, $n = 0, $iSize = 0
    Local $hSearch, $sFileName, $sRegexFilesInclude, $sRegexFilesExclude = "^$", $sRegexFoldersExclude = "^$"

    Local $sRegexMask = StringReplace( StringReplace( StringReplace($sMask, "|", "\|") , "?", "\E(?:.|.?$)\Q"), "*", "\E.*?\Q")
    Local $aFilters = StringSplit($sRegexMask, "\|", 3)

    $sRegexFilesInclude = "(?i)^(?:" & StringRegExpReplace(StringReplace($aFilters[0], ";", "|") , "([^|]+)", "\\Q$1\\E") & ")$"
    If UBound($aFilters) > 1 Then $sRegexFilesExclude = "(?i)^(?:" & StringRegExpReplace(StringReplace($aFilters[1], ";", "|") , "([^|]+)", "\\Q$1\\E") & ")$"
    If UBound($aFilters) > 2 Then $sRegexFoldersExclude = "(?i)^(?:" & StringRegExpReplace(StringReplace($aFilters[2], ";", "|") , "([^|]+)", "\\Q$1\\E") & ")$"

    While 1
        $hSearch = FileFindFirstFile( $aDirs[$n]  & "\*.*"  )
        If $hSearch <> -1 Then

            While 1
                $sFileName = FileFindNextFile($hSearch)
                If @error Then ExitLoop

                If @Extended Then
                    If NOT StringRegExp($sFileName, $sRegexFoldersExclude) Then
                        $iCountDir += 1
                        If $iCountDir >= UBound($aDirs) Then Redim $aDirs[UBound($aDirs) * 2]
                        $aDirs[$iCountDir] = $aDirs[$n] & "\" & $sFileName
                    EndIf
                Else
                    If StringRegExp($sFileName, $sRegexFilesInclude) AND NOT StringRegExp($sFileName, $sRegexFilesExclude) Then
                        $iSize += FileGetSize($aDirs[$n] &  "\" &$sFileName)
                        $iCountFile += 1
                    EndIf
                EndIf
            WEnd

            If NOT $iRecMode Then ExitLoop
        EndIf

        FileClose($hSearch)

        If $n = $iCountDir Then ExitLoop
        $n += 1
    WEnd

    If NOT $iExtMode Then Return $iSize

    Local $aResult[3] = [ $iSize, $iCountFile, $iCountDir ]
    Return $aResult
EndFunc

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

Is cool but seems slow of almost double times respect of the script of the post #9

193ms vs 383 ms approx. Same configuration, recursive and same folder. Two separate script lauched many times, the first version was faster in any test

Do you think can you speed up a bit? As i said ( with all due respect to your code ) i need something faster as possible. Thanks

Edited by Terenz

Nothing is so strong as gentleness. Nothing is so gentle as real strength

 

Share this post


Link to post
Share on other sites

I spend to last 2 hours to try to optimize my code, but no, I cannot make a really faster version, sorry... I cannot suggest you a best way than my first one.

BTW, my >_DirGetSizeEx could be useful for someone else, so I will work on it with enough tests before posting it in the Examples section

Thanks for you return.

Share this post


Link to post
Share on other sites

#17 ·  Posted (edited)

I'm sure it will be useful for many people. I think all that RegExp slow down that function, in my case i'll use your previus version, if no other one has a fastest way. Thanks to you.

Edited by Terenz

Nothing is so strong as gentleness. Nothing is so gentle as real strength

 

Share this post


Link to post
Share on other sites

I made some changes in my code to have just one regex execution by loop (fusion of two regex into a big one). The result is the same...

If you want to see the result :

; #FUNCTION#====================================================================================================================
; Name ..........: _DirGetSizeEx
; Description ...: Returns the size in bytes of a file list by extension.
; Syntax ........: _DirGetSizeEx($sDir[, $sMask = "*"[, $iFlag = 0]])
; Parameters ....: $sDir                - The directory to search in.
;                  $sMask               - [optional] Filter for results. Default is "*" (all).
;                                         Filter for result. Multiple filters must be separated by ";"
;                                         Use "|" to separate 3 possible sets of filters: "Include|Exclude|Exclude_Folders"
;                                             Include = Files/Folders to include (default = "*" [all])
;                                             Exclude = Files/Folders to exclude (default = "" [none])
;                                             Exclude_Folders = only used if $iRecur = 1 AND $iReturn <> 2 to exclude defined folders (default = "" [none])
;                  $iFlag               - [optional] 0 (default) = Returns the size
;                                                    1 = Extended mode is On -> returns an array that contains extended information (see Remarks).
;                                                    2 = Don't get the size of files in subdirectories (recursive mode is Off) ; Return values .: Success = The size in bytes, or a single dimension array
;                  Failure = -1 and sets the @error flag to 1 if the path doesn't exist.
; Author ........: jguinch
; Remarks .......: If you use the extended mode then the array returned from this function is a single dimension array containing the followingelements:
;                    $aArray[0] = Size
;                    $aArray[1] = Files count
;                    $aArray[2] = Dirs Count
;===============================================================================================================================
Func _DirGetSizeEx($sDir, $sMask = "*", $iFlag = 0); OK
    If NOT FileExists($sDir) Then Return SetError(1, 0, -1)
    If NOT StringInStr(FileGetAttrib($sDir), "D") Then Return SetError(1, 0, -1)

    Local $iExtMode = 0, $iRecMode = 1
    Local $aDirs[1] = [ StringRegExpReplace($sDir, "\\$", "") ]
    Local $iCountDir = 0, $iCountFile = 0, $n = 0, $iSize = 0, $iFolderFilter = 0
    Local $sRegexFilesExclude = "()", $sRegexFoldersExclude = "^$"
    Local $hSearch, $sFileName, $sRegexMask, $iAddFolder, $aRes, $iUBound
    Local $aFilters = StringSplit(  StringReplace( StringReplace( StringReplace($sMask, "|", "\|") , "?", "\E(?:.|.?$)\Q"), "*", "\E.*?\Q"), "\|", 3)
    Local $sRegexFilesInclude = "(?|(\Q" & StringReplace($aFilters[0], ";", "\E)|(\Q") & "\E))$"
    
    If BitAND($iFlag, 1) Then $iExtMode = 1
    If BitAND($iFlag, 2) Then $iRecMode = 0

    If UBound($aFilters) > 1 Then $sRegexFilesExclude = "(?!(?|(\Q" & StringReplace($aFilters[1], ";","\E)|(\Q") & "\E))$)"
    $sRegexMask = "(?i)^" & $sRegexFilesExclude & $sRegexFilesInclude
    ConsoleWrite($sRegexMask & @CRLF)

    If UBound($aFilters) > 2 Then
        $iFolderFilter = 1
        $sRegexFoldersExclude = "(?i)^(?:\Q" & StringReplace($aFilters[2], ";", "\E|\Q") & "\E)$"
    EndIf

    While 1
        $hSearch = FileFindFirstFile( $aDirs[$n]  & "\*.*"  )
        If $hSearch <> -1 Then

            While 1
                $sFileName = FileFindNextFile($hSearch)
                If @error Then ExitLoop

                If @Extended Then
                    $iAddFolder = 1
                    If $iFolderFilter AND StringRegExp($sFileName, $sRegexFoldersExclude) Then $iAddFolder = 0
                    
                    If $iAddFolder Then 
                        $iUBound = UBound($aDirs)
                        $iCountDir += 1
                        If $iCountDir >= $iUBound Then Redim $aDirs[$iUBound * 2]
                        $aDirs[$iCountDir] = $aDirs[$n] & "\" & $sFileName
                    EndIf
                ElseIf StringRegExp($sFileName, $sRegexMask) Then
                    $iSize += FileGetSize($aDirs[$n] &  "\" & $sFileName)
                    $iCountFile += 1
                EndIf
            WEnd

            If NOT $iRecMode Then ExitLoop
        EndIf

        FileClose($hSearch)

        If $n = $iCountDir Then ExitLoop
        $n += 1
    WEnd

    If NOT $iExtMode Then Return $iSize

    Local $aResult[3] = [ $iSize, $iCountFile, $iCountDir ]
    Return $aResult
EndFunc

Share this post


Link to post
Share on other sites

Seems not work:

$sDir = @SystemDir

$fTimer = TimerInit()
$aSize = _DirGetSizeEx($sDir, "exe", 1)
ConsoleWrite(Round(($aSize[0]) / 1048576, 2) & " MB / amount of files: " & $aSize[1] & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF)

$fTimer = TimerInit()
$aSize = _DirGetSizeEx($sDir, "*", 1)
ConsoleWrite(Round(($aSize[0]) / 1048576, 2) & " MB / amount of files: " & $aSize[1] & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF)
0 MB / amount of files: 0
148.524742669809 ms

906.08 MB / amount of files: 5471
278.603844902076 ms

Don't found any exe file? For all file instead give the correct result


Nothing is so strong as gentleness. Nothing is so gentle as real strength

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0