Sign in to follow this  
Followers 0
tbaror

limit files list in folder to array?

13 posts in this topic

Hello,

i am trying to write a function that will first check the amount of files exist in a folder and if its bigger for example 100 files will add only the first 100 files to array.

Since the function _FileListToArray i work with could get stack since sometimes i works with folders consist of thousand of files

Please advice

Thanks

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Something like this ?

#Include <File.au3>
#Include <Array.au3>
#Include <Math.au3>

$FileList = _FileListToArray ( @DesktopDir )
_ArrayDisplay ( $FileList, "$FileList" )
$FileList = _ArrayLimitSize ( $FileList, 100 )
_ArrayDisplay ( $FileList, "$FileList" )


Func _ArrayLimitSize ( $_OldArray, $_Size )
    Local $_NewArray[1], $_Start=0, $_Ubound=UBound ( $_OldArray ) -1
    If $_OldArray[0] = $_Ubound Then $_Start = 1
    For $_I = $_Start To _Min ( $_Size, $_Ubound )
        _ArrayAdd ( $_NewArray, $_OldArray[$_I] )
    Next
    $_NewArray[0] = UBound ( $_NewArray ) -1
    Return $_NewArray
EndFunc ;==> _ArrayLimitSize ( )
Edited by wakillon

AutoIt 3.3.14.2 X86 - SciTE 3.6.0WIN 8.1 X64 - Other Example Scripts

Share this post


Link to post
Share on other sites

I did a quick adjustment to _FileListToArray to accept a maximum return count parameter:

Usage is identical to the original, but with one extra paramenter.

; #FUNCTION# ====================================================================================================================
; Name...........: _FileListToArrayEx
; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArrayEx($sPath[, $sFilter = "*"[, $iLimit = 0[,$iFlag = 0]]])
; Parameters ....: $sPath   - Path to generate filelist for.
;               $sFilter - Optional the filter to use, default is *. Search the Autoit3 helpfile for the word "WildCards" For details.
;               $iLimit - Maximum number of files to return. Default is 0, for unlimited.
;               $iFlag   - Optional: specifies whether to return files folders or both
;               |$iFlag=0(Default) Return both files and folders
;               |$iFlag=1 Return files only
;               |$iFlag=2 Return Folders only
; Return values .: @Error - 1 = Path not found or invalid
;               |2 = Invalid $sFilter
;               |3 = Invalid $iFlag
;               |4 = No File(s) Found
; Author ........: SolidSnake <MetalGX91 at GMail dot com>
; Modified.......: Tvern (added maximum return count)
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;                               $array[0] = Number of Files\Folders returned
;                               $array[1] = 1st File\Folder
;                               $array[2] = 2nd File\Folder
;                               $array[3] = 3rd File\Folder
;                               $array[n] = nth File\Folder
; Related .......:
; Link ..........:
; Example .......: No
; Note ..........: Special Thanks to Helge and Layer for help with the $iFlag update speed optimization by code65536, pdaughe
; ===============================================================================================================================
Func _FileListToArrayEx($sPath, $sFilter = "*", $iLimit = 0, $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|", $iCount = 0
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\" ; ensure single trailing backslash
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/:><\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    $hSearch = FileFindFirstFile($sPath & $sFilter)
    If @error Then Return SetError(4, 4, "")
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then ExitLoop
        If ($iFlag + @extended = 2) Then ContinueLoop
        $iCount += 1
        $sFileList &= $sDelim & $sFile
        If $iLimit And $iLimit = $iCount Then ExitLoop
    WEnd
    FileClose($hSearch)
    If Not $sFileList Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc

Share this post


Link to post
Share on other sites

Since the function _FileListToArray i work with could get stack since sometimes i works with folders consist of thousand of files

I don't see that happening anytime soon when run on typical stock PC (lest it's a 128Mb 386SX).

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

I don't see that happening anytime soon when run on typical stock PC (lest it's a 128Mb 386SX).

I was thinking the same thing. Even with a system like you suggested you'd need a pretty large amount of files to cause significant speed/stability issues.

Still, I suppose the OP has his reasons to ask what he did and it was easy enough to make a quick adjustment for the function.

Share this post


Link to post
Share on other sites

I was thinking the same thing. Even with a system like you suggested you'd need a pretty large amount of files to cause significant speed/stability issues.

Still, I suppose the OP has his reasons to ask what he did and it was easy enough to make a quick adjustment for the function.

Hi All and thanks for the function,

The systems i worked with is usually from new Pc to remote storage's shares (Netapp usually) sometimes getting up to 70,000~30,000 files per for folder on Giga network based and those storage's are heavily used for storing HD,DV recording,playback.

Thanks

Share this post


Link to post
Share on other sites

@tbaror

... and 70000 files of max NTFS filename (255 chars) are around 17Mchars, which we will round up (no TM here!) to a whopping 18 * 2 = 36Mb of array storage (worst case).

Knowing that average filenames hopefully tend to be much, much shorter than 255 characters, do you still really think it's such a problem for real world PC, that it would cause memory problems?

Just to clarify, I'd have to agree if you stated that you wanted to minimize time to collect that many filenames from a NAS, but not RAM storage of them.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

@tbaror

... and 70000 files of max NTFS filename (255 chars) are around 17Mchars, which we will round up (no TM here!) to a whopping 18 * 2 = 36Mb of array storage (worst case).

Knowing that average filenames hopefully tend to be much, much shorter than 255 characters, do you still really think it's such a problem for real world PC, that it would cause memory problems?

Just to clarify, I'd have to agree if you stated that you wanted to minimize time to collect that many filenames from a NAS, but not RAM storage of them.

Exactly "minimize time to collect that many file names from a NAS" which seems that the process is stack i wanted to emphasis that by writing its "getting stack".

as for the 70,000 limit i am aware of that there is a script that when files getting to certain amount level its getting divided into sub-folders.

in fact maybe you can help me with related performance script that i am currently writing i need opinion about certain test i am doing which is using the function GetFileSize and measuring the amount of time its took getting the response from remote storage , if threshold lets say 40ms is reached then alert is triggers.

My question would you by chance possibly knows how reliable this function in terms of pure network query spent time or its depends much on local machine cycle time?.

BTW the part of my initial function help asking where to get certain list of files and use it as rotation query so wont be used by storage cash.

Thanks

Edited by tbaror

Share this post


Link to post
Share on other sites

@jchd

I believe the problem isn't the stack, I believe he meant it gets stuck because it's taking so long. I may be mistaken but that's how I read it.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

@BrewManNH

Right

@tbaror

TimerInit and TimerDiff will do pretty useful timing, independant of the hosting machine. In practice such query time may be vastly variable depending on NAS load and that is hard to foresee in enterprise environments.

Maybe your should monitor query response time and if you observe repeatitively slow response, decrease number of files limit accordingly.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

@BrewManNH

Right

@tbaror

TimerInit and TimerDiff will do pretty useful timing, independant of the hosting machine. In practice such query time may be vastly variable depending on NAS load and that is hard to foresee in enterprise environments.

Maybe your should monitor query response time and if you observe repeatitively slow response, decrease number of files limit accordingly.

Thanks for the info ,noted :unsure:

Share this post


Link to post
Share on other sites

Another, possibly simpler way: allow yourself a decent fixed amount of time to perform the task, say 100 files @ 40ms = 4s. Give yourself a safety margin and round this up to 5s.

Launch an AdLibRegister function with 5s timeout, pointing to a function which will raise a STOP_IT flag that your querying loop can poll at every new query, exiting the loop as soon as the timeout is done.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Thanks again great idea i wasn't aware about AdLibRegister surly will use it.

I have more question but now for work. :unsure:

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0