Beta _FileListToArrayRec - added functionalities

[Melba23]

Hi,

This recent thread brought up the subject of allowing user regexes within _FileListToArrayRec. guinness then suggested using a callback function so that users could do even more specific selection. As a result I have developed a Beta version of the UDF adding such functionalities - and also added the option to return a 2D array when using a custom function, something that has been a frequent request in the past.

The main changes to the UDF code are as follows:

 

- 1. The UDF will now accept a user-defined regex in any of the 3 sections of the $sMask parameter - such a regex must be enclosed in double colons ( ::regex:: ) to distinguish it from a normal path/wildcard string. No error checking is done on the regex itself - the user must ensure that it is correct.

- 2. The main UDF will now accept a function in place of the $sMask parameter and internally has separate sections for running this "user function" and "normal" code to avoid any slowdown when used without a user function - combining the two meant that at least one If had to be added to each item parsed, which increased the execution time by about 5%. Note that when used with a user function, the $sMask parameter is forced to "*" (all items on the path with no exclusions) and the $iReturn parameter only defines whether files and/or folders are to be returned (the user function must deal with all other restrictions). The $iRecur parameter still works as before, but the $iSort & $iReturnPath parameters are ignored (the former because it can only work on 1D arrays (see point 4 below), while the user function must determine the latter). This was done to ensure that the internal UDF code runs as fast as possible because calling the user function for each returned item can obviously add significantly to the overall execution time.

- 3. The user function must accept 3, and only 3, parameters (the full path, name and attributes of the item) and must return the data to be included in the final returned array - if nothing is to be added, then the @error macro must be set. Otherwise there is no restriction on the user function itself - although as it is run on every item on the path, it should be as fast as possible.

- 4. If a user function is passed to the UDF, an added UDF parameter ($iUserRet_Cols) comes into play. This determines the number of columns in the returned array - the UDF defaults to a 1D array but setting the parameter to a higher value will return a 2D array with that number of columns. Obviously the user function must return suitably formatted data to be added to the return array - a simple string if the return array is 1D; an array with sufficent elements to fill the columns of a 2D return array. Note that if the number of elements in the array passed by the user function does not match the defined return array, then nothing is added.

As a result of all thesse changes, there is almost no change to the execution speed of the UDF in "normal" mode, the "direct regex" mode can make the return list more accurate, and there is now a very flexible "user function" option allowing experienced users to fine-tune the return and add further data.

Here is the Beta function with a short example included:

FLTAR_SRE_and_Func.au3

Comments on the utilty and implementation of the new functionalities would be most welcome.

M23

Edit: Now with regex syntax checking.

Edited February 9, 2015 by Melba23

[mLipok]

As a base you used this function from 3.3.13.19 ? or not relased yet 3.3.13.20 ? (i do not remember if there was any changes in 3.3.13.20)

I want to compare changes with WinMerge.

edit:

... for learning and for testing.

Edited January 29, 2015 by mLipok

[Melba23]

mLipok,

The base code was the current repository version of the function - so I suppose you could say 3.3.13.20. But I do not believe that WinMerge will help you too much - when I tried a comparison it shows that almost everything was changed whereas most of the "changed" code was just moved inside If..EndIf structures. Better to work through the code and see for yourself.

M23

[jguinch]

Excellent idea Melba ! It will be useful !

[czardas]

Since I only recently updated to Windows 7, I still don't have much software installed on this machine: it's quite complicated maintaining legacy software but I have no choice right now. Anyway, I just installed beta and, after some inconvenience (AV deleted the exe), I took a look at this. I think these are quite cool features.

Edited February 3, 2015 by czardas

[Melba23]

jguinch & czardas,

Thanks for the feedback.

At time of posting 23 downloads are shown - anyone else from among those members feel like commenting? Or does anyone else feel like downloading and testing?

M23

[czardas]

I was just wondering what might be the consequences of someone entering a badly formatted regexp. I don't suppose there's a way to catch such an error, or is there?

[Melba23]

czardas,

I did think about that when rewriting the UDF but decided that parsing the regex within the UDF would be too great a burden. However, rethinking the matter I feel that perhaps running the passed regex through a simple StringRegExp call and checking for errors might might be a way to do a basic sanity check - although I still believe that we must leave most of the regex responsibility with the coder. I will see what can be done.

M23

[czardas]

I hadn't thought so deeply about it, but it occurred to me that no checks at all may cause a number of issues. I was thinking if it was possible to circumvent some such problems, it would be preferable. I haven't looked in any detail at the code, I just remember you mentioning something like this earlier on.

Edited February 8, 2015 by czardas

[Melba23]

czardas,

Something like this would be easy to add:

; Check if regex passed
If StringLeft($sList, 2) = "::" And StringRight($sList, 2) = "::" Then
    ; Set it directly
    $vMask = StringTrimRight(StringTrimLeft($sList, 2), 2)
    ; Run basic check on regex pattern
    StringRegExp("::", $vMask)
    ; If pattern error
    If @error = 2 Then Return 0

The UDF would fail (with the existing parameter-linked code in @extended) if a pattern syntax error is detected. Obviously this is no guarantee that a regex will return what the user intended, but will at least check that the syntax is legal.

M23

[czardas]

It's probably the best you can really hope for. I thnk it's definately worth considering. Throw some bad patterns at it and see if it breaks anything.

Edited February 8, 2015 by czardas

[Melba23]

czardas,

 

  Quote

Throw some bad patterns at it and see if it breaks anything

What do you think I did before posting?

Now if you use a regex pattern with a syntax error that returns @error 2 (as per the StringRegExp page in the Help file) then the UDF fails with @error/@extended pointing to the parameter that gave the error. Best that can be done I feel.

M23

[czardas]

I'm glad you did that. 

[H2Fooko]

Hello,

My first post ;-)

Thank you for this usefull function.

My 2 cents: I tryed the following regexp: "::^.*(A01|EKI)[ -].*\..*$::" as a pattern in your function that gives an error but this one  "::^.*A01[ -].*\..*$::" doesn't.

So I was wondering if the regexp's "OR" symbolysed by | was available or if I did a regexp syntax error ?

Thanks

 

[Melba23]

  H2Fooko,

Welcome to the AutoIt forums.

What error is returned by the function? When I test those patterns I get no errors.

M23

Edit: I see the problem - the "|" is being interpreted as a parameter delimiter and not as a RegEx "OR". I will look into the problem.

Edited August 13, 2017 by Melba23

[Melba23]

H2Fooko,

I have a solution, but it requires the user to make a small change to the syntax used for the RegEx. If "|" (OR) is used within a RegEx, it must be doubled to "||" so that the UDF can distinguish it from the standard parameter delimiter. So your RegEx would read:

The UDF then replaces any "||" found with a user-defined Unicode character (unlikely to be found in a filename), splits the entire string on any "|" delimiters, and finally resets any Unicode characters back into a single "|" so that the RegEx functions correctly within the PCRE engine.

This amended code works for me, can you please check that it works for you too?

And do you consider that the proposed solution of doubling the "OR" character is acceptable from a user's standpoint?

I have a better solution. The UDF now parses the entire parameter replacing any "|" (OR) characters within a RegEx (defined as being bounded by "::") with a user-defined Unicode character as explained above so that the parameter can be correctly split on the delimiters. Once split, the user-defined Unicode character can then be changed back to get a working RegEx. This means that the RegEx syntax can remain as expected.

Please check that this version works for you as well as it does for me: FLTAR_SRE_and_Func.au3

M23

Edit: Added a bit more error-checking when looking for RegExes within the parameter.

Edited August 16, 2017 by Melba23