Is the integrated sort within the recursive FLTA migrating to production worthwhile?

[Spiff59]

I agree, and he is always open to suggestions. But reading through this thread, it seems there have been more digs and fewer concrete suggestions.

@Spiff - I did not mean my post to sound harsh, apologies if it came across that way. You may well have a better way of doing things; I'm just saying - put it in code format so we can all benefit.

 

I was thinking it was a bit harsh, but you've erased any of that impression completely with your follow-up

I have a long history of being a major proponent for the development and incorporation of a recursive FLTA.

I spent a year and a lot of effort with others on thread 96952 that racked up 260+ posts and had at least some level of input from every Dev, Mod, and MVP in existence at the time.  That thread generated many good ideas, and many good function examples. Something coming out of that 2009 thread is the reason Melba has my name in the credits here. I have also pulled from that thread and posted in the Examples forum a 64-line version that I believe has never been beat for speed and has most of the same bells-and-whistles as this UDF.  But I don't want the function I posted.  It's based on an older Autoit. It lacks some fixes necessary for compatibility with Windows7 and 8.  I never worked out fully what the most logical interactions should be between all the file/folder/include/exclude filters/parameters.  It has sat a couple years and gotten dusty.  I mean nothing negative by this, but Melba very kindly spends a lot of time here, helps a lot of people, and makes a lot of friends.  Wanted or not, that creates a momentum, an unguided PR campaign, behind things like this UDF that enabled it to steamroll anything in it's path.  But the truth is, this IS the best routine. Melba spent a lot of time dotting i's and crossing t's, creating tables to document the complex filter interactions, he's improved the edits, the recursion level parm is useful and requires little overhead.  This is the most up-to-date, mature, refined recursive FLTA we have. Melba deserves a lot of credit and I want this routine in production! Well...

Back to my former nasty self...

Why explode the code 800%, duplicate a bunch of existing library functions, and create a maintenance nightmare for a sub-function that won't be used half the time, and already exists elsewhere.  I think I've said this all before.  Refer to my other posts here for more of my nastiness.

[Spiff59]

I'm home now, have had dinner, and had time to read all the posts that were made while I was feverishly composing a detailed response to a prior post and a request to rename this thread that had just appeared out of nowheree with my name on it. SUffering anxiety, I then accidentally deleting that response.after nearly 30 minutes of work.

Am feeling much improved with a full stomach, and I realized that if you (melba23) had made me the owner of this thread, then I had the access to rename it ,whcih I've now done.  You have an awesome and frightening power, one you need to be cognizant of. You've hands-down been the number one provider of assistance in the forums for a couple years.  You've graciously help nearly everyone at some point, myself included.  That is very selfless, commendable and of great benfit to the forums.  It does also create for you the largest number of users who feei grateful, or appreciate, or fond of, or even indebted to you.  When you publicly express displeasure at someone, I fear you run the risk of having a portion of your fan club dog-piling on the person in your sights.  I felt the "Why should Melba23's recursive FLTA UDF go to production?" title you authored on my behalf , would to those not familair with our history, the history of efforts to get a recursive FLTA into production, and the plethora of recursive FLTA candidates, only create the following inaccurate inferences:

1. That I was against the inclusion of a recursive FLTA in production.  That is the opposite of the truth. I've been a strong advocate for over 5 years, starting less than 60 days after I registered, of filling what I consider a major omission in the UDF library. A recursive FLTA is a high-demand function that is to complex to be composed on-the-fly by the average user. I used to play first-responder to some of the forum questions about a recursive FLTA, and recognized that having a standard function would reduce threadl count in the forums and the load of those servicing those requests.

2. That I'm against Melba23's UDF in it's entirety. I've made no such statement anywhere in the thread.

3. That I'm against Melba23's wish to migrate his UDF to production.

4. That since I'm against UDF"s like Melba's, am against Melba's UDF in particular, and am against what Melba wants to do with it, then I'm therefore against Melba himself.

While I was composing the response I eventually fat-fingered into oblivion, every "a new post has been added" popup had me picturing another mob member carrying a pitchfork and leaving me a nastygram reading basically "Spiff, you're and asshole".

I feel the new title is fair and representative of my post. let me know if you disagree.

I'm having a hard time seeing how my original post is a vicious flame almost warranting disciplinary action.

I was not wimpy about it, but I assaulted only the concept of retaining the integrated sort into production.

I'm sincerely all-ears and ready to repent once I understand where I went over-the-top.

I am, and have been, familiar with the internals of your UDF. The alternate sort algorithms, auto-selected for efficiency depending upon filter criteria.  It's prety slick really, but doesn't negate the opinion I hold that putting that into production is a KISS violation. The big version would still be available to those wanting it.

I don't understand what this has to do with the UDF? But it's fairly easy to see, go to his profile and you will see "Last Active".

 

Pardon, the basis of my reference to Valik would not be known to many.  He twice swept into the year-long effort for a quality recursive FLTA in thread 96952. Once was early in the thread where he walloped .someone for promoting a (programmatically) recursive candidate. Much later in the thread, when I was playing an active role, Valik popped in again. With his famous tact , we were informed that it would be a cold day in hell when something like our 80-line almost-alpha-candidate would be replacing a function in production that was currently 27-lines. We received a lecture about bloat, and some choice adjectives were leveled my way in describing the sort of person who might entertain tripling the size of a current function.  The thread soon fizzled out when ideas for additional development ran out.. There were a few competing candidates at the end, all top-notch compared to the other versions found in the wild.   The final result of the year-long effort was a bugtracker that I submitted to replace the existing _FileListToArray() function. The ticket was accepted and the change made it's way to production.  I was still feeling the lumps from Valik's ugly stick as the rewrite I submitted consisted of a minor speed-enhancment and a correction to error reporting. The previous 27-line function had been reduced to 18-lines.  I was discouraged that the grand vision held by many of a recursive FLTA function in production had been shot down in flames. It felt as if all the impetus for such an endeavor had been depleted from everyone and that it would likely be years before interest would regenerate to the level where another attempt would be made. An attempt that might again end up only a masochistic exercise in futility.

Edit: I reread my first post again, after having last looked it over 4 hours ago. I did definately come in with an attitude that I expected a knock-down drag-out fight would ensue over my suggestion.  I do apologize for the smart-ass attitude, but I did not intend and do not see that the post actually gets personal or belongs on anyones epic flame list. It may be I was predisposed to expect a fight, and that when the subject is about a recursive FLTA, that we're both oversensitive to each others statements.

Edited August 3, 2013 by Spiff59

[Melba23]

Spiff59,

Your gracious apology accepted with pleasure - please accept mine for any unintentional offence in return.

M23

[guinness]

If size and number of lines is a concern, then do not fear. What I've been doing recently is re-working examples to offer the same functionality (& error checking) but optimised, like you did with _FileListToArray.

Take for example _PathSplit (see header for the author,) this was consolidated into a 10 line function with the help of DXRW4E, as I found it to be bloated for what it did.. Oh and I should mention the 37% decrease in size by removing headers from the includes. Function usage is already documented.

So granted, though I'm trying to work to the same standards as Valik, sometimes I might make mistakes, I'm a novice after all.. But, I'm willing to listen and learn, I think all MVPs are. Plus, we're moving forwards, not backwards, which is why I decided Melba's FLTAR was a good candidate given the countless downloads and bug testing.

My comments are generic and not related to FLTAR.

Edited August 3, 2013 by guinness

[JohnOne]

For me, removing function headers because of the size of a download for the sake of a few lousy megabytes or a few extra minutes compile time is silly.

Or anything else regarding size which trades detail and/or completeness.

[guinness]

JohnOne,

I think you would understand the benefit if you had to change the documentation in two locations for every help file issue reported. If you prefer doing twice the work, that's fine. It's just not how I prefer to work.

Edited August 3, 2013 by guinness

[JohnOne]

I understand, but I'm looking at it from a users point of view, same as will thousands of others.

AutoIt has had great admiration across the internet for it's documentation Awsomeness, scaling it back is silly.

It's like putting less lettuce on burgers because the burger guy is overworked.

I know you do tons of work on the help file, and don't take it personally but thousands of people outside of us who do not post here, don't care.

They just want the Awsomness to continue.

[trancexx]

JohnOne,

I think you would understand the benefit if you had to change the documentation in two locations for every help file issue reported. If you prefer doing twice the work, that's fine. It's just not how I prefer to work.

Out of curiosity really, how does that correlate with idea of having multiple functions with the same signature or purpose in several standard UDFs? This idea is wrong from bottom to top. Are you sure you guys have all the bases covered? You said something about the no no thing with including array.au3. Why is that big no no? And please don't say it's because jpm or whoever told you. What's your opinion on the subject?

Just to say before anyone freaks out, I have no opinion on the topic. I can give you evaluation of the code that was added, but my guess is that you wouldn't like it.

[guinness]

I understand, but I'm looking at it from a users point of view, same as will thousands of others.

If the user is using the headers as documentation, then they might want to re-check their code for errors. What has happened since the UDFs came to be is docs have been updated but not the headers. I would say it's more like having to bins of lettuce and taking out the mouldy lettuce in one, but forgetting to do the other & vice-versa. Sooner or later you will have one bin that is just mouldy lettuce as someone forgot about. Case in point, the headers weren't being updated.

Did you look at what is there now?

Edited August 3, 2013 by guinness

[guinness]

trancexx,

Please share what you would have done. Thanks.

[trancexx]

trancexx,

Please share what you would have done. Thanks.

Does that mean you won't reply to my post?

[guinness]

My opinion and limited knowledge is irrelevant, as no doubt I will be wrong with my assumption. So I would rather listen to what you have to say, if that's OK?

[trancexx]

My opinion and limited knowledge is irrelevant, as no doubt I will be wrong with my assumption. So I would rather listen to what you have to say, if that's OK?

But you have strong opinions then, considering your limited knowledge. Besides I think your knowledge is not that limited.

More correct would have been to include Array.au3 than to replicate functions from there. If some function from Array.au3 wasn't fit for the job then that function may not be good at first place and should have been "fixed".

More than one quarter of functions from File.au3 works with arrays. It's to be expected therefore that one way or the other Array.au3 will get included at the end by the user. This would mean that the final script will have multiple functions included that all does the same thing. That will increase the sizes of their scripts, not make them smaller.

Someone may say "but the user will strip unnecessary functions", and that's just another argument pro including Array.au3. If the Array.au3 is included then very likely the public function from there will be used and they will be part of the final script just as "internal" functions from File.au3.

From the maintenance point of view, it's indeed nightmare. Spiff59 is completely right there. You also see that considering you said that was the reason why header were removed.

It's issue waiting to happen. And when it does it will be hard to trace it and fix it because fixing at one place will require fixing at multiple places. Sure it may be easier now when everything is fresh, but what about a year or two from now, when some new faces working on UDFs show up who wouldn't know that there are ten of the same functions in ten different UDFs.

I see the same thing with Security.au3 where someone removed included WinApi.au3 and replaced calls to functions from that UDF with internall calls. This is wrong. No sane script code will use Security.au3 without including WinApi.au3 one way or the other.  So in this case more correct would be to include WinApi.au3 (or whatever UDF the functions reside in now) than to do what's been done. Bigger picture is missing.

[AZJIO]

guinness
 

given the countless downloads and bug testing.

Judging by the last to a mistake you ate illusion.

[czardas]

I remember some of my early posts where I got a complete hammering. I tried to improve and there came more and more criticism. At one point I very nearly lost my rag. Now I look back and think about some of what I learned. from those encounters. I actually learned more than I have from everyone else since. Getting your code ripped to shreds is not nice, but if the person doing it knows their stuff you should really be grateful that they took the effort and spent the time with you. I'm not advocating rudeness but rather people don't take things so personally and do not fear criticism. I do it to myself all the time: my earlier post was a small and insignificant detail in the greater scheme of things.

I thought Spiff59 made a strong argument, now backed up by trancexx. I didn't notice any rudeness to be honest and I'm a bit surprised he felt the need to apologize for having strong feelings about something.. I understood the reference to Valik. One man's meat is another man's poison.

Edited August 3, 2013 by czardas

[AZJIO]

See simplified

expandcollapse popup

#include-once

Func _FileListToArrayRec($sPath, $sMask = '*', $iFlag = 0)
    Local $vFileList
    $sMask = _CorrectMask($sMask)
    If $sMask = '|' Then Return SetError(2, 0, '')
    If Not _FO_IsDir($sPath) Then Return SetError(1, 0, '')
    If StringRight($sPath, 1) <> '\' Then $sPath &= '\'

    __FileListToArrayRec($vFileList, $sPath, $iFlag)
    If $sMask <> '*' Then
        $sMask = StringReplace(StringReplace(StringRegExpReplace($sMask, '[][$^.{}()+]', '\\$0'), '?', '.'), '*', '.*?')
        ; $vFileList = StringRegExpReplace($vFileList, '(?mi)^[^|]+\|(' & $sMask & ')\n', '')
        Local $aFileList = StringRegExp($vFileList, '(?mi)^([^|]+\|(?:' & $sMask & '))(?:\n)', 3)
        $vFileList = ''
        For $i = 0 To UBound($aFileList) - 1
            $vFileList &= $aFileList[$i] & @LF
        Next
    EndIf
    $vFileList = StringReplace(StringTrimRight($vFileList, 1), '|', '')

    If Not $vFileList Then Return SetError(3, 0, '')
    $vFileList = StringSplit($vFileList, @LF)
    Return $vFileList
EndFunc   ;==>_FileListToArrayRec

Func __FileListToArrayRec(ByRef $sFileList, $sPath, ByRef $iFlag)
    Local $sFile, $s = FileFindFirstFile($sPath & '*')
    If $s = -1 Then Return
    While 1
        $sFile = FileFindNextFile($s)
        If @error Then ExitLoop
        If @extended Then
            If $iFlag = 0 Or $iFlag = 2 Then $sFileList &= $sPath & '|' & $sFile & @LF
            __FileListToArrayRec($sFileList, $sPath & $sFile & '\', $iFlag)
        Else
            If $iFlag = 0 Or $iFlag = 1 Then $sFileList &= $sPath & '|' & $sFile & @LF
        EndIf
    WEnd
    FileClose($s)
EndFunc   ;==>__FileListToArrayRec

Func _FO_IsDir($sTmp)
    $sTmp = FileGetAttrib($sTmp)
    Return SetError(@error, 0, StringInStr($sTmp, 'D', 2) > 0)
EndFunc   ;==>_FO_IsDir

Func _CorrectMask($sMask)
    If Not $sMask Then Return SetError(2, 0, '|')
    If StringRegExp($sMask, '[\\/:"<>]') Then Return SetError(2, 0, '|')
    If StringInStr($sMask, '**') Then $sMask = StringRegExpReplace($sMask, '\*+', '*')
    If StringRegExp($sMask & '|', '[\s|.]\|') Then $sMask = StringRegExpReplace($sMask & '|', '[\s|.]+\|', '|')
    If StringInStr('|' & $sMask & '|', '|*|') Then Return '*'
    If $sMask = '|' Then Return SetError(2, 0, '|')
    If StringRight($sMask, 1) = '|' Then $sMask = StringTrimRight($sMask, 1)
    If StringLeft($sMask, 1) = '|' Then $sMask = StringTrimLeft($sMask, 1)
    Return $sMask
EndFunc   ;==>_CorrectMask

But I think it is better to make >iteration

Edited August 3, 2013 by AZJIO

[AZJIO]

iteration

expandcollapse popup

#include <Array.au3>
$timer = TimerInit()
$FileList = _FileListToArrayRec('C:\Program Files\Daum\PotPlayer', '*.exe', 1)
If @error Then Exit MsgBox(0, 'Сообщение', '@error = ' & @error)
$timer = Round(TimerDiff($timer) / 1000, 2) & ' сек'
_ArrayDisplay($FileList, $timer & ' - все файлы')

Func _FileListToArrayRec($sPath, $sMask = '*', $iFlag = 0)
    Local $vFileList
    $sMask = _CorrectMask($sMask)
    If $sMask = '|' Then Return SetError(2, 0, '')
    If Not _FO_IsDir($sPath) Then Return SetError(1, 0, '')
    If StringRight($sPath, 1) <> '\' Then $sPath &= '\'

    Local $hSearch, $c = 0
    $hSearch = FileFindFirstFile($sPath & "*")
    If $hSearch = -1 Then Return SetError(3, 0, '')
    Local $aSearchList[125][2] = [[$hSearch, $sPath]], $sName
    
    Do
        $sPath = $aSearchList[$c][1]
        $sName = FileFindNextFile($aSearchList[$c][0])
        If @error Then
            FileClose($aSearchList[$c][0])
            $c -= 1
            ContinueLoop
        EndIf
        If @extended Then
            $c += 1
            $aSearchList[$c][1] = $sPath & $sName & "\"
            $aSearchList[$c][0] = FileFindFirstFile($aSearchList[$c][1] & "*")
            If $iFlag = 0 Or $iFlag = 2 Then $vFileList &= $sPath & '|' & $sName & @LF
            ContinueLoop
        Else
            If $iFlag = 0 Or $iFlag = 1 Then $vFileList &= $sPath & '|' & $sName & @LF
        EndIf
    Until $c < 0

    If $sMask <> '*' Then
        $sMask = StringReplace(StringReplace(StringRegExpReplace($sMask, '[][$^.{}()+]', '\\$0'), '?', '.'), '*', '.*?')
        ; $vFileList = StringRegExpReplace($vFileList, '(?mi)^[^|]+\|(' & $sMask & ')\n', '')
        Local $aFileList = StringRegExp($vFileList, '(?mi)^([^|]+\|(?:' & $sMask & '))(?:\n)', 3)
        $vFileList = ''
        For $i = 0 To UBound($aFileList) - 1
            $vFileList &= $aFileList[$i] & @LF
        Next
    EndIf
    $vFileList = StringReplace(StringTrimRight($vFileList, 1), '|', '')

    If Not $vFileList Then Return SetError(3, 0, '')
    $vFileList = StringSplit($vFileList, @LF)
    Return $vFileList
EndFunc   ;==>_FileSearch

Func _FO_IsDir($sTmp)
    $sTmp = FileGetAttrib($sTmp)
    Return SetError(@error, 0, StringInStr($sTmp, 'D', 2) > 0)
EndFunc   ;==>_FO_IsDir

Func _CorrectMask($sMask)
    If Not $sMask Then Return SetError(2, 0, '|')
    If StringRegExp($sMask, '[\\/:"<>]') Then Return SetError(2, 0, '|')
    If StringInStr($sMask, '**') Then $sMask = StringRegExpReplace($sMask, '\*+', '*')
    If StringRegExp($sMask & '|', '[\s|.]\|') Then $sMask = StringRegExpReplace($sMask & '|', '[\s|.]+\|', '|')
    If StringInStr('|' & $sMask & '|', '|*|') Then Return '*'
    If $sMask = '|' Then Return SetError(2, 0, '|')
    If StringRight($sMask, 1) = '|' Then $sMask = StringTrimRight($sMask, 1)
    If StringLeft($sMask, 1) = '|' Then $sMask = StringTrimLeft($sMask, 1)
    Return $sMask
EndFunc   ;==>_CorrectMask

Need to define parameters. It seems to me redundant flag 0

expandcollapse popup

#include <Array.au3>
$timer = TimerInit()
$FileList = _FileListToArrayRec('C:\WINDOWS\system32', '*.exe', 1)
If @error Then Exit MsgBox(0, 'Сообщение', '@error = ' & @error)
$timer = Round(TimerDiff($timer) / 1000, 2) & ' сек'
_ArrayDisplay($FileList, $timer & ' - все файлы')

Func _FileListToArrayRec($sPath, $sMask = '*', $iFlag = 0)
    Local $vFileList
    $sMask = _CorrectMask($sMask)
    If $sMask = '|' Then Return SetError(2, 0, '')
    If Not _FO_IsDir($sPath) Then Return SetError(1, 0, '')
    If StringRight($sPath, 1) <> '\' Then $sPath &= '\'

    Local $hSearch, $c = 0
    $hSearch = FileFindFirstFile($sPath & "*")
    If $hSearch = -1 Then Return SetError(3, 0, '')
    Local $aSearchList[125][2] = [[$hSearch, $sPath]], $sName

    If $sMask <> '*' Then
        $sMask = StringReplace(StringReplace(StringRegExpReplace($sMask, '[][$^.{}()+]', '\\$0'), '?', '.'), '*', '.*?')
        $iAll = 0
    Else
        $iAll = 1
    EndIf
    
    Do
        $sPath = $aSearchList[$c][1]
        $sName = FileFindNextFile($aSearchList[$c][0])
        If @error Then
            FileClose($aSearchList[$c][0])
            $c -= 1
            ContinueLoop
        EndIf
        If @extended Then
            $c += 1
            $aSearchList[$c][1] = $sPath & $sName & "\"
            $aSearchList[$c][0] = FileFindFirstFile($aSearchList[$c][1] & "*")
            If $iFlag = 0 Or $iFlag = 2 And ($iAll Or StringRegExp($sName, '(?i)^' & $sMask & '$')) Then $vFileList &= $sPath & $sName & @LF
            ContinueLoop
        Else
            If $iFlag = 0 Or $iFlag = 1 And ($iAll Or StringRegExp($sName, '(?i)^' & $sMask & '$')) Then $vFileList &= $sPath & $sName & @LF
        EndIf
    Until $c < 0
    $vFileList = StringReplace(StringTrimRight($vFileList, 1), '|', '')

    If Not $vFileList Then Return SetError(3, 0, '')
    $vFileList = StringSplit($vFileList, @LF)
    Return $vFileList
EndFunc   ;==>_FileSearch

Func _FO_IsDir($sTmp)
    $sTmp = FileGetAttrib($sTmp)
    Return SetError(@error, 0, StringInStr($sTmp, 'D', 2) > 0)
EndFunc   ;==>_FO_IsDir

Func _CorrectMask($sMask)
    If Not $sMask Then Return SetError(2, 0, '|')
    If StringRegExp($sMask, '[\\/:"<>]') Then Return SetError(2, 0, '|')
    If StringInStr($sMask, '**') Then $sMask = StringRegExpReplace($sMask, '\*+', '*')
    If StringRegExp($sMask & '|', '[\s|.]\|') Then $sMask = StringRegExpReplace($sMask & '|', '[\s|.]+\|', '|')
    If StringInStr('|' & $sMask & '|', '|*|') Then Return '*'
    If $sMask = '|' Then Return SetError(2, 0, '|')
    If StringRight($sMask, 1) = '|' Then $sMask = StringTrimRight($sMask, 1)
    If StringLeft($sMask, 1) = '|' Then $sMask = StringTrimLeft($sMask, 1)
    Return $sMask
EndFunc   ;==>_CorrectMask

Edited August 3, 2013 by AZJIO

[TheSaint]

SUffering anxiety, I then accidentally deleting that response.after nearly 30 minutes of work.

Perhaps you need my little >Web Pad program I've added to Example Scripts?

I've shared your pain.

[Spiff59]

I had no intention of re-opening some sort of competition between opposing versions of recursive FLTA's.

Melba's is an excellent routine and the only one we have that is thoroughly tweaked, tuned, and tested.

My main argument is that it is inappropriate to marry disparate functions. It results in unnecessary growth, and increased maintenance efforts. Allowing this one, a disk i/o function combined with the sort function, will also encourage others to promote more combined "all-in-one" functions. They may save the novice coder a few lines, but behind the scenes they would create nothing but headaches.

Being an optional parameter, the control code for the sort parameter is not deeply entwined within the function. It could be easily removed , along with the 12 helper sub-functions, without degrading the integrity of the remainder of the function.  It would still be a mature routine without the sort parameter.  

I do feel bad at making the suggestion at all, because a lot of effort has gone into the sort portion of this function. The auto-selection of the most efficient sort algorithm based upon the input parameters is pretty cool.  But it was not enough to overcome the alarm going off in my head saying that this is bad practice.  The full-version with embedded sort could still be available to any who needed it.

I'm done, have presented my view, and will see how the chips fall. My intentions were not personal, but only to keep Autoit clean and on a solid path. Excuse me for kicking off my argument in a snarky tone. I'm reminded of many times when IT managers pissed me off by evoking "standards" to shoot down something I thought was really spiffy. Now I'm being the standards nazi.  The world won't end regardless of what decision is made. If anyone replies "What specifically are you proposing?", I'm going to go bananas.  Since the first post all I've said is:

1. Autoit needs a recursive FLTA in production.

2. Melba's function is the best choice.

3. It should go to production sans the embedded sort code.

Edited August 5, 2013 by Spiff59

[Bert]

 I can see you have obviously put lots of thought into this. I challenge you to take it to the next step and write a UDF to do what you have in mind.

or...Is there some reason you do not write your own and share it?