SMF - The fastest duplicate files finder... [Updated 2023-Jun-03]

[AdmiralAlkex]

Can I use SMF to find all files called "[7 numbers].jpg"? Example: 1234567.jpg

And in that case, how?

[KaFu]

I'm not a RegEx expert, but setting the search input on the main screen to "(d{7}).jpg" and setting the "using" combobox to "RegEx Search" should do ...

[AdmiralAlkex]

You know more RegEx than me.

That returns a lot of other things like

just-cause-2-20101001001102509.jpg

and

263535345.jpg

I only want files that are exactly "7 numbers" + .jpg

[jchd]

This is because the regexp is not anchored to the left, thus it matches if there are at least 7 digits followed by ".jpg" the whole thing being possibly preceeded and/or followed by something else.

Try making the RegExp "(?i)^d{7}.jpg$"

which breaks down into:

(?i) makes the match case-insensitive

^ matches only at start of subject

d{7}.jpg followed by 7 digits and then ".jpg"

$ matches only at end of subject

That should not match abcd1234567.jpg nor 123456789.jpg nor 1234567.jpg.old

I wonder if the capturing parenthesis around the filename are required?!?

[AvvA]

Hi,

I'm using the v1.8 portable on a win7 Pro x64 and I perhaps found a small bug.

I find out that full MD5 hashes aren't right. I tested with big and small files too, to check with short MD5 and it seems to have the same problem.

I used my own program to check out (which uses default AutoIt _Crypt_HashFile() ), the CorZ checksum, and an online md5() too, just to be sure.

The hashes seem to be consistent, but the numbers returned aren't right. By consistent I mean that, in example, when doing a duplicate search the dupe files returned are alright.

I'll follow this topic, but as it doesn't keep this setting for long, don't hesitate to ask me by PM if you need informations, or reprimand me because I made a big wrong discovery .

[KaFu]

Hi AwA,

you are absolutely right, full MD5 hashes (full CRC32 and full SHA1 hashes too) are wrong in SMF 1.8. Thanks for reporting the issue , I didn't saw that!

And as you said they're consistently wrong. I've checked the calculation and quickly found the issue.

The hashes are calculated in 1 Megabyte chunks. I do this to make the calculation interruptable.

Now the issue was that all chunks were exactly 1MB large, even the last chunk which is only partially filled with the rest, e.g. 12.467 Bytes. The rest of the last chunk was filled with zeros.

That's why the hashes were still consistent, but not comparable with the ones created with other calculation methods. I've fixed this in the upcoming version 1.9, which I'll release as soon as AutoIt 3.3.8.0 is released ...

Btw, I've looked at your page. "UAC Pass" is a really nice piece of software, I've started coding something similar I called "UAC Elevator" (using the "Schedule.Service" object), but development on that one stalled . Why don't you post it on the forums?

Best Regards

KaFu

[AvvA]

Hi KaFu,

Thank you about explaining what was going wrong, it is always interesting to understand things.

About UAC Pass, thanks for the flowers .

I guess it is a shy attitude, this is really an insignificant tool compared to SMF.

Also the source are a bit messy... well, they are messy, and I see you, AutoIt MVP, coming with your whips ^^. Well, it makes me realize that I push back this moment since the beginning (cleaning the source code), I'll publish them on my site as they are, and then I'll make a topic here. I'll buy protection against whips, and clean the sources when I will have finished my current project.

I take advantage of this post to propose some ideas about SMF.

I see 2 things which could be nice, the first could help me with my current duplicates project (as I try a lot, without success, to convert the functions to AutoIt), the second could make it useless, but I don't care

- Why not list all hard links to a file in a tooltip when hovering the hard link number of a file (FindFirstFileNameW / FindNextFileNameW),

- Why not propose, after auto-selecting the 1st duplicate of a file, to hard link the other dupes to the 1st one? (either with a DLL call, either with integrated AutoIt FileCreateNTFSLink() )

This second idea includes that dupes must be on the same drive, which must be NTFS formatted.

Cheers,

AvvA

Edited December 19, 2011 by AvvA

[KaFu]

No worries on UAC Pass. I've seen something similar made in Delphi, but this is to my knowledge the first implementation in AutoIt... and the number of bugs and potential improvements in SMF diffinitly is somewhere in the thousands ...

As to your suggestion... I gave them some thoughts and came to the conclusion I won't implement them in SMF, sorry. FindFirstFileName requires Vista+ and thus would be of an overall reduced usefulness. The second suggestions is imho so special (I personally never have worked with NTFS links) that the effort would greatly outrun the benefits (for me personally ). But don't hesitate to PM me if you've got some questions on the functions themselves, I'll do my best to support you.

Thanks for making suggestions at all! In general I'll love to see SMF improved with unique features ...

Best Regards

[KaFu]

2011-Dec-31, Changelog 1.8 > 1.9

• Fixed some bugs and improved overall performance
• Updated Dlls and Binaries

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...

[KaFu]

2012-Feb-11, Changelog 1.9 > 2.0

• Fixed some bugs and improved overall performance
• Updated Dlls and Binaries

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...

[ldub]

I get errors at compile time (SMF v2.0)

(Windows XP Sp3 + Autoit 3.3.8.0)

Edited February 11, 2012 by ldub

[KaFu]

Thanks for the feedback. I know of these, these either don't have any effects (1, the function is a string and thus can be obfuscated => "will/could lead to errors" applies to calls using a variable as the function name) or are in functions that are not used in SMF and thus stripped in the obfuscation run itself (all others). Start the compiled exe and you will see that it works without problems (at least non generate by these errors )...

Edited February 11, 2012 by KaFu

[ldub]

Thanks a lot

SMF is awesome , you too !

[KaFu]

Thanks for the appreciation, glad you like it ...

[KaFu]

The upcoming Version 2.1 will have improved search speed and especially a reworked report functionality. Now the displayed preview icons can be changed on the fly and you can select between detail report, icon and tile view (with many style related customizing options, like adjust spacings, colors and fonts).

I'm currently in the process to wrap up all changes and retest all functionalities to make it fit for an official 2.1 release. However there are some many functions now, with so many interdependencies, that it's truly hard to say when it's really ready .

So everyone is invited to an open Beta test, here's a link to the current executable (source will be released with official version change again).

SMF_2.1_Beta.html

Here are three sample screenshots of the report, one in detail report view, one in icon view and one in tile view, showing exactly the same result set.

Best Regards

[adamchieef]

Thanks for the useful topic. Here's a program Duplicate Files Deleter another one to find and remove duplicate files.

[KaFu]

2012-Dec-27, Changelog 2.0 > 3.0

• Reworked the overall search loop and underlying SQL statements to improve speed
• Redesigned SMF_TNP.exe to create thumbnails using AutoIt native functions / does not use AutoItObject dll anymore
• Added Quicksave and restore functions
• Reworked the Export functions
• Report: Added icon and tile view
• Report: Added several new style and option settings
• Report: Updated "Secure Delete" function, uses wraithdu's excellent
• Report: Updated "Copy & Move" functions, uses Yashied's excellent
• Fixed a lot of bugs and improved overall performance
• For sure implemented a lot of new bugs , please keep reporting
• And much, much more I can not remember, but a whole version number is justified for this release ...
• Updated to SQLite DLL to v3.7.15.1
• Updated MediaInfo DLL to version 0.7.61
• Updated TrID Definitions to version 2012 Dec 23

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...

Edited December 27, 2012 by KaFu

[KaFu]

2013-Mar-20, Changelog 3.0 > 3.1

• All SMF functionalities should now work with long filenames too (max buffer = 4096)
• Added optional password to recover from tray
• Report: Added ToolTip Previewer for large previews of the currently selected file
• Report: Fixed bug which occasionally froze whole report (WM_COMMAND loopback)
• Report: Updated "Copy & Move" operations to handle long filenames, removed copy.dll and added own wrapper for wraithdu's
• Fixed a lot of other bugs and improved overall performance
• Updated to SQLite DLL to v3.7.16
• Updated MediaInfo DLL to version 0.7.62
• Updated TrID Definitions to version 2013 Mar 15

Example of the ToolTip Previewer in action:

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...

Edited March 20, 2013 by KaFu

[tcurran]

When I attempted to delete individual duplicate files, SMF deleted my entire Pictures folder. Luckily, it was simply moved to the Recycling folder, so I was able to restore it. But needless to say, I'm going to stop using SMF until you figure out what's wrong.

[johnfrag]

this is important tool. But I think you can post it by some minimize. It is very big. I use this tool for my pc <snip>

Edited March 21, 2013 by Melba23
Removed link