SMF - The fastest duplicate files finder... [Updated 2023-Jun-03]

[KaFu]

SMF is the Software-Tip of the Day in the Computerbild (German Computer magazine with quite a reach) .

http://www.computerbild.de/artikel/cb-News-Software-Search-my-Files-Freeware-Windows-7-Explorer-5991507.html

[KaFu]

Changelog v1.5 > v1.6

• Fixed report crash on deletion of long file names (> 260 characters)
• Fixed some bugs related to resources not release properly on exit
• Updated to SQLite DLL to v3.7.6.3
• Updated TrID Definitions to version 2011 May 25
• Updated 3rd party tool: FTM - File Types Manager (Win7 only)
• Added BitmapRip to Report - simple tool to extract embedded bitmap (JPEG, PNG, GIF) files from any given input file
• Relocated selection treeview to separate window and added some selection options
• Improved overall look and feel
• Added some file information to search for:

  [-] FullPath Length

  [-] FullPath IsANSI

  [-] FilePath Length

  [-] FilePath IsANSI

  [-] FileName Length

  [-] FileName IsANSI

  [-] Compression State

  [-] Number of Hardlinks

  [-] File Index Number

• Added a lot of file hashes and checksums to calculate:

  [-] CRC16

  [-] ADLER32

  [-] BLAKE224, BLAKE256, BLAKE384, BLAKE512

  [-] BMW224, BMW256, BMW384, BMW512

  [-] CUBEHASH224, CUBEHASH256, CUBEHASH384, CUBEHASH512

  [-] ECHO224, ECHO256, ECHO384, ECHO512

  [-] MD2, MD4

  [-] SHA224, SHA256, SHA384, SHA512

  [-] SHABAL192, SHABAL224, SHABAL256, SHABAL384, SHABAL512

  [-] SKEIN224, SKEIN256, SKEIN384, SKEIN512

  [-] Special thanks to Ward for his excellent AutoIt Machine Code Algorithm Collection UDF!

• Improved overall search speed, special thanks to Ascend4nt for his execellent _FileFindEx UDF!
• Fixed errors in "Computer Info" output result HTML files

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...

Edited May 29, 2011 by KaFu

[Ascend4nt]

Looks excellent KaFu.. letting it do a run-through right now.. but there's a weird quirk I am noticing - my taskbar suddenly lost all buttons. Not even SMF or folders are showing! I have to use Alt/Win-Tab to get to other programs. Even cancelling and closing SMF keeps the taskbar clear. Weird.

[KaFu]

Looks excellent KaFu.. letting it do a run-through right now.. but there's a weird quirk I am noticing - my taskbar suddenly lost all buttons. Not even SMF or folders are showing! I have to use Alt/Win-Tab to get to other programs. Even cancelling and closing SMF keeps the taskbar clear. Weird.

, I've had that bug too (on XP only), thought I've found it, but seems like it's still in there. Closing the explorer.exe process in taskmanager and restarting it will bring the buttons back. F..., I don't have a clue where to start searching, will have to code a helper app to monitor the taskbar and notify me when it's deleted (hidden?) to track down the issue. Maybe it's related to the IShellFolder object I use in SMF_TNP.exe?

Alas... On portablefreeware.com another bug was reported, searching for duplicates on a very large amount of files (> 200.000) might result in an "Could not allocate memory" error, because I read the files to a temp array in one go, that part needs to be reworked too, to calculate the dups in smaller batches (that should be done quickly).

I fear I have to recall v1.6 for now until I've found the bug.

Edited May 30, 2011 by KaFu

[Ascend4nt]

Ahh, that's too bad.. hate those bugs that are hard to track down. (especially in that amount of code!).

I'm actually running it again after a reboot, and I don't seem to have the disappearing taskbar buttons problem now. I wonder what changed, or was it chance? I know the first run was immediately after portable-install, don't know if that's related..

(Oh, and I'm running this on Win7 x64 if that helps.)

[KaFu]

(Oh, and I'm running this on Win7 x64 if that helps.)

That's even worse, I've only noticed the issue on XP, never on my Win7-64bit... I've deleted the download of v1.6 for now until I can track it down (sigh, I guess a monitoring helper app is the way to go). Thanks for reporting!

Edit:

Maybe it's related to this issue described in the comments for SHGetFileInfo on MSDN?

http://msdn.microsoft.com/en-us/library/bb762179%28v=vs.85%29.aspx

"Don't use ImageList_Destroy on the ImageList you get"

I do destroy the returned Imagelist in SMF.exe on exit...

Edited May 30, 2011 by KaFu

[Ascend4nt]

Good luck, I've never successfully used that function.. at least I think thats one I've tried to use. (I tried a few MSDN functions to help when parsing .LNK shortcut files, but the PIDL/SHITEMIDLIST and all that junk wasn't set up properly or I wasn't parsing it right.. meh)

One question - when you load search database results, are you not supposed to be able to view them? I tried loading a previous search and clicking 'Show Report' but it wouldn't give me any results, and the 'Select Report Data Columns' showed one checkbox with no text

*edit: tried on 1.6 and 1.5

Edited May 30, 2011 by Ascend4nt

[KaFu]

*sigh*, the bugger is just too complex. That's an error in the saving process, the "sqlite_master" table does not seem to be saved correctly. That one at least should be easy to fix by finally implementing jchd's excellent Thanks for the feedback!

Edited May 31, 2011 by KaFu

[KaFu]

Hiho,

here's a Beta of SMF which "should" fix

• Taskbar vanishing (had orphaned code in SMF to embed a GUI into the desktop, I think this caused the error)
• Mem allocation error on huge amount of duplicates to analyze
• Save / Load error of Database

SMF_Beta.html

Eager for more feedback ...

Regards

[Ascend4nt]

I did a few trial runs on it, and here's what I found:

The search and database save/load works pretty well (though with a bug I'll mention in a sec.), and the taskbar seems unaffected so far.

Now the problem: The database for some odd reason is taking the last search database and overwriting parts of it. For example, my 1st search resulted in a 44MB file, which I then saved and 'purged'. I closed and restarted SMF, then did a new search, and saved it to the same place (different filename though). What resulted was the same 44MB file for only 27 files. Made no sense for it to be that big, I thought - so I opened it with WinHex and looked into it. The result? It was the same database I had previously saved, but the first part of it was overwritten with new results. The rest of the database file still had results from the first search I did. This happened in further tests too.. all 44MB, all somehow copying and overwriting the last database's results.

On a side note.. I thought it was kind of a large file size, but looking through it I saw it stores everthing in plaintext in the file, so ~ 121,000 files with default attributes (+MD5) resulted in that. I guess compression would have to be done outside of the database.. however, I'm curious - does SQLite offer any encryption options? I'd like to use it some day, that's one reason I ask.

Oh - one final question - with the .DB file, you also save a .TXT and .INI file. The database load fails without those files present in the same folder. Is there no way to incorporate all that information into the database, instead of having 3 separate files?

Ok, thats all. Thanks!

Edited June 1, 2011 by Ascend4nt

[KaFu]

Ascend4nt, thanks for this excellent feedback!

The reason why the database still contains old records is the result of the default setting of this SQLite pragma:

http://www.sqlite.org/pragma.html#pragma_auto_vacuum

If you do not set it manually, this behavior is true for all SQLite operations.

I've now added additional manual VACUUM commands directly after the search has finished and as part of the "Purge" function. In the Beta above the manual VACUUM command is only triggered as part of SMFs exit function.

The database for some odd reason is taking the last search database and overwriting parts of it. For example, my 1st search resulted in a 44MB file, which I then saved and 'purged'. I closed and restarted SMF, then did a new search, and saved it to the same place (different filename though). What resulted was the same 44MB file for only 27 files. Made no sense for it to be that big, I thought - so I opened it with WinHex and looked into it.

I can not explain this behavior at the moment. Did you close SMF.exe properly or via a ProcessClose / Taskmanager End Process call? Because the VACUUM command was applied on a clean exit and the database should shrink if purged before.

Here's an update with the added VACUUM commands, take a look if this behavior still exists (couldn't reproduce it).

SMF_Beta.html

On a side note.. I thought it was kind of a large file size, but looking through it I saw it stores everthing in plaintext in the file, so ~ 121,000 files with default attributes (+MD5) resulted in that. I guess compression would have to be done outside of the database.. however, I'm curious - does SQLite offer any encryption options? I'd like to use it some day, that's one reason I ask.

Afaik not natively. There was a discussion about an encryption extension on the board... Yep, here it is:

Oh - one final question - with the .DB file, you also save a .TXT and .INI file. The database load fails without those files present in the same folder. Is there no way to incorporate all that information into the database, instead of having 3 separate files?

Could indeed need a cleanup . That's just the way the program grew. Data to a single table in the database (lots of performance potential there too), GUI Ini-Settings to the Ini-File and searched locations to a plain txt file. It would be time consuming to substituted all Ini call references with Sqlite read and write operations to something like an separate ini-table... and it wouldn't really improve the program performance . I was more thinking of wrapping all three files into a single file with an .smf extension.

Best Regards

[Ascend4nt]

I can not explain this behavior at the moment. Did you close SMF.exe properly or via a ProcessClose / Taskmanager End Process call? Because the VACUUM command was applied on a clean exit and the database should shrink if purged before.

I closed it with the red X on the GUI. The purge was done before the close however. Also, on a subsequent load, I loaded the database, and purged it again before closing.

Here's an update with the added VACUUM commands, take a look if this behavior still exists (couldn't reproduce it).

Whatever you did, its now working well. The databases are properly sized now

I was more thinking of wrapping all three files into a single file with an .smf extension.

That would probably make the most sense given your situation.

Ahh, and thanks for the info on encryption. Good to know.

[KaFu]

2011-Jun-02, Changelog 1.6.0.0 > 1.6.0.9 (re-released as v1.6 because of minor changes only)

• Bug fixed: The taskbar might have been hidden by the old v1.6. This should now be fixed, if not, please report
• Added some more VACUUM commands to clean up old results from the database properly
• Fixed an issue with saving and reloading the database
• Fixed an error with "Could not allocate memory" on huge amount of duplicates to analyze

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...

[liuyifan494]

Thank you for sharing！

[dmob]

Great script! I am learning so much from your scripts (not to mention picking 'em).

[cengizhan]

While smf (1.6.0.9) is running. i cannot use taskbar or explorer. i had end process with task manager. i am using windows 7 sp1 x64

Also how can i select all duplicates except one? selecting all removes all copies.

Edited June 5, 2011 by cengizhan

[KaFu]

While smf (1.6.0.9) is running. i cannot use taskbar or explorer. i had end process with task manager. i am using windows 7 sp1 x64

I thought that I solved that ... Can you tell me the operation, when this happens (e.g. report opens, report closes, search started...), I just can not find the problem...

Also how can i select all duplicates except one? selecting all removes all copies.

You can't at the moment, but this is a simple addition to the report and I'll add it to the next release.

[KaFu]

2011-Jun-06, Changelog 1.6.0.9 > 1.6.2

• Bug fixed: The taskbar might have been hidden by the old v1.6. This should now be fixed...

  [-] I think I identified a rogue call to FileIconInit as the source for the error. Please report if I'm wrong (again)

• Added button to report to select all duplicates except the first one

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...

Edited June 6, 2011 by KaFu

[KaFu]

2011-Aug-11, Changelog 1.6.2 > 1.7

• Updated to SQLite DLL to v3.7.7.1
• Updated MediaInfo DLL to version 0.7.47
• Fixed some minor bugs & design issues
• Added option to export thumbnails from report
• Added some file information to search for:

  [-] IsUPXed (part of "Executables Info" search option)

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...

[KaFu]

2011-Oct-21, Changelog 1.7 > 1.8

• Major Design Revision
• Fixed some bugs and improved overall performance
• Updated to SQLite DLL to v3.7.8
• Updated MediaInfo DLL to version 0.7.50
• Updated TrID Definitions to version 2011 Oct 17

Source and Executable are available at

http://www.funk.eu

Best Regards

Updated first Post... Enjoy ...