Jump to content

Duplicate Files Finder - MD5 Hash CheckSum


ptrex
 Share

Recommended Posts

could u post me the used UDF s?

i got a error for:

$WM_MENUSELECT

and

$WS_OVERLAPPEDWINDOW + $WS_VISIBLE + $WS_CLIPSIBLINGS)

should have the beta installed.. and also have the udf´s u unclided.. but no included .au3 has this parameters inside :)

thanks

Link to comment
Share on other sites

Au has undergone some changes since this program has be released.

#include<StaticConstants.au3>

If a const is not declared, always search the AU3 include files first, most of the time it's hidden there somewhere.

Link to comment
Share on other sites

Link to comment
Share on other sites

  • 3 months later...

I registered the Dll but it still says Comm error com object not registered

My Projects:[list][*]Guide - ytube step by step tut for reading memory with autoitscript + samples[*]WinHide - tool to show hide windows, Skinned With GDI+[*]Virtualdub batch job list maker - Batch Process all files with same settings[*]Exp calc - Exp calculator for online games[*]Automated Microsoft SQL Server 2000 installer[*]Image sorter helper for IrfanView - 1 click opens img & move ur mouse to close opened img[/list]
Link to comment
Share on other sites

Link to comment
Share on other sites

Nice script ptrex!

Because I used SQLite recently I have some optimization tips for you:

- use indexes for SQL tables (for columns used in WHERE)

- use BEGIN at start and COMMIT at end (without this each _SQLite_Exec will internally do commit transaction which slows down processing)

- don't calculate $sFileDate,$sDateStamp for small skipped files (skipped due to size filter)

- don't use Sleep (25) in main FOR NEXT loop 

- refresh progressbar and "Time Elapsed" only when it's neccessary --> avoid flicker/high CPU usage

- do INSERT INTO Duplicates in batch --> collect INSERTs into string variable and and call _SQLite_Exec() less times

- don't assign $DataCol = GUICtrlCreateListViewItem() --> it's not used

- use _GUICtrlListView_BeginUpdate / _GUICtrlListView_EndUpdate --> smoother/quicker ListView display

- don't do DROP TABLE at end of script --> it's not needed in case of memory DB

- use _SQLite_GetTable2d() instead of _SQlite_Query + _SQLite_FetchData --> I think it will be faster for big result sets - see next post from Kafu

-  use optimized MD5 hash calculation code --> search forum for examples

http://www.autoitscript.com/forum/index.ph...st&p=687067

http://www.autoitscript.com/forum/index.ph...st&p=557689

See attached optimized version of your script (except MD5 code - it's original untouched, and also _SQLite_GetTable2d not used).

Note I didn't run it so it's not tested yet.

I like that idea (search for duplicates this way) very much so I will rewrite also MD5 hash later probably and I will test it.

DuplicateFiles_MD5_SQL_New_pz.au3

Edited by Zedna
Link to comment
Share on other sites

Also another possible speed optimizations:

- skip small files (according to filter for size) already inside _FileListToArrayEx()

- _FileListToArrayEx() --> there are too many Redim statements inside FOR loops --> it should be before loop because Redim is relatively slow operation

Edited by Zedna
Link to comment
Share on other sites

- use _SQLite_GetTable2d() instead of _SQlite_Query + _SQLite_FetchData --> I think it will be faster for big result sets

I don't think so, In SMF the usage of _SQLite_GetTable2d() was in fact a large performance issue, and I switched back to use _SQlite_Query + _SQLite_FetchData to create my own array.
Link to comment
Share on other sites

I don't think so, In SMF the usage of _SQLite_GetTable2d() was in fact a large performance issue, and I switched back to use _SQlite_Query + _SQLite_FetchData to create my own array.

Interesting info.

In fact I used only _SQLite_GetTable2d() for small result sets in my project without any speed problems

so my above presumption was based only on idea that one DllCall should be faster than many DllCalls in cycle.

EDIT:

Coincidentally my project was for searching duplicates too.

But it was for searching duplicate objects stored inside PowerBuilder's PBL libraries (163 files with 550 MB size in summary containing 21 000 objects).

Firstly I used my own AutoIt's array to collect/search object names/sizes/dates but this way searching lasted 35 minutes.

After redesigning to use SQLite (memory database with indexes) it lasts only 20 seconds!!

So SQLite is really awesome in some cases.

Edited by Zedna
Link to comment
Share on other sites

@Zedna,

Good to see that you debugging my old version :D

Which I left long time ago. And switch over to a build-in MD5 Hash version.

Anyhow some of your suggestions are valid, others are not.

I am interested to see what your new version has to bring.

Regards,

ptrex

Link to comment
Share on other sites

  • 2 weeks later...

@Rose

What are the differences in these two programmes

I can"t tell the differences.

Since there is very limited info on how the Clone Remover tool works, on the site.

how can I use the one from this article?

Just read the first topic on how to get started.

You need to download the MD5 COM object from the site mentioned there. And follow the installation instructions,

Then run the au3 script, open a folder, start it, and see what duplicates appear.

regards,

ptrex

Link to comment
Share on other sites

  • 1 year later...
  • 7 months later...
  • 7 months later...

Duplicate Files Finder - MD5 Hash CheckSum

Someone in the support forum was pointing to the Xstandard MD5 Hash COM object.

For those of you who don't know what a MD5 Hash CheckSum is :

Well this would be an example on how to use the CheckSum to find duplicated files or Scripts on your system :

Download here XStandard - MD5 Com Object

This small application is my first using the new syntax of 3.2.9.10.

It contains most of the extensive functionality that AU3 has to offer :

- DLLCalls

- SQL

- In Memory Database

- COM object

- RegisterMsg

- Native AU3 functions

- ...

post-6305-1198791790_thumb.jpg post-6305-1198791800_thumb.jpg

DuplicateFiles_MD5_SQL_New.au3

Free Duplicate File Finder

Au3 - 3.2.13.13 version and higher

DuplicateFiles_MD5_SQL_New.au3

The speed of the application is more than reasonable.

It even scans your server if needed.

Enjoy finding duplicate scripts, audio, zip, exe, ... files

Enjoy !!

regards,

ptrex

@Polyphem

You need to have the SQLite DLL avaiable on the machine that runs the code.

Best way is to have the DLL somewhere in the Windows root folder.

This will solve your problem.

@JamesB

Indead !!

Since almost anything is accessible through SQL these day.

And it won't be long before the OS will support native SQL.

So you better get used to it. :)

regards,

ptrex

__________________________________________________

Thanks For This Useful Info....

Link to comment
Share on other sites

  • 5 years later...

For a duplicate files finder, Duplicate Files Deleter is one of the good software. It will give you a comprehensive list of all those files and you can decide for yourself what you want to do with them. It's easy to use. Hope it will help you.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...