MattHiggs

Duplicate bookmark removal

5 posts in this topic

Hey all.  So I had an issue recently where xmarks, a cross-browser bookmark synchronization add-on, has completely jacked up my bookmarks (duplicates everywhere, items in incorrect folders, just a disaster).  Needless to say xmarks is now on my crap-list of never touch again, but I still needed a way to get my bookmarks back to normal.  Unfortunately, the categorization part will have to probably be done manually, but I figured removing the duplicates should be scriptable.  This script is the result of that need: you start by going to a browser which has the messed up bookmarks, go to the bookmark manager, then exporting the bookmarks as a html bookmark file.  Then you just open aforementioned file within this scripts prompt, select a save location for the new html file that will be generated, and wait for the script to finish.  Now, as of now, this script doesn't really let you combine multiple html files into a single one with all unique values added in due to the nature of my issue, but if you need this functionality, I could always add it in (or you could, doesn't really matter to me :P).  So, without further ado:

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#AutoIt3Wrapper_Change2CUI=y
#AutoIt3Wrapper_Res_SaveSource=y
#AutoIt3Wrapper_Res_Language=1033
#AutoIt3Wrapper_Res_requestedExecutionLevel=highestAvailable
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#cs ----------------------------------------------------------------------------

 AutoIt Version: 3.3.15.0 (Beta)
 Author: Matt Higgs

 Script Function:
    Remove duplicate bookmarks from exported bookmark html file

#ce ----------------------------------------------------------------------------

; Script Start - Add your code below here
#include <Array.au3>
$open = FileOpenDialog ( "Select html containing exported bookmarks", "", "HTML file (*.html)", 3 )
$save = FileSaveDialog ( "Where do you want to save modified bookmark file?", "", "HTML file (*.html)", 18 )
ProgressOn ( "Please wait", "Scanning bookmark file for values", "", Default, Default, 18 )
$oIE = FileReadToArray ( $open )
$num = @extended
Local $old[$num]
For $h = 0 To $num - 1 Step 1
    $old[$h] = StringStripWS ( $oIE[$h], 3 )
Next
$new = _ArrayUnique ( $old )
$html = ""

For $i = 0 To $num - 1 Step 1
    ProgressSet ( ( ( $i + 1 ) / $num ) * 100, $i & " out of " & $num, "Please Wait" )
    If StringCompare ( StringLeft ( StringStripWS ( $oIE[$i], 3 ), 6 ), "<DT><A" ) = 0 Then
        $in = _ArraySearch ( $new, StringStripWS ( $oIE[$i], 3 ) )
        If @error Then
            SetError ( 0 )
            ContinueLoop
        Else
            $new[$in] = ""
            $html = $html & $oIE[$i] & @CRLF
            ContinueLoop
        EndIf
    Else
        $html = $html & $oIE[$i] & @CRLF
        ContinueLoop
    EndIf
Next
$file = FileOpen ( $save, 2 )
FileWrite ( $file, $html )
FileClose ( $file )
ProgressOff ()

 

Share this post


Link to post
Share on other sites



Just a hint in case you're refering to Fifefox bookmarks: you may have better time using the SQLite bookmark database directly rather than having to mess up with the html export. Depending on what you need to do exactly you may even be able to get along with a good SQLite 3rd-party manager (SQLite Expert is perfect) without having to write a single line of AutoIt code.

Tech points to consider: your profile bookmarks are in C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile>.default where the places.sqlite DB lives. This is a WAL mode DB, so there are likely two extra files places.sqlite-shm and places.sqlite-wal when Firefow runs. Don't just copy the .sqlite file alone, it will be corrupt if not used with the two other files.

Then for safety duplicate the moz_bookmarks table to "bookmarks_backup" then again to "bookmarks_with_dups" and de-dup from there using SQL. When you're sure to get your SQL correct that delivers the result you want you can empty the real bookmark table and copy back your resulting output there (don't DROP the table, just delete from it). Afterwards, you can drop your backup tables when you're sure everything works fine, no hurry since FF will obviously ignore the extra tables you've created for the operation.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
7 hours ago, jchd said:

Just a hint in case you're refering to Fifefox bookmarks: you may have better time using the SQLite bookmark database directly rather than having to mess up with the html export. Depending on what you need to do exactly you may even be able to get along with a good SQLite 3rd-party manager (SQLite Expert is perfect) without having to write a single line of AutoIt code.

Tech points to consider: your profile bookmarks are in C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile>.default where the places.sqlite DB lives. This is a WAL mode DB, so there are likely two extra files places.sqlite-shm and places.sqlite-wal when Firefow runs. Don't just copy the .sqlite file alone, it will be corrupt if not used with the two other files.

Then for safety duplicate the moz_bookmarks table to "bookmarks_backup" then again to "bookmarks_with_dups" and de-dup from there using SQL. When you're sure to get your SQL correct that delivers the result you want you can empty the real bookmark table and copy back your resulting output there (don't DROP the table, just delete from it). Afterwards, you can drop your backup tables when you're sure everything works fine, no hurry since FF will obviously ignore the extra tables you've created for the operation.

You know, I had discovered that while I was researching various aspects of automating bookmark processes, but for the life of me, I couldn't figure out how to query the database for the information that I needed using autoit's splite udf.  The array I kept getting was null.  And I did all of this without removing any of the databases or files from their original position.  Now I'm not saying you are wrong, that is on me, the sqlite UDf confused the hell out of me.  Or are you suggesting skipping the sqlite udf altogether and just run SQL directly within the script?

Share this post


Link to post
Share on other sites

As I said, you can perform all SQL operations you need from within a solid 3rd-party SQLite manager. Writing a script around this is just calling for pointless complications. I warmly recommend SQlite Expert.

Just ask if you need assistance doing so.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
6 minutes ago, jchd said:

As I said, you can perform all SQL operations you need from within a solid 3rd-party SQLite manager. Writing a script around this is just calling for pointless complications. I warmly recommend SQlite Expert.

Just ask if you need assistance doing so.

Ok.  I will definitely look into that.  The positive about doing it this way though is that it can be done with whatever browser you use: Chrome and firefox both have bookmark managers that let you export and import html files, so that was the other benefit of doing it this way.  But I will still definitely see if I can't ready database files directly.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now