ID3 Clean

[llewxam]

I was trying to clean up my MP3 collection and was not happy.  Not happy because apps claiming to remove all ID3 tags were leaving lots behind.  What do geeks do when they are not happy?  Yup, do it themselves!

 

It turned out to be MUCH more difficult than I had imagined!  ID3V1 are easy, they come at the end of the file and are fixed length.  ID3V2 are insane, and can be many times the size of the song itself!  Combine that with other re-tagging or tag removal apps doing a bad job at housekeeping and you can have some pretty broken tags.

 

MP3 files are designed to be streamable and are built in packets called frames.  The frame headers contain information such as sample rate, samples per frame, bitrate, and padding, and those 4 key pieces of information are used to determine how many bytes the frame will be.  Of course it isn't actually that simple, since some of those factors depend on what MPEG version it is and if it is layer 1, 2, or 3.  But once you find a good header and can get the frame length, you jump to the next position and check to see if it is a valid header, get the frame length, rinse and repeat.  It is suggested to verify at least 3-5 headers but I found that totally insufficient.  My collection of nearly 11,000 songs in various encodings showed me that to be reasonably certain that we were out of the ID3V2 madness I had to verify at least 10 headers, but that was often not even enough.  I settled on 50 being the default value, and allow 150 verified headers to be REALLY sure.  There of course is a time penalty, but it is surprisingly quick - my standard benchmark of 423 songs, some of which have 5 or more ID3V1 tags and ID3V2 tags that go over 300KB deep, only takes 2 and a half minutes verifying 50 frames.

 

Enjoy

Ian

 

Edit 9/26/15 - updated code

ID3 Clean 1.24.au3

Edited September 26, 2015 by llewxam

[AoRaToS]

Trying this out now, looks cool!

Edited September 20, 2015 by AoRaToS

[TheSaint]

Thanks for sharing.

With my own program, I just remove any tag that is not standard for my requirements. I don't allow any others, no matter what they are. That usually just leaves me to clean up the Comments field. I also double-check with MP3Tag, because sometimes tags can be tricky and sneaky ... like including APE tags etc.

I will see how well yours compares, but I imagine yours is batch enabled, whereas mine is per album (or track) only, so yours could indeed be a handy tool to keep at hand.

Of course, we don't generally have issues with our own rips.

(though I also remove reference to encoder, etc)

[llewxam]

Indeed mine is meant for a batch, it can do album-level and I did test it that way but I mainly tested on lots of albums.  And though I didn't try it, I think it would crash and burn if you input a single file, though I am away from my computer at the moment and can't test the theory.

 

I did use your UDF at some point but ran in to problems, as you said "we don't generally have issues with our own rips" is very true, but NO other tools I have tried are able to keep on slogging through several broken tags like this does.  Not that I tried them all, but the higher-ranking ones from a few Google searches and all would leave things behind or just crash on some songs.

 

I played around with UDFing this but there are just too many arrays that got recreated for each song, waaaaaay too much of a performance penalty.

 

So yes, PLEASE let me know if I am failing anywhere, and I will maybe add a few quick lines to allow it to work on 1 file, not a bad idea.  I was also thinking of adding a context menu to get information on a specific file if anyone was interested, I'm not but hey, I'm not the only user now, am I??  

 

Ian

Edited September 20, 2015 by llewxam

[mLipok]

added here.

https://www.autoitscript.com/wiki/AutoIt_Programs#File.2FData.2FImage_Processing.2FViewers

Please put there a more appropriate description.

[llewxam]

added here.

https://www.autoitscript.com/wiki/AutoIt_Programs#File.2FData.2FImage_Processing.2FViewers

Please put there a more appropriate description.

Done.

 

Ian

[llewxam]

(though I also remove reference to encoder, etc)

Just curious - by this do you mean you remove the DOZENS of LAME headers some files have?  I did not experiment with this or research it, but in casual observance I thought they should not be removed.  If they don't matter, they may go next!    I found them highly annoying when looking at the files in a hex editor.

 

Ian

[TheSaint]

I only work with the Tags, both standard and custom user created. I don't touch anything in the header, that isn't in an ID3 tag. If you create an MP3 with Lame, then in my experience you get an encoder tag that says what Lame version created the MP3. I don't need that info myself, so I wipe it.

The only tags I keep are pretty basic ID3v2.3 ones. So nothing special like Lyrics, Performer, Writer, etc. I keep it simple, with sensible rules that I find easy to recall, because things can so easily get out of hand (slowing down ripping and cataloging).

Track Number (zero based and not 1 0f 10, and not disc numbering, etc).

Track Title (artist detail not allowed in this field). If a live track, then (live) is appended. I generally try to keep the text in this field as short as possible, so (bonus track) is (bonus) or (live bonus) and similar for (demo) etc, with all lower case unless a title (John's radio edit). I don't use illegal file name characters in any of my tags, so '/' or '\' is usually '+' or '-'. Hash numbering is used for duplicate names, for file name reasons, i.e. Black Night #2 (live).

Artist Name (this is track artist and guests only). Always uppercase for main artist and unless equal footing, titlecase for guests. Depending on the nature of the relationship, a '+' or '&' is used. An 'And' is always shortened to an ampersand.

Album Artist (only used when a Various compilation or a guest artist is included on a track, and only for that track). Guest names don't usually appear here, just the album artist name.

Album Title (with a disc indicator if more than one disc, i.e. The Wall (Remastered) - CD1). The only text that comes after a CD number, is a special title, usually in brackets, i.e. Anthology - CD1 (Rarities), for sorting reasons.

Year (4 digits only, original or first release, and mandatory). Annoyingly many don't bother filling the year in. In my view, it is more important than Genre.

Genre (this is generally two words, with the second being Rock, except when Classical or Jazz or Country or Blues in their pure form, else they get combined with Rock, Pop, etc) (When not completely sure or can't be bothered working it out, I often just use Rock on its own). I only use about 20 or so secondary descriptions, that I term Category. Common ones I use often are Heavy Rock, Jazz Rock, Southern Rock, Latin Rock, Folk Rock, Pop Rock, Fusion Rock, Orchestral Rock, etc.

Comment (this is often blank for my own rips, but can contain detail about extra artists and recordings, especially Live ones). Remastered Year is located here as well.

NOTE - I play my stuff via Playlists, so I'm not concerned with Album Title differences for multi disc. But typically in that scenario, I would have a combined playlist and a playlist for each disc, to give maximum flexibility.

P.S. I just know you wanted to know all that detail.   I have so much more I can tell you if you want.

P.S.S. I can't recall off the top of my head, but I think Folk is also a single Genre. The singles are all based on the original ID3v1 genres, but I don't bother with Misc and the others. So basically everything else revolves around them, and are sub categories.

Edited September 21, 2015 by TheSaint

[llewxam]

lol, I'll share back things that you may not need to know...

 

The only reason I spent NINE weekends in a row working on and testing this is because the artist tags not always being the same was causing my Android phone to break up albums despite the songs being in the same folder!!  After ignoring it for years I snapped; it takes snapping to become motivated enough to start working on an 11,000 MP3 collection!  And really, MY collection has been stripped bare now for weeks, but I felt compelled to share my solution so I had to milk the code for every performance improvement I could find, re-testing against a set of albums over and over, while making sure I didn't break anything by running against my entire collection periodically, which I copied over from my NAS to my puter, then copied albums from that folder to another one to test the code on  I literally put almost 6TB of writes on my 240GB SSD according to the Magician software, probably sucked a year off its life!!  

 

Ian

Edited September 21, 2015 by llewxam

[JohnOne]

For the record, how long to process 11,000 tracks?

[llewxam]

haha, about 2.5 hours, on a 3.6GHz processor and very fast SSD.  I still haven't done the "ultimate" test of running it on the NAS, that should be about a day I imagine.  But I may break down and (shudder) rename all songs to a standard much as TheSaint uses first!

 

Ian

 

Edit:  most songs get stripped VERY fast, whole albums can take under a second, but some songs are full of false-positives and some had ID3V2 over 1MB long!

Edited September 21, 2015 by llewxam

[JohnOne]

If I had to do something like that, I'd probably slow the script down and just let it run in the background during the course of the day.

[llewxam]

But then I'd never know how long it would take!    When it is time I'll start it before work and just let it go.

 

Ian

[JohnOne]

Longer it's running, the longer I can wait in the pub

Edited September 21, 2015 by JohnOne

[JohnOne]

You might think this silly, but when you are working with loops of that size it's worth squeezing every tiny bit of optimization.

For example "if @error == 1 Then" I think AutoIt might internally convert both ends of that to a string before testing.

So "if @error = 1 Then" over 22000 times or whatever might spare you a minute or two.

 

EDIT:

Also "If $ElapsedSeconds >= 60 Then" is making two checks

Might be better "If $ElapsedSeconds > 60 Then" (you might also be losing a second each minute there with = )

Edited September 21, 2015 by JohnOne

[llewxam]

lol @ 60 seconds, oops!  As for the @error, not 100% I follow, but I'm once again not at my computer, will have to sit down and look at it.

 

Thanks

Ian

[JohnOne]

== is a case sensitive comparison for strings in AutoIt. Since AutoIt is loosely typed and very forgiving, it is reasonable to assume that internally it casts both @error value and the test value to strings before making the comparison. But I'm not 100% on that.

[llewxam]

Thanks for the suggestion on == vs = JohnOne, but I changed all == to = and it made no difference in speed.  Worth a shot!  

 

I did, however, fix the >= 60 seconds bit.  No other changes.

 

Ian