Unofficial archive. This is a read-only, community-run archive of the Apple Support Community. It is not affiliated with or endorsed by Apple Inc.

how to delete duplicated songs all at one time?

I would like to get help deleting duplicated songs, I do not know how they got their in the first place.

165 replies

Some how I got duplicates of my library on iTunes and to get rid of all the new duplicates I went to the library I looked for the date the new duplicate songs were added to the library then I went to columns and where you see Artist, song,

bit rate, time, etc and added 'date added' once that's done click on the box enclosing 'date added' and you will see all songs added by date. Proceed to the last one of the songs with that date press command to highlight it then scroll all the way to the first song and press 'shift' to highlight all of the songs added on that date then simply press delete.

you will be asked if you want to delete these songs...That's it.

iTunes has two modes for displaying duplicates. The first is accessed via File > Display Duplicates and lists all tracks in the current playlist where the same track title occurs more than once. This is, more often than not, a rather loose definition of the term and can bring back obviously different songs, e.g. studio and live versions of the same song, the same song recorded by different artists, or by the same artist but on different albums, or completely different songs that have nothing in common but their title.


A more useful feature can be found if you hold down the SHIFT key and then select File > Display Exact Duplicates. This gives a list of all tracks where more than one track has essentially the same metadata, name, track no., artist, album, etc. (I'm not sure what the exact criteria are for an exact duplicate as far as iTunes is concerned, but it's good enough for most needs).


There are three types of files that can show up as exact duplicates:

  1. Multiple library entries for the same physical file
    These can occur if your media location is temporarily unavailable, e.g. due to a disconnected external drive, iTunes becomes aware and marks the tracks as missing, you then reconnect the drive and add the media again. iTunes won't recognize that the files are already in the library and will create duplicate entries.
  2. Multiple copies of the same physical file
    This can happen if you have "Copy to iTunes media folder when adding to library" enabled and accidentally add the same folders from outside of the iTunes Media folder into your library.
  3. Different copies of the same track, in different formats or bit rates
    This can happen if you decide to rip a CD that is already in your library on a later occasion or, for example, decide to create MP3 versions of some existing AAC files and then don't remove the originals.

Note you could also have files which have the same audio data but different metadata, e.g. the same album ripped twice but with a slightly altered album title, however there isn't an obvious way to highlight these within iTunes.


Once you have used the Show Exact Duplicates feature you can s 00004000 et about deleting all but one copy of each duplicate file. If all the duplicates have recently been added into your library then you could sort the list on the date added column, select a range of tracks with SHIFT-click and delete all the recent entries. If the duplicates have been added into the library at various times then sort the list by album or track name, select all but one of each group of matching tracks with CTRL-click and delete the selection. Probably best to work one screenful at a time.


Of course the issue isn't quite as simple as it seems. If the duplicates are of type 1 then you want to delete the tracks from the library, but not delete or send the underlying files to the recycle bin as there is only one copy of each file referenced by the duplicate entries. For tracks of type 2 you do want to delete the duplicate physical copies leaving one remaining version. For files of type 3 you probably want to decide which of the two or more copies you want to keep, the smallest to take up less room, the largest because it is the best quality, the MP3 version because it is the most portable or more than one because each has their uses for you.


By now you should have some idea of why deduping is not a trivial issue. Of the three types I have described it turns out that cleaning up type 1 duplicates is quite easy to automate. I've written a script called DeDuper which can go through a selection of tracks, spot those with the same path and remove the redundant entries. As a bonus it retains the entry with the earliest date added value, adds in the play & skip counts from the deleted entries and sets the last played/skipped dates to their most recent values. Although the script would work on the whole library it will be much more efficient to use the iTunes Show Exact Duplicates tool before running the script. You should also backup your library or, at the very least, the iTunes database iTunes Library.itl before running the script. Chances are you have type 2 duplicates and the script is of no use to you in its present form...


I hope to extend the script over the next few days to cope with type 2 and type 3 duplicates, ideally making the distinction between the two types, handling type 2 automatically (hopefully moving the removed files to somewhere they could still be restored from if required) and prompting for the user to choose which file to keep in the case of type 3. Again metadata would be merged. Hmm, just realized I need to do something about rating too.


tt2

The DeDuper script referred to in this thread is written by me and is free for anyone to download and use. There is some guidance on using my scripts at the top of the scripts page, but here is a more detailed description.




A step-by-step guide to using the DeDuper script


  1. Start by backing up your iTunes library. See this backup tip if you don't already have a strategy for this.
  2. Download the script if you haven't done so already. This is a direct download link to DeDuper that you can right-click on for download options: Save to your downloads folder, desktop or somewhere else you can find it easily when you want to use it.
  3. If it isn't already shown press CTRL+B to enable the menu bar, then from the Songs view hold down the SHIFT key and from the menu bar select View > Show Exact Duplicate Items. (Clicking the Same Album button seems to work if you forget to hold shift.)
    You should get something like image below, click the image to see it in more detail. Click on the heading of the Album column if needed to change the sort order and use CTRL+J to enable artwork if desired.
    User uploaded file
    Note there are 4 copies of the first track. In the second album only tracks 1 to 4 are duplicated which is why only those tracks from that album show up in this view.
  4. To test how the script works select a sample of tracks rather than everything. In my case I'm going to select the Adele albums only by clicking on the first track, then scrolling to the last one, and shift-clicking that. This highlights all the songs between the two selections. Like this:
    User uploaded file
  5. Double-click on the script to run it. Because you've downloaded this script from the web you will probably get a warning message asking if you want to open it.
    User uploaded file
    Obviously this is where you need to decide if you trust my advice. If you do click Open. If you want to prevent repeat warnings you can uncheck the box that says "Always ask..."
  6. Assuming you click Open you should now get a dialog box like this:
    User uploaded file
    Review the text, cancel if the numbers don't agree with your intentions, otherwise click Yes.
  7. Depending on your settings for User Account Control and Internet Explorer you may get a progress bar displayed as the script evaluates the duplicates, otherwise you will just get the findings:
    User uploaded file
  8. Click Yes to remove the duplicates. When they have been removed you will get a results screen:
    User uploaded file
    Note that all the duplicates of Adele's tracks have been removed leaving the originals behind.
  9. If you're happy the script does what it is supposed to press CTRL+A to select all the duplicate tracks and run it again. Click the Done button at the top of the display to exit from the show duplicates display when you are finished with it.


Note that it is worth testing the script on one or two albums first to get a feel for it and also to determine if you get shown the progress bar. If not you need to be prepared for the script to take time processing the library with no feedback as to how it is getting on, or easy way to interrupt the operation. Tips for allowing the progress bar to work are at the bottom of the scripts page. All smart playlists are recalculated with each removed track so run time is probably a power relationship to library size, i.e double the size of the library and the script could take 4 times as long to run. If the progress bar won't display it might be wise to work with smaller chunks, e.g. start with all artists that begin with A, then B, etc. rather than tackling the whole library in one hit.


Once the library has been cleaned of the duplicates check it through before deleting what has been put in the recycle bin. When you are happy nothing has been deleted that should not have been you can update your backup.


tt2

Hi,


  1. Apple's official advice is here... HT2905 - How to find and remove duplicate items in your iTunes library. For a Mac use Alt or Option while accessing the menu to reveal the Show Exact Duplicates command.
  2. All copies of all tracks with the same details are listed. Either the same song name and artist for the regular feature, or the same song name, artist, album, track no. & disc no. for exact duplicates. You need to manually select all but one of each group to remove. Sorting the list by Date Added may make it easier to select the appropriate tracks, however this works best when performed immediately after the dupes have been created. If you have multiple entries in iTunes connected to the same file on the hard drive then don't send to the trash.


My script is for Windows, Dupin Lite should do something similar on a Mac, or tackle the duplicates by hand. Whichever way you choose backup before you start in case you do something too radical and need to recover.


tt2

Thank you for the incredible amount of work you have put making iTunes palatable for the rest of us. My iTunes library is on my home NAS. I read through most of this thread and then downloaded DeDuper today from your website on a test of 25 duplicates:

I get this error:

Line: 503

Char: 55

Error: The track is not modifiable

Code: A0040203

Source: (null)


Thank you for your assistance!

TomQA

Yes, I believe that's the problem. You are probably right, it could happen getting playlists from Macs, too. I made a quick hack and it fixed my problem, but I don't know anything about VB, and I'm sure my hack isn't complete.


This is all kind of silly, anyway. iTunes should ignore duplicates if they are in exactly the same place in the filesystem. Oh well.

.vbs fles should be opened with the Microsoft (R) Windows Based Script Host.


User uploaded file


You should be able to right-click on the file and choose the appropriate option, or use the Folder Options dialog to correct the action for the file type.


tt2

Turing, now I am getting a different error. As for the blocked issue, I could see that the folders which contain the music files were set to be read only folders (even though the file itself was not readonly). Does that give you any kind of hint?


ringUser uploaded file

Dear Turing, this is the error i get when running the script. UAC is off, i'm using 'find exact duplicates' and select them all correctly. The maps aren't set to 'read only' My OS is Win7. Would be awesome if you could help me.

User uploaded file

btw awesome help troughout this thread by you ๐Ÿ™‚

Backup your library.


You can download the Deduper script by right-clicking on this direct link:http://samsoft.org.uk/iTunes/DeDuper.vbs and then clicking download, save as or whatever your browser offers.


You should be able to double-click the downloaded file to execute it. Windows will probably give some kind of warning about running the file, if you accept it, then it should run. It relies on Windows Scripting Host which is normally installed and enabled by default. This page of the DeDuper thread has some stuff on installing/enabling it if needed.


When you run the script it should look something like this.


User uploaded file

(Click on image to enlarge)


And the prompts should lead you through the rest of it.


tt2

Had a slight mishap today with my iTunes library... Long story short: I backed up an old .itl file when I reverted my iTunes back to 10.7 (I had installed 11.0.4 yesterday for the iOS 7.0 update and hated the interface). Using the old .itl file somehow created duplicate tracks everywhere.


I can get the DeDuper script to work, but only on the manual/preview setting in which I have to approve every single delete. My library is currently displaying at 190 GB (roughly 35 GB of that are duplicates), thus I would really, really like to have the automated part to work! The error that displays is below:


User uploaded file

turingtest2 wrote:


Perhaps dbPoweramp has some setting you can change, or maybe you need to kill a "watchdog" process?


tt2


Yes, I had a look and there is a configuration box but I would be clutching at straws to be honest. Maybe I'll uninstall it and see how it goes.

User uploaded file

turingtest2 wrote:


Not being familiar with the software I'm clutching at straws too, but those "pop-up information tips" might be where the software offers to help if you add new media and warns when you try to delete it...


tt2


I think this is not going to be possible with the software I have conflicting with the scipt and iTunes etc ๐Ÿ˜Ÿ. I tried to deselect the pop-up information and did get a smaller box here;


User uploaded file


So this looks like a VLC file so I configured iTunes for .m4a and got the same box but with the iTunes 'logo'.

Thank you for your script. When I try using it, it runs into an error. I'm not sure if there is an issue creating the shell object or the folder object. I am running Windows 7 x64 and the latest version of iTunes 11.


An thoughts on how to fix this? Thanks for your help.


User uploaded file

Interesting. Not sure why there is the "\..\..\..\" but this is what I get for FilePath and FolderName. It seems like the playlist file list where it is located and then points to where the file is located. The two paths are seperated by the "\..\..\..\"


User uploaded file


User uploaded file

Hi turingtest2,


I just read this entire thread and am amazed with your dedication to helping strangers fix this problem. Sincere thanks!


Now I apologize if this is a stupid question, or if it has already been answered, but how exactly do you run the DeDuper script? I downloaded it and tried to open it with iTunes (already having switched to displaying exact duplicates and highlighting all the songs) but I didn't see anything happen. In another post you mentioned that a progress bar may show up, though I didn't get one.


Am I attempting to run the script properly? The method I used felt a bit too simple. While I never got an "iTunes can't open this file" type message, I also haven't seen anything happening.


Thanks so much!

Thanks for your quick reply! That is what I initially attempted, but windows didn't know how to open the file. It gave me the "search the web for appropriate program" or "browse" options, so I decided to try to open it with iTunes. I think I guessed wrong. What should I have done when that window popped up?


Thanks again for your help, friend.

One clue (for the poster with the issue) is to look at the "icon" for the script. If it's the generic Windows icon, yes, WSH is likely not installed.


That eHow artice says it should be installed by default too. Hmmm...

I have Windows 7 and it seems like it is installed by default. I had to do a bit of digging on how to enable it, as intially the scripting host didn't have the enable option. This Yahoo Answer post explained how to create it. Unfortunately, iTunes was saved as the default program to open the file. I tried to open the program with regedit (C/Windows/regedit) but it would not open. The error message said "The specified program is not a registry script. You can only import binary registry files from within the registry editor."

turingtest2 wrote:


... This leads me to two ideas that I've been mulling over but I should really find the time to complete. First a script to eliminate repeat entries in a playlist for the same object. Second a reworking of my scripts so that the first job they to is make their own internal deduped list of objects which they then work with to prevent this kind of problem...


tt2


I haven't got around to rewriting all of my existing scripts but I've updated DeDuper so it can cope if running on the contents of a regular playlist that includes the same track entry more than once. The script will now clean this fourth type of duplicate from a regular playlist (a playlist duplicate) instead of generating errors.


I'll probably get around to writing a dedicated playlist cleaner that can tidy all playlists in the library, but DeDuper can do that task one playlist at a time. I also now have a system for making sure other scripts won't behave unexpectedly when working with playlists that contain repeated items.


tt2

Sir,


I have exactly the problems you described, but I am running Itunes on Windows.


I have 0 experience with scripts and similar codes. Therefore, I dont even know how to run your script, even though I got the code.


Could you please assist me?


Much appreciated,

Joรฃo

Hi, what language is Windows using? I'm not quite sure why the error occurs on that particular line but the ones just preceding it are designed to send duplicate files to the recycle bin. This can't be done directly so involves invoking the Windows Explorer right-click menu. The script is looking for Delete in the list of commands so my guess is things go wrong if that command isn't there.


tt2

joaokessler wrote:


Thanks for replying. The Delete option is on the right click menu yes. Do you think it could have anything related to being windows instead of mac?


No, my scripts are exclusivley for Windows rather than Mac so that shouldn't be an issue.


As to the other error two possibilies come to mind, either the file is locked in some way that prevents the script from removing it, or it gets deleted but in a background process allowing a small window in which it stll appears to exist. Do you get a lot of the errors, or just a few? Have you checked to see if the named files still exist?


tt2

Hi, I've uploaded a minor update which might prevent this error. It is a little tricky to pinpoint since it isn't going wrong for me. Windows always reports folders as being Read-Only even just after you're cleared the Read-Only flag so that won't be relevant. Generally your own account (or everyone) and system need full access to all the files in the iTunes & iTunes Media folders. Offhand I don't actually know which account VBScript uses for file system & shell operations, but I would assume it is the active user.


tt2

What language is Windows set up for?


Line 491:


If File.Verbs.Item(I).Name="&Delete" Then


will need an edit for non-English regions. The & shows the letter that would be underlined when looking at the context menu, i.e. Delete.


tt2

Hi, that sounds like the file might not have download properly. I've inspected the version that was there and can't see or reproduce the error. Either way I've uploaded a slightly updated build which is working fine at my end. Still trying to find time to complete a major upgrade that will detect files on alternate paths to the same folder.


tt2

iTunes 11 has removed the Show Duplicates/Exact Duplicates functions so I've added two new scripts to make playlists of Duplicates and Exact Duplicates to my scripts collection.


If deduping by hand use shift-delete to remove selected tracks from the library as well as the playlist.


The Exact Duplicates script makes a good starting point from which to run my DeDuper script if you don't want to do it by hand. Please take note of the warning to backup your library before deduping.


tt2

When you right-click on a file what word is the equivalent to delete? Google translate offers Verwijder. Assuming the w is underlined in the context menu, like this Verwijder, then the line becomes:


If File.Verbs.Item(I).Name="Ver&wijder" Then


tt2

While 'Verwijderen' is indeed the correct word, the hotkey is none of the letters in it.
http://windows.microsoft.com/nl-BE/windows7/Keyboard-shortcuts states that the hotkey is either the Delete button or Ctrl+D. What should I put in the script?
Thanks by the way.


Edit: it appears i was wrong and it should be 'Ver&wijderen'. The script deleted 108 tracks.

While now i won't get a error message, the script doesn't recognise many of the dupes. Your other script 'Duplicates' and the feature of Itunes (I have yet to download the latest version) do recognise that there are dupes.

(http://i.imgur.com/GF1sv.pnghttp://i.imgur.com/lbnNI.png)

The script works with what iTunes called "Exact Duplicates" these tracks have the same Song Name, Album, Artist, Track Number and Disc Number. The "Duplicates" view shows songs which simply have the same Name. There are plenty of legitimate reasons why the same song title can appear in your library, even on the same album, without really being a duplicate. Exact duplicates should however be exactly the same song and there is no need to have more than one copy of those.


I notice, for example, that you have tracks listed from the Adele album 19 also repeated as 19 (Deluxe Edition). To clean up using the script you would first have to choose one "correct" album title and apply it to all of the tracks. (My preferences is to remove excess labels like Deluxe, Bonus Tracks and Live, but it's up to you.) If you're having to do that it may be easier to clean up by hand, but the script will preserve ratings, playlist membership, play counts, etc. which you may want to keep.


tt2

I unticked "Show Itunes in the Cloud purchases", and also tried "Hide Music in the Cloud" under the view menu, but I still get the dialog. Regardless, your script still makes it SO much easier. I might just relent and start clicking away at itunes dialog this weekend with a cold beverage.

Hmm, may help to close & reopen iTunes. Would you have ripped something you once purchased at better quality from CD? I suspect you're only getting that prompt from purchased/protected tracks that appear as dupes, but of course if that is most of them then that's a drag.


tt2

Yeah - I rebooted the machine after the change. Its no biggie really. I can click the dialog 900+ times sometime this weekend. Its definitely not something caused by your script. If it matters I'm using itunes 11.0.2.25 under win7 x64.

I'm not going to uninstall and reinstall just to get itunes to behave like it should on just these boxes. It is what it is for some reason. icloud is running in the tray but i don't think that would have anything to do with it. No worries - thanks for the excellent script!

Well the only thing I can think of is that you've managed to download your purchases more than once or somehow physically copied and reimported them into your library. I'd have to run some tests which I'm not set up to do where I am now, but there may be a faster 00004000 way to eliminate those by showing only purchased tracks in a smart playlist, sorting on date added, and then shift-deleting the more recent imports. (As long as you don't need the playlist/playcount merging features of the script).


tt2

What version of Windows? Is the Windows Scripting Host installed and enabled?


http://www.ehow.co.uk/how_8392805_enable-script-host-windows-7.html


If you're still having trouble could you post a screen shot of the error message?


tt2

Hi TT2,


I'm getting an error just before the delete stage.

Line: 375

Char: 39

Error: Object required: 'T'

Code: 800A01A8

Source Microsoft VBScript runtime error


OS is a fully up to Windows 7 SP2

iTunes version is 11.0.2.26, I'm just updating to 11.0.3 to see if that works (however I never find iTunes updates to be particularly reliable)


any ideas on what the problem could be? or what software I can use to debug the script? (I get the feeling that this is an obvious question but please humor me)


Thanks!

The current version doesn't cope if the source playlist has multiple references to the same track. This shouldn't happen if you've started with the Songs view and then use Shift > View > Show Duplicate Items, but if you've prepared a playlist elsewhere and dragged suspect dupes into it for sorting later that could be why. I'm scratching my head for other possible reasons just now but if that's not it let me know and I'm sure I can track it down.


Or better yet finish off the next version. ๐Ÿ˜‰


tt2

thanks tt2


I've just run it again after only selecting whats shown when I view exact duplicates as I was previously running it with all duplicates, and all songs.

However I'm still getting the same crash, so its crashing on all possible combinations of tracks: "All Songs", "Duplicates" and "Exact Duplicates".


I am asking it to only delete true duplicates ("no" on the final confirmation dialogue) could this have something to do with it?


Thanks for your support!

No the choice isn't a problem. Have I mentioned turning off iTunes in the cloud yet?


Use View > Hide <Media Kind> in the cloud or Edit > Preferences > Store and untickShow iTunes in the cloud purchases to hide the cloud items.


I've really got to find some time to update all my scripts 00004000 to account for cloudy items. Been snowed under at work lately but let me know if that makes any difference. If not I'll see what else I can come up with.


tt2


I've just had a thought, would it be possible to modify the script so instead of deleting the dupliates it moves the tracks that it would have normally deleted into a separate playlist? Then I can simply delete them by hand.

That's probably a good idea... I've been pressed for time lately so I haven't had a chance to do any testing so see if I can work out was going on but adding to a playlist should be feasible. I'll try to work on it tonight.


tt2

Hi TT2 -


I'm running Windows 8 and iTunes 11 and used your script to create the ExactDuplicates playlist, which worked perfectly. I then selected the files in the playlist and ran the DeDuper script. It runs, but doesn't detect any duplicates "No duplicates were found from the xx (where xx is the number of tracks I have selected) processed tracks: No duplicates were removed." I've done this with the source being the Exact Duplicates playlist and the show exact duplicates items list. I have amlost 13,000 duplicates so I'm hoping to avoid doing this manually. Any ideas what's going on?


Thanks!

Hmm. Just fired up a test library, deliberately created the three types of dupes the script is able to find, and it clears them perfectly. I'm running iTunes 11.0.3.42 on 32-bit versions of Windows 7 & XP. I don't have access to a Windows 8 system as yet but perhaps I can set up a trial virtual machine.


I had hoped to work on the "make a playlist of the dupes" version last night but work intervened. Perhaps I can make time today. In the meantime it might be worth testing what happens if, having revealled the exact duplicates, you then narrow down to a single artist using the search box (click the magnifying glass and untick "Search entire library") and run the script on the smaller selection.


tt2

A bit of testing later, it looks to me as though the problem is when you are trying to de-dupe large number sof tracks. Works fine with 999 selected items, any more and it fails. I'm currently trying to find the exact threshold.


Erm.. just did a few more (the last two tests) that have disproved the above theory. I hope the tests are of some use.


Tests are below.


28 Duplicates from 112 Selected Items

28 Physical

No Problems


954 from 2251

798 Physical

156 Alternate

- 'No' - Delete all but alternate dupes

- Fail 00004000 ed.


Line: 375

Char: 39

Error: Object Required: 'T'

Code: 800A01A8


114 from 245

108 Physical

6 Alternate

- 'No'

- Success

108 Deleted

6 Skipped


375 from 856

301 Physical

74 Alt

- 'No'

- Success

301 Removed

74 Skipped


416 from 1110

333 Physical

74 Alt

- 'No'

- Failed - error as before


423 from 1000

336 Physical

85 Alt

- 'No'

- Success


416 from 1037

321Physical

95 Alt

- 'No'

- Failed


253 from 1346

201 Physical

52 Alt

- 'No'

- Success

Hi, yes thanks. I'm just reviewing the script now and while I still can't see for certain why it isn't working I can see scope for checking that the current line 350 has assigned a valid track object. If it doesn't that might explain the failure on line 375. Of course that doesn't explain how I've managed to get an invalid object reference stored in the list of things to delete earlier in the script, assuming that is what is going on when it fails. ๐Ÿ˜•


tt2

Weird! ever since I consolidated (thank you for your help, again, you're my hero), it won't let me dedupe!

(newest iTunes, newest Deduper, Windows 8.1) -

keeps giving me the info "There was a problem deleting the file ..." within the iTunes Media folder, ending with a 1 or 2, thus an exact duplicate. the script works fine, only the deleting part is somehow not working. will go back and read the 18 pages I missed since last time, maybe you've already answered that question. ๐Ÿ™‚

I'm not sure if this will help but when I receive the first delete confirmation and answer "No", this message appears and the script stops:


---------------------------
Windows Script Host
---------------------------
Script: C:\Users\johng_000\Downloads\DeDuper (1).vbs
Line: 518
Char: 9
Error: 0x80270000
Code: 80270000
Source: (null)

---------------------------
OK
---------------------------

I haven't managed to get the archiving routine working yet, but something clicked while I was thinking about the problem. I've a feeling I solved it before for someone else by changing the way the file is deleted. The attempt to delete files into the recycle bin essentially emulates deleting a file via Windows Explorer, but it we're not interested in saving them then a straight file system delete command should go through unchallenged. I've added a new control variable UseTrash which is currently set to true so that most users get the protection that it provides. Download a fresh copy of the script and change the line (currently line 137) that reads:


UseTrash=True ' Attempt to send local deleted files to trash


to


UseTrash=False ' Attempt to send local deleted files to trash


Then save the edited script and run that... Usual caveats about having a backup and testing on a small group first.


tt2

Chris1955 wrote:


Thanks for your script. It helped me get out of the mess I had created. You are a genius ! Is there a similar script or Program that works in Mac ?


Try Dupin or Dupin Lite from Doug's Scripts.


tt2

DeDuper took forever (days) because of the massive number of duplicates I had, but it worked like a charm - still no progress bar but not complaining. Thanks for your time, expertise and the new control variable! It's people like you who share their talent on the net that makes the lives of people like me a little easier.


I owe you a beer but it may have to be a virtual one.


Take care!

Hello all.


I'm having the same issue reported back in 2011, and have had trouble figuring out which is the original file after several backups; many of the file in my Itunes media library effectively do not exist - they have no physical location on my hard drive, but Itunes thinks they are there. It sounds like dedupe is the anwer, but i'm not tech savvy enough to know how to use a script, and can't find anywhere that's not a spamming site, or one that wants to charge you $30 for the privelege of using free software.


So how do I use a script, or where can I download dedupe for a reasonable amount?


This has been a problem i've been trying to solve for years through painstaking song by song deletion, so help is much appreciated.

I don't see a Kind column, and my library has only music and a few podcasts. With your explanation, I need to look at my NAS and find out how permissions are controlled. I just noticed that the iTunes folder doesn't have the usual Windows folder Properties | Security tab. However, I have full access to the NAS share, and I can add to the iTunes library, so I wouldn't have thought security permissions was a problem.

Regards,

TomQA

The script runs without a progress bar which doesn't bother me but when I go to delete the files it only deletes 997 at a time. I then have to run the script again and the same result happens...


Any help would greatly be appreciated...

TT2,


Sorry my whole question didn't post.


As far as the error, the script would run just fine and show me all of the duplicates. However when it went to send them to the recycling bin it would only send 997 items. I just ran the script multiple times until the duplicates were under 997 items, still made it a lot easier. Thanks for being so quick to respond...

OK, I've been struggling with this for ages, if someone posted a solution or a similar solution, i apologize, but I didn't want to read the whole post, i just wanted to share this solution for any who struggled with duplicates as i have.


THIS SOLUTION WORKS, IS EASY AND COST NOTHING.


1) First up, back up all your audio to a separate folder (not necessary, but always safe).

* If you have multiple locations where you added songs from, you can consolidate iTunes media to get them all in one folder then back the music in the iTunes Audio Folder, or manually cut and paste all music into one single library (easy to find location of your choice, I'm on PC so I stuck with my music) as i did*

2) Download iTunesDSM which can be found here, http://itunesduplicatesongmanager.blogspot.ca/

3) Open iTunes, go to edit, preferences, advanced, and under iTunes media folder, change location to an empty folder anywhere (create one if need be)

4) In iTunes, click on songs, select all, delete. (this will delete the song from iTunes but not from the save location)

5) Close and open iTunes, (not necessary, but recommended)

6) Go to edit, preferences, advanced, and under iTunes media folder, change the folder to your library (sometimes this won't work, if this doesn't work here's a workaround, Go to File, add folder to library, and once all the songs are added, you can change the folder in preferences) wait till all songs are added.

*These above steps ensure that you do not have any duplicated media due to the date added duplicating issue*

7) Open iTunesDSM.

8) Click on options, add directory, and locate the music library you are using for iTunes.

9) Click Select All, enable the options of "backup selected duplicates to a folder" (make a temp folder on the desktop and use that, this is not necessary, but just to be safe) and the option "delete selected duplicates from music folder.

10) Click Clean Duplicates. (This may take several minutes if you are backing up the duplicates.

11) Once that is complete, Click Remove Orphaned Track. (gets rid of all those annoying songs with an exclamation next to it.

12) Close and open iTunes, then hold shift and click view, show exact duplicates. There should now be none, if there are any, it should be the odd few that you can manually delete.

*If any duplicates re-appear you can repeat steps 7-12 since and just save the back up of the library if you have the space*

The DeDuper script solution being discussed in this thread also works, costs nothing, is easier to use, and unlike the one you suggest preserves/merges ratings, played/skipped counts/dates, playlist membership and original date added values.


But then I'm slightly biased since I wrote it. ๐Ÿ˜‰


tt2

Hi,


I'm getting an error when I run the DeDuper script;


Pass 3, processing 16,701 of 26,729

Scanning for physical duplicates


Script: M:\<folder name>\DeDuper.vbs

Line: 614

Char: 18

Error: The track is not modifiable.

Code: A0040203

Source: (null)



My song files are on a NAS, and I'm getting the error when I selected ALL on the Exact Duplicates list. I tried the script on a much smaller group and it worked fine. I moved the .vbs file off the NAS before running it, but same error.


Any ideas?


many thanks for your work.

Not sure what that is about, but I've added a little error trapping code around that statement. Should you get an error again it may become clear which track is being processed when it occurs. With luck there will some feature of the file which will explain the cause.


tt2

Thanks for the response.


I don't know if you updated the script where we can download it (http://samsoft.org.uk/itunes/DeDuper.vbs still shows 1.0.2.5 - June 29th 2014 as the latest version) but I ploughed on anyway.


the problem track is somewhere in the "J"s; given that the error appeared at "243 of 345" I guessed that it had to be the 243rd song title beginning with J. But even after deleting +/- 10 tracks either side of the suspect, I get the same error running the script; no new information than in my original post BTW.


Does the script examine tracks in alphabetical order?

Clear your browser's cache and try again. It should be labelled Version 1.0.2.6 - October 7th 2014.


Processing order in my scripts is normally in reverse order of the tracks as shown in the display (this prevents certain kinds of errors) but one should not assume that iTunes will give them to the script in precisely that order.


tt2

Brilliant. Now in one of those images could you point out some of the tracks that FindTracks is failing to pick up, or mark out the successes, whichever is easier. I'm interested in the failures at this point.


tt2

Hmm, download a fresh copy of the script. I made a minor change two days ago while troubleshooting a different issue, but it might solve the problem for you too. So far I've picked up all of the first dozen test tracks I've made to match your examples.


tt2

Aha, I thought we were still trying to repair the broken links!


I'm thinking what you want is a DeDuper that ignores Album, Disc & Track number? Danger with that is that if you have both live and studio versions that aren't distinguished by track names then you'll throw out one at random. It would take a little thinking about and testing, but I could probably shoehorn that in as an option.


In fact, dirty and untested, simply change these 6 lines in DeDuper (currently starting at line 246):


Case 2 ' Physical dupes, same details and size

If L<>"" Then Key=LCase(.Artist & "\" & .Album & "\" & DN & "." & .TrackNumber & " " & .Name & "=" & .Size)

Case 3 ' Alternate dupes, same details but different size

If L<>"" Then Key=LCase(.Artist & "\" & .Album & "\" & DN & "." & .TrackNumber & " " & .Name)

Case 4 ' Missing dupes, same details but file missing

If L="" Then Key=LCase(.Artist & "\" & .Album & "\" & DN & "." & .TrackNumber & " " & .Name) Else Key=""


for these:

Case 2 ' Physical dupes, same details and size

If L<>"" Then Key=LCase(.Artist & "\" & .Name & "=" & .Size)

Case 3 ' Alternate dupes, same details but different size

If L<>"" Then Key=LCase(.Artist & "\" & .Name)

Case 4 ' Missing dupes, same details but file missing

If L="" Then Key=LCase(.Artist & "\" & .Name) Else Key=""


And it should do what you want.


I guess you'll also want a more liberal pattern matching system for the SyncStats script as well. ๐Ÿ˜‰


tt2

Yes! I was trying to repair broken links.


But then, when I wasn't able to repair all of them, I added the songs that weren't repaired and tried to merge them with DeDuper (which worked for some). And I have very few duplicates of songs. Usually picked which one I liked most.


Yeah, the SyncStats, I plan to use when I get this first broken links part complete. I haven't tried it yet. The play counts on my iPod are hundreds more, for some reason.


I think, I used to save Playlists, and those playlists contained play counts. When iTunes deleted or corrupted my playlists in updates, I think iTunes reset the playcounts to the "music" counts on this computer.



Should I try Find Tracks again (is that the one that is a new version?) and then run DeDuper again?

Try a fresh download of FindTracks first. If you've only got one remaining physical file track from two originals (one currently lost and one found) then iTunes should show the same metadata for both once things are reconnected, and the unaltered DeDuper will then behave as it is meant to.


tt2

The fuzzy matching scheme obviously works better when there aren't so many tracks all lumped into the same folder, and with so many sharing the same starting characters, nevertheless there are some surprises there. ๐Ÿ˜ฎ๐Ÿ˜•


Looking at it there will be a fuzzy match from the Name to the Artist. I'm sure I can find a way to tweak the script to prevent those. See Soundex for the rule scheme. Cream & Cryin' both get a soundex code of C650.


tt2

I've added another tweak to FindTracks so that it uses two soundex patterns rather than one when the search string contains " - ". Hopefully this won't negatively impact the behaviour elsewhere (I haven't had time to test or consider it deeply enough) but it should prevent the song name is similar to the (completely unrelated!) artist name matches that you've been seeing. Grab a fresh download and see if it has helped.


tt2

I take it that is the newer version of the FindTracks script? I've probably got to force it to skip some of the other search patterns that it is using for your library. Hopefully I can look at it again sometime tomorrow. SyncStats is going to need a tweak if you hope to match tracks on the device that might have valid track no. or album data with those in the library that don't (or vice versa).


tt2

Apologies for not getting back to you sooner. Been busy with Real Lifeโ„ข. Still haven't got around to tweaking FindTracks further as I think that will need more of a hack, but for SyncStats and ImportDevicePlaylists you should be able to go in and edit this line:


Signature=LCase(.Artist & "\" & .Album & "\" & .DiscNumber & "." & .TrackNumber & " " & .Name)


in a function called Signature and replace it with something like:


Signature=LCase(.Artist & "\" & .Name)


to reflect the properties that you still have in your files after you've edited them. Differences in album, disc number and track number will be ignored but as long as a track with matching artist and song name exists it will be treated as a match.


tt2

My PC died and I replaced it with a MacBook. I lost iTunes but not my music files as most were stored on external drives. I can import them to a new iTunes library but, want to avoid duplicates. The trouble is that the file names are in disarray. For example, sometimes "The Beatles" are just "Beatles" and album names may be Anthology Disc 1 or Disc One. Some of the titles include a track number, sometimes there is no album name or it's "Unknown Album". I have about 340GB of music files mostly ripped from CDs, a few purchased, and a few recorded live. What is the best way to get the library reconstructed with the fewest duplicates or errors?

I looked at the 9000 duplicate songs in my Itunes folder and found a common thread that I am using to get the duplicates out somewhat quickly. The duplicate filename ends with either a space then 1.mp3 , 2.mp3 , 1.m4a and 2.m4a. I began by using a program called SearchMyFiles to search my Itunes music folder for filenames that ended with each of these file endings using wildcard options. I then selected the results of the individual searches and moved them out of the Itunes folder into a new folder called duplicate songs on a spare hard drive . Afterwards, I re scanned the Itunes folder for duplicates and found it to be clear. You might also check for duplicate album cover Jpegs and other files using the same methods.

Good luck


BH

I looked at the 9000 duplicate songs in my Itunes folder and found a common thread that I am using to get the duplicates out somewhat quickly. The duplicate filename ends with either a space then 1.mp3 , 2.mp3 , 1.m4a and 2.m4a. I began by using a program called SearchMyFiles to search my Itunes music folder for filenames that ended with each of these file endings using wildcard options. I then selected the results of the individual searches and moved them out of the Itunes folder into a new folder called duplicate songs on a spare hard drive . Afterwards, I re scanned the Itunes folder for duplicates and found it to be clear. You might also check for duplicate album cover Jpegs and other files using the same methods.

Good luck


BH

I just used the DeDup script on nearly 6000 duplicate files. It worked like a charm. And I don't know a thing about "scripts." I just followed the directions and let the DeDup script do it's thing. It saved me hours, or days, trying to manually delete all those duplicates. I highly recommend the DeDuper.vbs script! THANK YOU!

Turing, I am having the following error when using DeDuper. Do you have any ideas of solutions to this? Thanks in advance.



User uploaded file

Now I am getting the following error too:



User uploaded file

Turn off the display of iTunes in the Cloud items (Edit > Preferences > Store) and/or repair permissions...


tt2

Lost & Found Playlists

  1. Create a playlistcalled Found (Use File > New > Playlist, then name it Found)
  2. Select everything in Music (use the Songs view) and drag it into the Found playlist (it may take some time to count the tracks that are to be dropped).
  3. Create a smart playlist (Use File > New > Smart Playlist) add a second rule by pressing the + symbol and edit the playlist so it matches the illustration below. Once you click OK to save the rules you can name the smart playlist Lost.


User uploaded file


tt2

With Music selected under Library in the iTunes source list, at the iTunes menu bar go to File and select Display Duplicates. Select and delete the duplicates there.

Spotted a race condition error and have uploaded a new build which also catches track ratings. Working on the next phase.


tt2

Thanks for the feedback. I'll have a look into that and get back as soon as I can.


tt2

Great, thanks for that. I'll post a revision up later. I'm on my iPad rather than at a PC just now...


tt2

probably my lack of tecnical expertise, but when I hit delete I don't think the underlying file gets deleted?

Yes, it is Option on a Mac, but Shift on Windows. This thread is primarily concerned with deduping in Windows. ๐Ÿ™‚


tt2

No I hadn't. Please see clarification.


Apple admins: Please remove my replies if this topic is meant to be closed.

one more try got me a new message again:

line: 1060

Char: 7

Error: The Interface is unknown

code: 800706B5

source: (null)

Just to confirm the latest version of the DeDuper script works fine under Windows 7 with UAC enabled.


tt2

I'm understanding that you are refering to the itunes folder within libraries>music. the folder that i have consolidated everything into.


all of my permissions are already on full control.

Excellent news. ๐Ÿ™‚


tt2

Thank you. Happy to help. ๐Ÿ™‚


tt2

Perhaps WSH isn't installed. See http://www.ehow.co.uk/how_7432762_install-windows-script-host.html.


tt2

That worked! Thanks so much. I just ran the program and it is deleted the dreaded duplicates as I type. Thanks so much, anonymous wizard from London! You're the hero this city needs.

Its working! Great script Turing, congratulations!

You're welcome. ๐Ÿ™‚


tt2

Thanks for replying. The Delete option is on the right click menu yes. Do you think it could have anything related to being windows instead of mac?

When installing deduper I get Windows script host error line 1072 char 39 expected end of statement code 800A0401 microsoft VB script comp error. How do I fix this

Language is Dutch. What should the edit then be?

Ok, thanks for all the help. You really are a great person.

You're welcome. ๐Ÿ˜Š


tt2

guys,

http://duplicatefilesdeleter.com is awesome... try it i guess it will help you out

shaimali wrote:


guys,

(a weblink) is awesome... try it i guess it will help you out

But that only deletes files in the file system. It does not delete them from iTunes.

Gemini will cost you ten bucks. But in my experience it's been worth it--on both of my machines. It's fast, colorful, efficient, accurate--and it has a safety feature.

I have a problem with the deduper script, it has an error message that comes up saying "activeX component can't create object: 'iTunes.Application'

code: 800AO1AD

Line: 595

Char: 3

You may need to perform a repair of the iTunes installation in order to register the automation object. What version of Windows are you running?


tt2

I'm running windows 7, most recent update.

Ah, I seems your suggestion worked, I did a repair installation via control panel and everything seems to be working fine now, thank you very much.

Excellent. ๐Ÿ™‚


tt2

Hi

OK - Checked, and they appear to be logical dupes.

i will dl your backup utility an give it a go!

thanks

Im afraid it hasnt worked, ๐Ÿ˜Ÿ it appears to have deleted many songs that were not duplicates...


How do i use synctoy to restore?


thanks

hmmm im afraid there are only about 8 i tunes in my recycle bin...

5640 sounds about right.


shall i use File > Add Folder to Library > Select iTunes Media and add them all back in from itunes on ext HD?

OK....How do i scan for orphaned files?

Ok, adding now... it is converting some at same time. it may take some time i guess. will report back when something happens!


thanks for your continued help.

You're welcome. ๐Ÿ™‚


tt2

No silver bullet I'm afraid. I guess you can import everything, run the Duplicate Song & Artist script, sort on date added and cream off the top...


tt2

No problem, thanks for taking the time to reply. ๐Ÿ™‚

You're welcome. ๐Ÿ™‚


tt2

At a guess it will be Option instead of Shift on a Mac. My scripts are for Windows, for Mac try Dupin or Dupin Lite from Doug's Scripts.


tt2

Unticked the cloud, trying it again... thank you for your extremely fas help, I'll keep you posted if does the trick...

tt2, thank you. That worked. In Danish it is "Sle&t"

You're welcome. ๐Ÿ™‚


Still haven't had a change to test it myself on Windows 8...


tt2

Hi, i am having an issue with the DeDuper script. After identifying what tracks need removed, the script tosses out an error

Line : 375

Char: 39

error: object required: 'T'

code: 800a01a8



help?

I never have cloud idems vissable, onand i have only been working in the main music folder. It comes up with the error reguardless of if i select 100 ttracks, or 2.

If you like i can try that. I am not much one with the coding, what should i use to edit the line?


<Email Edited by Host>

I am not worried aobut it. the email account is almost twenty years old so all the spam places pretty much have it ๐Ÿ˜ OK, i will go give it a shot and see if it changes anything.

I just wanted to say a big THANK YOU for that script. I had been dreading dealing with those dupes in my library and the script made them a breeze! You're my hero tt2!

You're welcome. ๐Ÿ™‚


tt2

When I go to the link to http://samsoft.org.uk/iTunes/scripts.asp the link for deduper takes me to a text file. What steps must I take to make that text file an executable script?

Right-click and use Download or Save As...


tt2

Has there been any fix for the Object T error. I am getting it too and tried the test scripted. It is erroring at line 350 now.


Thanks for the help.

You're welcome. ๐Ÿ™‚


tt2

Hi turingtest2,


Any luck looking into that script? Thanks

Hi tt2 - I am having an issue with line 467 and char 55. It says the track is not modifiable. How do I fix this?

Okay so here's the thing, with the recycle bin box unchecked when deleting files ON my laptop via Windows Explorer, I dont get a delet confirmation alert but do when the file is on the NAS.

Thanks for your script. It helped me get out of the mess I had created. You are a genius ! Is there a similar script or Program that works in Mac ?

You're welcome. ๐Ÿ™‚


I don't recall that prompt being part of my script. Was it perhaps a prompt from the Windows file system?


tt2

Yes, forgot to mention it. It was indeed from the Windows file system.

Alex

I wonder if those particular tracks were marked as read-only.


tt2

Thanks again tt2, worked like a charm! I was able to continue through several runtime errors with no problem.

Great. ๐Ÿ™‚


tt2

You're welcome. ๐Ÿ™‚

tt2

this is a very helpful post. i wish i had found it an hour or so ago.

Sorry I cannot turn back time. ๐Ÿ™‚


tt2

me too. but it is recoverable. ๐Ÿ˜

Good to hear. ๐Ÿ™‚


tt2

Try repairing iTunes from the programs and features control panel.


tt2

Perfect! Oh, so simple.


Thank you for all your hard work tt2.


Cheers!

You're welcome. ๐Ÿ™‚


tt2

Is your system language something other than English?


tt2

No, it is English.

I agree. I think I just couldn't get your script to work. Thanks though!

No worries, I'm glad you found an alternative solution. ๐Ÿ™‚


tt2

ok thanks I'll remember this for future reference. But as for getting them back what should I do

Is there a line number associated with the error?


tt2

I think the FindTracks might work better if it only searched for song name and artist. Some of the album's are different (IE Greatest hits) and I think that's stopping some of the matching.

You're welcome. ๐Ÿ™‚


I'm glad you got there in the end, though I would love to know what was tripping things up. ๐Ÿ˜•


tt2

Sorry for the delay. When using the FindTracks script are you pointing it at D:\Users\Michael\Music?


tt2

I Am. It fixed around 300-400 songs

Can you see any pattern to the tracks it won't fix? How many are we talking about?


tt2

How hard would the SyncStats tweak be?

1.Open the Music library in iTunes.

2. Sort your duplicate list.

3.Delete the duplicates.

All of them? Then there is nothing left! Do they go in the trash, or not? If it was easy nobody would bother asking, and nobody would get in a mess when they get it wrong.


tt2

SyncStats with that tweak worked.


929 were processed of which 874 were updated and 55 not found.


My VB isn't good, is there a way it can print the 55 missings?

I'm away from a computer for a couple of days (posting from an iPad) but some kind of log is a great idea. Might help with troubleshooting the other issue too.


tt2

I tried your deduper script and got the following error:

Line: 587

Char: 11

Error: Permission denied

Code: 800A0046

Source: Microsoft VBScript runtime error


Any suggestions? Thanks!