Spotlight not finding all files on hard drive after reindexing

jinet wrote:

I don’t think I understand the connection between the process that you killed or fixed and spotlight functions. Is there one?

Probably not. I was really just going off on a rant about poor software quality. I had lost 466 GB of free storage in a few hours yesterday. That's how I noticed this problem.

However, when I was digging through console output, I did notice that dasd was doing a lot of work with Spotlight. That's not surprising. Spotlight is very deeply embedded in the operating system. It's all connected.

But I'm back at my Sequoia computer today. So I took another look at this folder. After poking around a bit, I think I've found the solution. The files I have in this folder are very old, circa 2007. And I think that was just the last time I copied them. They're older than that.

I checked the one file I can't find and I saw that it had Finder Info attached:

$ ls -al@ Desktop\ Danger.jpg -rw-r--r--@ 1 jdaniel staff 520802 Nov 1 2007 Desktop Danger.jpg com.apple.FinderInfo 32

When I inspected, it looked like corrupt data:

$ xattr -p com.apple.FinderInfo Desktop\ Danger.jpg ???????? $ xattr -px com.apple.FinderInfo Desktop\ Danger.jpg 3F 3F 3F 3F 3F 3F 3F 3F 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Since the metadata is obviously bogus, no harm in deleting it...

$ xattr -d com.apple.FinderInfo Desktop\ Danger.jpg

I tried it again on a similar file that also wasn't findable. In this case, I had an active Finder Spotlight search window up with no results listed for the file name. As soon as I removed the Finder info from the file, it showed up in the window.

So I guess the problem is solved. Now I just need to find all the files on the system with corrupt Finder info. Peachy.

Your experiment is the best clue here. If the files turn up as soon as the subfolder moves out, Spotlight isn't skipping those files, it's skipping everything inside the original parent folder. Two things do exactly that. One is the privacy list in System Settings > Spotlight > Search Privacy: anything on it is ignored along with every folder inside it, so check whether that parent, or a folder above it, ended up there at some point. The other is the name: Spotlight won't index a folder whose name ends in .noindex. Click the parent, press Command-I and look at the Name & Extension field to see its full name. And Alfred reads Spotlight's index, which is why it misses the same files, while Find Any File searches the disk itself and finds them.

— Dmitry · developer of ComfortFiles

jinet wrote:

Thanks. I use FAF looking mostly for contents of file and have used EasyFind and FAF for other locating file names (rarely, though). Yes, FAF locates the file I was seeking with Spotlight, but it is awkward: I have to search somewhere in the path that contains it (that's how FAF works). I was surprised that Alfred, my regular choice for finding files/folders by name, could not find the file, neither could Spotlight.

I use Find Any File when I don't get good results quickly with the Finder File=>Find menu item, which for me almost always works quite well. But Find Any File always works, so it's my back up. Also, I always search over the entire drive, I don't specify any paths or limitations.

Since you are having trouble with the built in tools, I would just use FAF. As Etresoft has shown, getting to the root cause can become complicated and might have a solution that is particular to a specific Mac. FAF has always worked quite well for me.

Try searching from the search bar in Finder. The results are more easily organized.

I don't think that Spotlight is a very good search app. I use EasyFind (web download) and others have recommended Find Any File (App Store - paid).

You could use Terminal to check the detailed permissions of the directories or the files inside them which don't copy to see if there's anything obviously different between those that copy and those that don't.

I'll assume you're OK with Terminal. If not come back.

ls -l

You could also check if there are any hidden dot files that might be messing up spotlight. I don't know if this is even a thing, but spotlight's a law unto itself so who knows.

ls -a

You could try adding the problem folder to Spotlight's privacy list, letting Spotlight re-index, then removing it. This is the 'standard' fix Spotlight failing.If this is what you mean by re-indexing the folder then I'm no more help. I'm sure that if you've done any searching for Spotlight fixes you'll realize that there are hundreds of posts about how to get it working but they are mostly different ways of rebuilding the index and if you've tried that I'm out of ideas. It does seem that for some people Spotlight gets stuck in a rut.

As to what it's for? I think of it as a nanny search. It doesn't find system files or hidden files or anything that a non-tech user might tinker with and mess up their Mac's config. It's also the background in-app search - so when you search in Notes or Mail it's Spotlight doing the legwork - which hardly inspires confidence.

Sorry I'm not more help.

jinet wrote:

But finder searches consistently miss a handful of folders (this has been true over several generations of MacOS) while it mostly gets the rest right.

Yes. I see the exact same thing. I have one folder where Spotlight doesn't work. I can duplicate the folder and Spotlight works in the duplicate, but never in the original.

So one might assume that I could just duplicate this folder, delete the original, and rename the dupe - problem solved! But how would I know? How do I know there aren't other folders where files aren't being found? Because I know that it's possible for one folder to be mysteriously excluded, it's also possible for 1000 other folders to be excluded. I'll never know because no results looks the same as not working.

Is there an explanation for why this small number of folders evades finder detection consistently over five or more years? It's not the files themselves but some structural feature in the directory in which these files live.

I've never found anything. In my case, I have the same folder on another computer and it works fine there. Like your folder, it's old. I have documents from 2007 in there. The machine where it doesn't work is Sequoia, whereas this one, where it does work, is macOS 27. I could easily fix it on Sequoia, but I don't on principle. It reminds me that Spotlight isn't reliable. I would rather know something isn't reliable than rely on it.

Thanks. I use FAF looking mostly for contents of file and have used EasyFind and FAF for other locating file names (rarely, though). Yes, FAF locates the file I was seeking with Spotlight, but it is awkward: I have to search somewhere in the path that contains it (that's how FAF works). I was surprised that Alfred, my regular choice for finding files/folders by name, could not find the file, neither could Spotlight.

But the problem is slightly different. It's not just about how to find a file, but why certain files are unlocatable.

Certain files—all within certain folders— do not respond to Spotlight or Alfred—very few, but a small handful. That is, any file in those folders will not appear in either search method. Nothing I do changes this (copying, renaming the copies, reindexing the folder). What DOES change things is moving the folder containing the files to another directory—which makes me think that the folders in question happen to contain hidden code that interferes with searches.

So my questions are: why would this happen? is there a way to locate any offending culprit (if there is one) and to eliminate it? That would make searches work.

Then: if Spotlight is so poor at locating files, what is it good at? (I care much less about this question.)

Thanks.

that is what i meant by re indexing.

my main question is the first one. Why is this happening with some folders and not others.

I did an experiment, and moved the sub folder to a new directory, and then searched with Alfred. Bingo. It found the file. When I put it the sub folder back in the original folder, Alfred no longer could find the file. This happens with only a few folders, which is why I am wondering what might be hidden in their directories, some flag or something, that prevents discovery using Finder apps.

jinet wrote:

Spotlight finds some but not all things on my HD and I don't understand why

Because it's junk.

Next question, please.

Shouldn't Spotlight shine in these searches?

There are many substances that may shine or at least glisten in the light. That doesn't mean they're good for you.

I'm beginning to agree. But finder searches consistently miss a handful of folders (this has been true over several generations of MacOS) while it mostly gets the rest right. So this is the other part of my question:

"This happens with only a few folders, which is why I am wondering what might be hidden in their directories, some flag or something, that prevents their discovery using Finder apps."

Is there an explanation for why this small number of folders evades finder detection consistently over five or more years? It's not the files themselves but some structural feature in the directory in which these files live.

BTW, finding these files (which actually are living in a Box directory) through Box's web interface works perfectly well.

tx

I've ruled out both possible offenders: nothing is marked as Private in Search Privacy in Spotlight in System Settings (placing the folder there and removing it, forcing a re-index, ends the same way). And the folder names do not have that extension, at least not that I can see with "Information". So there may be some other hidden flag somewhere. Changing my folder structure is not a great option, since that runs alphabetically. The parent is "Arendt, H." Not many ways to change it w/o disrupting the existing folder structure.

There are other peculiarities with Alfred (not sure about spotlight): it reads letters in file names (say) sequentially, ie if the author appears first in the file name but is entered last in Alfred, it draws a blank. This is not my current issue, however, just noticing it on this discussion board.

Interesting that you have found this behavior too. You write (not sure how to copy it as a quotation): "So one might assume that I could just duplicate this folder, delete the original, and rename the dupe - problem solved! But how would I know? How do I know there aren't other folders where files aren't being found? Because I know that it's possible for one folder to be mysteriously excluded, it's also possible for 1000 other folders to be excluded. I'll never know because no results looks the same as not working."

I know which files I am seeking and not finding by title, so there is never a doubt. I don't do much content searching with Spotlight, none with Alfred, only with FAF, which works, and sometimes with Box's web interface, which sometimes works.

jinet wrote:

Interesting that you have found this behavior too. You write (not sure how to copy it as a quotation):

That's the quote tool:

I know which files I am seeking and not finding by title, so there is never a doubt.

Same here.

I don't do much content searching with Spotlight, none with Alfred, only with FAF, which works, and sometimes with Box's web interface, which sometimes works.

I do sometimes use the content searching feature. I know I've written some code to do something years ago, but where is it? I would like Spotlight to work, but it simply doesn't.

This is just standard, buggy modern macOS behaviour. I just had to restart my computer because I mysteriously lost 500 GB of storage in a few seconds. Luckily a reboot fixed it! But still strange. Maybe it's related to that "dasd" Apple process that's been taking 100% of my CPU for quite a while now? I have no idea what this thing is doing.

etresoft wrote:

Maybe it's related to that "dasd" Apple process that's been taking 100% of my CPU for quite a while now? I have no idea what this thing is doing.

I fixed it!

The fix was very simple, all I had to do was run the following Terminal command as root:

kill -9 <dasd PID>; rm -Rf /private/var/db/DuetActivityScheduler; rm -Rf /private/var/db/powerlog/Library/PerfPowerTelemetry/BackgroundProcessing; rm -Rf /private/var/db/DuetActivityScheduler;

Who says macOS doesn't "just work"?

My problem is not that I cannot find a single file in a directory but that the entire directory (a folder's worth of files) is invisible to Spotlight. I know of two such folders that have always, it seems, defeated detection in Spotlight. I'll just live with it.

I don’t think I understand the connection between the process that you killed or fixed and spotlight functions. Is there one?