iCloud Drive files live on the Mac, and they are mirrored to iCloud Drive at iCloud.com. If you have Optimize turned on then, to save storage space, you have allowed the Mac to decide to keep some lesser used files off the Mac. Is that you you've done-- chosen to Optimize your iCloud Drive?

As you can see, I don't do that, so I don't have delays. Of course, I do have short delays if I use those files on my iPad, which is optimized. The delays are typically less than a minute-- maybe a couple of minutes for large Keynote files.
Are you needing to keep your files off the Mac? What is your storage on this Mac? At System Settings>General> Storage, my Mac says 745 GB of 1TBB used. What does yours say?
Even if you have your Mac Optimized, you can choose to keep some files local and remove others by right-clicking the file:

In general, to get quick downloads you need good wifi and internet connections. And you may need to leave the device (especially mobile devices) alone so that they don't think that they are slowing down your work by accessing the internet.
See this:
Understanding iCloud Drive - Apple Community