Thanks for the tip about the absence of a title in photos in Mac Photos app. I had not seen this photo title.
I added to photo "2017-08-27_123243.jpg" in Mac Photos app the title "2017-08-27_123243.jpg". When I sort by title in Mac Photos correct sorting takes place. When I sort by creation date, incorrect sorting still takes place. I think adding a title and sorting by the titles gets around the original problem.
Somewhere in the EXIF metadata in the photo there is an incorrect date and time that is being sorted by. I have tried many viewers under macOS and Windows. ACDSee Photo Studio 8 is the only one that shows "Date/Time: 05-09-2017" in the EXIF data, which differs from "02-09-2017" in other photos. The discrepancy is caused by editing this photo in the Photo Ninja app. Photos edited with different programs may have metadata issues.
After much searching with different tools, I found "ModifyDate: "2017:09:05 17:38:27" in the EXIF information. The XMP metadata also shows MetadataDate and ModifyDate incorrectly. I used the following command to set the ModifyDate equal to the DateTimeOriginal:
EXIFTOOL "-ModifyDate<DateTimeOriginal" /path/to/photo.jpg.
This fixed the original problem and my blaming Mac Photos app was wrong.
The search for the non-chronological order started by the iPhone Photos app. It does not show this order and does not allow adjustment in the iPhone app. When I noticed that Mac Finder and Mac Photos showed the same problem, I started working with those first. The problem with the iPhone Photos app was solved by syncing the modified photos between my Mac with my iPhone.
Thanks for the help.