It's fine to run a Photos Library from an external drive, but as the message indicates, the drive must be formatted in either APFS format or Mac OS Extended (Journaled) format. The drive must be connected directly to the Mac by cable, not networked, clouded, NASed, etc. Additionally, the drive can not have had Time Machine on it since it was formatted. See this for more:
Move your Photos library to save space on your Mac
Older OSs didn't check to see if the Library was on ExFAT, and the Libraries would slowly corrupt, becoming totally useless. So it's lucky that Photos stopped you in time!
Libraries can be stored on ExFAT--but they can't be used from there. So you Library is probably OK. But you will need to copy it to another drive or volume. Or, you copied this Library from a previous drive-- is that right? If so, you can re-format the new drive, erasing the copy, and copy again.
To format a drive, you use the app Disk Utility. You choose the drive, the Erase, and then choose enter the name and format:

If your new drive is solid state, then certainly choose APFS (not encrypted or case sensitive. If it's a spinning drive, you can choose Mac OS Extended (Journaled).
Here are Apple's instructions:
Erase and reformat a storage device in Disk Utility on Mac
Keep me informed!