Safari not asking where to save downloads despite setting

Safari is downloading to the downloads folder instead of asking me where I want to download it, despite the fact that I have "Ask for each download" set in the settings. How do we fix this?