Can I switch off or lower the fan on the exhaust air heat pump?

Yes, you can do that, but it can affect your indoor climate and the ventilation adjustment that has been made.