It's just an anecdote, but: I remember playing in a band with an amateur musician friend who told me had perfect pitch, and it was "very annoying", according to him, when we transposed songs to fit our voices. They just "sounded wrong" to him. He would make beginner mistakes since he relied on pitch memory rather than listening to us to know what to play/sing.

I have no idea how it works in general, but it seems like it was a problem for at least this one guy.