The GP is saying that is coloured how he interacts with women. I interpreted it as being in the context of work. That is not the same as refusing to work with an entire gender.