Sicetnon3 makes a good point. In the absence of any medical or drug related allergies, it could be psychological. It does NOT mean that you are mentally ill!!
However, anxiety can manifest all the symptoms you describe even if you yourself do not feel anxious. Could be you are thinking about your PCa subconsciously - a form of PTSD in some people.
Radiation can do a lot of nasty things but some of your symptoms/feelings are outside the field of treatment so that shouldn’t cause any issues.