I had proton radiation without medication with rectal cancer. I can attest to fatigue. It did seem to lessen after the first of six weeks.
With orgovyx I suspect a fatigue factor would increase gradually as you lose a certain amount of muscle. I think it important to take naps when you feel the need,but exercise can keep the tiredness at bay.
I probably disagree with your radiologist.