I can't advise you on your question, since it wasn't my form of treatment. I had 20 doses of proton beam therapy. I didn't have any reaction to the therapy until after my 5th treatment when I had some pain during urination. I don't think one treatment would have an effect. What I don't understand is that you said a one time HDR. There's a fair amount of prep for the HDR, if they are talking about either Photon or Proton radiation. You have to know the size and location of the prostrate and be able to aim the radiation precisely at the target. Also, they take turns radiating one side and then the other. Not sure how they would ensure that the whole prostrate would be covered. The LDR and HDR are very different types of treatment. It seems like they would do one or the other. By the way, the proton beam therapy is painless and takes less then a minute for each treatment. The only issue may be some urination pain that can be treated. I think the latest photon treatment is similar.
You mentioned that hormone therapy is out of the question. My Gleason score for my worst cancerous cells was 4+4. Yours would be considered medium risk and mine is high risk. Since mine was high risk my doctor at Mayo prescribed 3 shots of Lupron with my radiation treatment. My survival chances increased by 20% with the Lupron. Although you can get by without it, I think the research says it also improves the survival rate for your situation. The hormone therapy does have some side effects but they do go away and at least in my case were tolerable.

Best of luck with your decision. Ask a lot of questions and go with what you feel most comfortable with.