Some people have no or little side effects from these drugs. Especially since most of the worst side effects seem to be the ones related to menopause, you are surely post menopausal already. If the doctor recommends it, I would try to keep an open mind. Talk to your doctor and you might find peace of mind one way or another.
How long is your course of radiation scheduled for? Did you have a lumpectomy?