I hear you on your situation.
Yes, we don't know if the cancer will ever return. My feeling is IF an AI is recommended after a post mastectomy/lumpectomy, I would give them a trial despite all possible awful side effects.
I am betting for one story of success with AI's, there is story of a failure, re-occurrence happens. We are all between a rock and a hard place. Our oncologist's can not predict the future, they simply give their best evidence best recommendations and we roll the dice.
Don't obsess about the past. You made your best decision, at that time, in the moment. What's done is done. Now move forward as best you can. The very best of health to you. Live in the present moment. That is all anyone of us have anyway.