I can certainly respect your choice of not taking AI, but I would question wether or not you need to try and look for a new oncologist. It is possible he got you mixed up with someone else but in oncology, that could be really bad if not for you speaking up.
I am guessing if you have a 62 year old son you don’t have a ton of estrogen naturally to be concerned about. With a small tumor and no history that should further give you peace of mind.
I could not imagine doing this without my husband. My thoughts are with you. When is your next appointment with oncology?