Osteoporosis DX and Overwhelmed

My recent dexascan showed that osteopenia has progressed to osteoporosis in the lumbar spine and wrist. I am 73 and taking Anastrazole following a breast cancer dx and lumpectomy 2 years ago. The Anastrazole further blocks estrogen so that is likely a big part of the progression.

I do not want to take any of the meds but confused as to what else to do since there is so much conflicting information. I stopped taking a calcium supplement and now just take a multivitamin and D3+k2.
I walk every day and do some core exercises. I have cervical stenosis so I avoid lifting anything heavy.
I have been trying to get calcium in my diet- almonds, prunes, almond milk, etc but now i read that some foods contain oxalates that block calcium absorption.

My doctors say I am not doing enough but at this point I don't know what is best. My Gyno recommended Akgae Cal but i have read conflicting information about that.

I have read horror stories regarding the bone meds and refuse to go that route. Many people say they have not had success with diet and exercise.
I am at the point where I am ready to just take my chances and hope for the best as foolish as that may sound.