I took 5 years of tamoxifen, followed by 10 years of anastrozole and although there were side effects, they were not debilitating.
I was in my 30’s, so my decisions would have been very different if I was in my 80’s. My decisions were helped by remembering that cancer has some pretty terrible side effects too. This made the hot flashes that last for hours more bearable and having to keep walking for the stiffness less irritating.
I know some women have very, very bad side effects, but there are more women who make it through the course of treatment, they just aren’t the ones typically posting on forums needing support.
Each of us has to decide what we can tolerate based on our own reaction to these medicines, but I would talk my doctor about quality of life before I give up on treatment. Likewise I would not refuse a treatment based on the reaction of another person. Maybe you will be one of those who has relatively no side effects and like @mir123 said you can always change course if it is too hard.
How long have you been on letrozole? How are you doing so far?