Wondering if I really need to be on Xarelto

I'm a 63 year old male on 20mg Xarelto for 5+ years now. I've always been very healthy (other than some bladder stone issues) and fit but in February of 2020 I got very sick. One night my heart just felt "wrong". Honestly can't remember the exact symptoms but I know it wasn't normal for me. Went to the ER with my wife. They checked me out and said I was fine.

The next three weeks I felt sicker and sicker, though no heart symptome. At first thought it was just cold or flu but I ended up spending two of those weeks completely flat on my back, so foggy I couldn't even read or watch TV. Halfway in I temporarily felt semi-OK and was worried about missing so much work so I went into the office, but nearly vomited on the bathroom floor and then barely dragged myself back home. Finally after a few more days I noticed my right calf was red and hot.

Back to the ER. They found me positive for A strain flu, a DVT and pulmonary embolisms. I got the heparin injection. My blood clots made so little sense to the ER doc though that he ordered a genetic test and I came up positive for Factor V Leiden, heterozygous. I got a prescription for Xarelto, told I had to be on it for life. Back at work 2-3 days later, recovering, but felt wobbly for maybe another two weeks at least, though able to work.

Later I suspected that although I'd tested positive for flu I might have had COVID-19 too ("flurona" as they say). Never occurred to them in the ER, but it was mid-Feb. 2020... and they had no test kits for COVID yet. I got a COVID antibody test as soon as they became available (I wanna say early June?) and it was negative, but the antibodies don't stick around very long, so even if I'd had it I still could very well have tested negative. My current doctor agrees that I could have had COVID.

Now I'm wondering if I really still need the Xarelto. My GP (osteopath) isn't willing to take me off. I have a referral to a hematologist to get his opinion, but he is booked up for quite a while and I won't likely get an actual appointment for at least another 6 months.

What I read online suggests I may not need it. I'm at a healthy weight, exercise 5-6 days a week, eat healthy, have never smoked, drink only rarely, and have no other health problems.

I WON'T go off the medication without a professional blessing, but I'm curious about what some others' opinions are, or what their experiences have been in this regard.