These are my thoughts based on a lot of reading, but with no real expertise backing me up:

There is little or no danger associated with apixaban. It is largely, and almost universally, well tolerated, and it only retards clotting, it does NOT prevent it. This is exactly what you want, and is what it was designed to do.

The medical community goes largely by your CHA2DS2- VASC score:
https://clincalc.com/cardiology/stroke/chadsvasc.aspx
Once your score rises to 1 or above, they get edgy and want to use caution. There are ethical and practical reasons, with the latter meaning liability...and not wanting to break their Hippocratic Oath...to do no harm. Negligence IS harm.

After any amount of AF, your heart begins a slow or a faster process called 'remodeling'. Both its substrate and its morphology begin to change as an adaptive response to the stresses the arrythmia imposes on the myocytes and the valves affected, particularly the mitral valve between that pesky left atrium and its left ventricle neighbor. There is also fibrosis, or simply...scar tissue. This happens inside the atrium, on its inner surfaces, where the strains and stresses are keenly felt. Fibrosis makes the task of fixing you via catheter ablation more difficult. Also, if left to run unchecked, your atrium will enlarge, and that, too, causes fibrosis.

Further, with even a short and distant experience with episodic AF, you run the risk of stroke. In fact, the cardiology field feels that most strokes that happen any time within about six months (!!!!) of your last AF episode, can be attributed, in all probability, to the AF. Not certainly, but the stats suggest rather strongly that,. once you show AF, you go on apixaban or equivalent for life. Some EPs will let you, or agree to let you, go off NOACs (new oral anti-coagulants) if you have no further arrythmia...AND...you have a non-leaking, firmly closed, Watchman device inserted into the left atrial appendage. This is typically after you have had a confirmation TEE (trans-esophageal echocardiogram), and that rarely happens before six months have passed since the installation of the Watchman.

With all that firehose material, may I ask, and I'm not being snide or prying, but why do you think you should not take it?