I understand the anxiety about having another seizure. For me, ultimately the only thing which helped was good seizure control. But ongoing fear is what keeps me from enjoying that cup of coffee, even now because caffeine is a seizure trigger for me.

I think that some anti-anxiety meds can actually lower your seizure threshold. And maybe you have already thought of this. But I would suggest that an epilepsy specialist, not a general neurologist, be consulted.