@bruceg
Memory problems are extremely common with Epilepsy, more often because of the medications.
Overall the brain is quite resilient. Although memory issues are not limited to only severe or frequent seizures memory damage is possible with any seizure.
My memory is horrible so I can relate. I have total amnesia of my first 15 years because of Convulsive Status Epilepticus (continuous state of seizure) and being in a coma for months.

Here is a bit of info on memory from the Epilepsy Foundation.
https://www.epilepsy.com/learn/challenges-epilepsy/thinking-and-memory/types-memory-problems.
Here’s something from Epilepsy Society in memory.
It explains a little about the temporal lobes.
https://www.epilepsysociety.org.uk/how-epilepsy-can-affect-memory#.XOg3KRZlDDs
What type seizures do you have Bruce, Tonic-Clonic, Focal or something else.
Jake