Memory problems are extremely common with Epilepsy, more often because of the medications.
Overall the brain is quite resilient. Although memory issues are not limited to only severe or frequent seizures memory damage is possible with any seizure.
My memory is horrible so I can relate. I have total amnesia of my first 15 years because of Convulsive Status Epilepticus (continuous state of seizure) and being in a coma for months.