I am sorry you are not doing well with your doctor. Having a doctor who you feel comfortable with is important. Yes it does take a while for your body to adjust after discontinuing a benzodiazepene. They can have a long half life and so some can stay in your system. Then your body goes through an adjustment to deal with no longer having the medication. There can be a psychological association with this as well as physical. You now have to deal with relying on yourself rather than medication. This stress can cause forgetfulness. I found that getting rid of my benzodiazepene actually increased my attention. I am retired now so did not care if I had a stressful situation. I just said I can deal with this at home.
You may not have that situation so maybe therapy would help. Talking can help reduce stress. Hopefully your talking about this here is helping some.
Also some antidepressants can cause memory loss. So it may be the antidepressant. Depression can also cause memory loss so maybe can get an increase from your doctor. These are all things to discuss with a doctor and hopefully you can talk to your doctor about why you feel he is not as helpful as you wish. Maybe you two could work on improving your communication. If not I hope you can see the other doctor. And something I do not have, patience to wait, may help as it may just take longer for you to get over the withdrawal and longer for the antidepressant to kick in. Please let us know how you are doing as we do care about you.