The calcium channel and beta-blockers all reduce cardiac output. The idea is to put the heart into a mild sedated state so that each beat is less forceful (so it's often prescribed for 'incipient hypertension'), but those beats also come less frequently, so a slightly slower heart. More of the drug each dose, the more the effect.

Some people don't seem to mind the drug and notice no real impediment. I felt some. I told my cardiologist that I thought some of my mojo was missing, but he immediately denied it and said I can't blame metoprolol for what I suspected was the new 'me.' Now, having read zillions of posts here and there about people's reactions to the blockers, I am convinced it's a real effect. And apparently, so are you.

You are always in the driver's seat here. If you feel awful on any medication, you have the choice to stop. However, to stay on the good side of any experts on whom you rely for treatment, you should explain carefully to them why you have elected to cease adhering to their considered professional opinion. They mad offer a remedy of an alternative that works for you...which would presumably lessen your awful anxiety. And, of course, you must then live with what follows....still your choice.