Saw Palmetto Withdrawal Symptoms

I am a 25 year old healthy fit male. Was taking Saw Palmetto pills for about 4-5 years to reduce hair loss due it blocking DHT. I've had testicular swelling with pain/soarness for about 5 months. I decided to quit taking the supplement and noticed an increase in pain as well very low Libido and impotence since then. It's been about 2 weeks. I've read of other people having similar experiences coming off Saw Palmetto and some of the consequences have been long term. Should I go back on the supplement (maybe once every 2 or 3 days to ween off) or stay off it and hope my symptoms subside soon.