Men can have high (but not super high) PSA from benign prostatic hypertrophy and from stimulation of the prostate from direct pressure on the prostate (cycle riding or sex). But a continually rising PSA or a year over rise of >20% is concerning for prostate cancer. Yes, you are correct that most men will die of something else than their prostate cancer. About 80% of prostate cancers are of the more indolent variety that usually doesn't kill the patient. But 20% are aggressive and are linked to rising PSA. Those cancers will metastasize and kill you. There are very good treatments available today. I hope you don't have the disease but putting your head in the sand, misinterpreting what your urologist is saying is not the best thing for your health.

Good luck whatever you do.