There are unfortunate problems that surface when people get into the neighborhood of 5 mg of prednisone after long term use. Prednisone supplies cortisol but suppresses your adrenal function. Your cortisol level is probably low at this stage of tapering. Your doctor is correct because it will be rough going at this stage because of adrenal insufficiency.
https://www.endocrine-abstracts.org/ea/0056/ea0056p44
The only way to get past this stage is to continue to taper slowly but some people are unable to successfully taper off prednisone completely.

That sick feeling you describe and overwhelming fatigue were my symptoms when my adrenals weren't producing enough cortisol. I also had muscle aches and pains but they were nothing like the pain I had when PMR was originally diagnosed.

Adrenal suppression is why we are told to taper off prednisone slowly and not stop it abruptly. I had adrenal insufficiency when I tried to taper off prednisone without a backup medication to help me control my inflammation levels.

Cortisol is a hormone that "regulates inflammation." . The backup medication that helped me was Actemra which kept inflammation in check while I tapered off prednisone. Actemra doesn't suppress my adrenal glands like prednisone does. There are other "steroid sparing" medications that may help. Hydroxycloriquine is one of them.

Fortunately, my adrenal suppression wasn't permanent and my cortisol level slowly returned to an "adquate level for normal days." My endocrinologist said I was still at risk on abnormally stressful days. It took about a year for me to start feeling better and my cortisol levels returned to normal.

Adrenal insufficiency and suppression of the HPA was hard for me to overcome. I took prednisone daily at moderately high doses for more than 12 years for PMR and other autoimmune problems. It is a complicated problem as the following link suggests ...

Good luck ... not everyone has a big problem. However, everyone will experience at least some problems after long term corticosteroid use.