Yes, there's no default safe choice with prostate cancer, just different sets of risks to choose from.

If I were 76 with non-metastatic prostate cancer and low PSA, I think I'd opt for an ADT holiday as well. The health benefits in preserving bone and muscle density and heart health (as well as the reduced diabetes risk) may well outweigh any dangers from faster cancer progression.

Best of luck!