I agree with your doctor. Intermittant ADT has been shown to be superior to Constant ADT in patients with biochemical recurrance. My limited understanding is that resistance occurs because the tumor devises a way to create it's own testosterone.
Stopping orgovyx looks like an opportunity to see if you need the medication and to delay castration resistance if you return to the medication.
You do have to keep careful track of your PSA.
Bless your choice.