We wish you well with that. Yeah, you definitely want to hang onto that organ if you can.
I too have run into surgeons pitching the line: "No problem; you can live without that." Easy for them to say: they're on the right end of the scalpel, and THEY don't have to live with the outcome. Not that they care: surgeons think with the tools they got, which means a scalpel. If they can't cut it, it doesn't exist & doesn't matter: they don't care about it. That's someone else's problem.
I lost my spleen in a bicycle accident in my youth. I understand you can live without certain organs - that one included - but there are DEFINITELY outcomes. Problems. You wouldn't choose to lose ANY organ if at all possible. There is, after all, a reason why they're there.
Regulating your immune & hormonal systems - which regulate everything else - without your adrenal - or thyroid - glands, is problematic at best. Not what you want.

Don't know about adrenals, but if the problem is a growth it prompts me to consider nodules in the thyroid. Just as you've found, surgeons are quick to whip it out, leaving you attempting to balance your systems on drugs for life. Even investigative procedures can frequently (50%) result in loss of the organ. I had a large-bore needle biopsy done. Why can't they treat it the same way we do plantars warts? Inject a little liquid nitrogen, freeze/kill the nodule cells & flag the thing for clean-up by your immune system. "No way anyone would do that!" was the response. So I hit Google: turns out they DO do that, first in Turkey, then Israel, now it's spread to Germany. We'll do it in the US then Canada in another 5-10 years, because... "weird", "we didn't think of it", "it's different", and (my favorite) because "it doesn't use my scalpel." < sigh>