I had a large ovarian cyst (18-20) cm removed, as well as a smaller one, both of which were benign. But the gyn oncologist and my gyn recommended removal, and a complete hysterectomy. I had prenatal DES (diethylstilbestrol) exposure, and a very slightly elevated CA-125 test. I was about 52. While your cyst is not likely to be malignant (that's what my radiologist said), my onc said there was a 20% chance of malignancy in part because of its large size and shape. Mine was increasing in size, so that's different from your situation. But perhaps yours could, and since you only monitor it at certain intervals, you might miss an increase. It would be better to remove it before it gets larger. And given your breast cancer history, I think I would seriously consider your docs' advice, especially since all of them concur.
If it is malignant (unlikely), it would be better to get it out sooner than later if it increases in size, when there might be complications.