kjc, acknowledging the wisdom in what you say about boundaries, I would yield to the doctors who "clearly" state that exposure to stress is harmful to this patient.
These are not the children but the brothers and sisters of the gentleman, probably the eldest whom they have always depended upon for help with those physical tasks.
It can be irresistible to set down boundaries with someone finally, when they are disempowered by dementia.
You can set those boundaries repeatedly, because they won't, can't remember, even if they understand in the moment.
When you have all the power, you lose nothing by being gentle. But, the gain is garnered by everyone.