Long Term visits

As a caregiver I moved my Father, who had Cognitive decline/Dementia, into my home. He lived in Colorado & I live in Massachusetts. He has adapted well. I found him neglected and financially abused by family members while he was living in Colorado. So I moved him in with me and received Guardianship. We have a lot of support services in the area for Dementia patients. The issue at hand is the demand from family members to bring him back to his primary home, which is being maintained for him by a family member. I am reluctant to take him back for visits, as his confusion increases with any change in environment or change in schedule. I have tried to keep zoom visits with family members available, but he is hard of hearing and finds them frustrating and then this progresses to sadness and anxiety. I have suggested that these same family members plan a trip out to Massachusetts for a visit, but they feel I should bring him back since I removed him from their area. How do I handle this the best way for my Dad? I truly want him to maintain connection to the family members, (his great grandchildren and persons not involved in the abuse & neglect.) but traveling with him and changing environments I fear with accelerate the confusion and Dementia. I need to do what is best for my Dad but feel I can't protect him from the abusers if I take him back into that environment. I really don't want to isolate him from them, but he has developed so many friends at his day care facility here, and he is flourishing, a side of him I never new existed, i.e., artistic talents due to the activities at the day care facility. I don't feel he is being isolated in his current environment, but the demands and guilt keep popping up. Is this just the fine line a caregiver has to walk? I am new to taking care of a person with Dementia, although not new to caretaking. Any suggestions on how to handle a very difficult situation would be welcomed.