His dementia making me feel ill

We live in a fairly small, rural community. My husband with Frontotemporal dementia likes to go for long walks everyday, up to or exceeding 5 miles daily. I don't walk with him due to walking challenges due to a car accident.
Since we are a small town, the small town grapevine was working, leading to my niece calling me to say one of her clients told her my husband has been trespassing and peeking in windows while on his walks.
This was of course disturbing, and has made me feel sick. I hope I don't get an ulcer with all the worry associated with his dementia.
So yesterday I walked with him, and it was frustrating for both of us since I walk so much more slowly than he does. He can no longer be trusted to walk alone. Unless I can find a volunteer to walk with him daily, I need to make that sacrifice to walk with him (he is agitated if he doesn't get his walks).
I should now be used to switching gears and making adjustments, but this one hit me hard because his time away while walking was my time alone to re-energize and repair; now what?
Thanks for letting me vent.