It's always a good idea to exercise moderation, and limiting red meat is good for colon health. High consumption of processed meats does have a strong correlation with many cancers, so it's best to make them a "sometimes" thing (e.g. go ahead at eat the hotdog they offer you at the church picnic, if you want, but don't have hotdogs once or twice a week at home).

If you're going with plant-based milk, oat milk is relatively inexpensive here in Canada, at least, has a good consistency, and is more environmentally-friendly than other plant milks (oat plants need little or no irrigation, for example, and they have low fertilizer requirements; they're basically weeds). But regardless, make sure you're still getting your calcium and protein if you give up dairy, because ADT very often leads to muscle and bone-density loss.