I have IBS-D and also dairy intolerant. I can have milk products if baked in something but can't drink milk or eat yogurt. I have been drinking soy milk for years and before that rice milk. I also drink oat milk which to me tastes better then almond milk. Maybe you should have a work up with a gastro. I also have to go along with the FodMap diet. Ever since I was a kid I knew certain fruits and dairy products bothered my gut.