It's an interesting question.

I am from a small family, no younger siblings, no cousins, nieces, or nephews for me to be around, or to be around me. My wife and I don't have children.

Also, I was a bachelor for a very long time. Bachelors are typically unwelcome at young family gatherings; at least I was unwelcome. I never had many opportunities to be around very young children.

Additionally, I tend to scare young kids. Even when young, I was a big clunky guy. Kids' usual reaction to me was to run away, screaming for mommy. (Oddly enough, a lot of young women had the same reaction.) Now that I'm older and disabled, I walk with a cane and move very awkwardly. Most little kids have never seen anyone like me. Hell, most adults haven't.

Finally, I am great with animals -- dogs, cats, and even snakes like me. But with kids, I get very anxious. Since becoming disabled, I don't have two arms available anyway. It wouldn't be safe for me or the kid if I tried picking one up.
(Your comment about me not having arms kind of hit home, though I'm sure you meant no harm.)

Oddly enough, older kids seem to like me. My wife tells me it's because I don't talk down to them. I talk to them like they're grown-ups, which they like, apparently. (I married into a *very* large family.) But that's completely hands-off.