According to a piece in The Week there’s a correlation between the amount of time a man spends with his kids and the size of his boys:

“Researchers found that the smaller the men’s family jewels, the more likely their wives were to report that they were involved parents–spending a lot of time feeding, diapering, and playing with their toddlers…

Men with smaller testicles may compensate by taking more care with the fewer children they do have (compared to big balled men)….

But study author James Rilling, an anthropologist at Emory University, tells cbsnews.com, ‘It could also be that when men become more involved as caregivers, their testes shrink.”

I’d argue that marriage contributes significantly to shrinkage as well.