This is a fascinating subject, my childhood was also filled with emotional stress and some physical. However, working as I do in mental health I speak to people day in day out who were subjected to similar, or often much much worse, and do not have IBD, my own brother also has no signs of IBD.
Inflammation however does seem to be linked. I've been following the work of Andrea Danese and others who seem to be finding correlation with childhood abuse and higher inflammation markers in adulthood and impaired physical stress responses.
/kclpure.kcl.ac.uk/portal/andrea.danese.html Interestingly, neglect seems to have the highest impact of all. I often wonder if neglect would go along with a very poor diet which might stress the digestive system from early on.
The growing number of babies and very small children with IBD from families with no obvious signs of neglect and abuse also points heavily back to the environment I think.