Sounds like you need some distance. You may win his trust back, but maybe not right now. It also sounds like some lines are blurred with friendship turning to sex right away--NOT trying to be judgmental but that just seems like it was all too soon. Sex complicates things, especially an already odd situation.
Have you spoken?
Another thing---you say "anxiety" ruined your relationship but I think it's important to acknowledge / realize at some point that YOU took part in this. Not just your condition. You can't wear it like a mask you hate and blame it when things don't go your way, yanno? We are so much more than our anxiety, even when it seems to engulf us.
I would focus on YOU. What you want. Not him, not that he's bad, but when you are whole you can bring that to the relationship. Sounds like he did nice things for you but may not all the things were nice. Anyone can bring you soup, yanno? Just my take. Hope to see you heal and one day in a healthy, satisfying relationship