What can i do about this relationship problem?

This may sound like im being a drama queen or something but im not too sure,I work 5 days a week and am usually in bed by like 11pm because im always knackered and home at like 9 and my boyfriend who lives with me comes to bed at around 4am or 5am or something like that but recently he seems to spend all his time with his friends he never used to see his friends much but now suddenly he does and tonight we had a compromise that he would go out in the afternoon while i was at work and then be home at 6 i text him to ask him to pick up coathangers so he has just came back and then said im just going to get your coathangers and maybe for a drink.......Is it okay that im back at 9.30 now i wouldnt mind if he hadnt have been out alot recently or that we had made a compromise so now he has had ago at me for making me feel guilty and has stormed out not without giving me a kiss and saying that he loved me but you can tell he is really pissed off at me :( he just doesn't understand why im upset and he just says "have you gone insane" if i start crying or something the thing thats got to me is just the fact that we had a compromise and he has now gone back on that because he says it would be nice to see his old friends again that he hasn't seen in ages and i agree but because we had a deal im upset now and just feel like im being really immature seeing as in 21 not 15 :P but i just dont know what to do btw we have been together 3 years it isnt like a new relationship