He's not my boyfriend....Does he have a right to be upset?

just ask him to tell you the truth. Make sure that both of you are on the same page that you only want to be fwbs. If he really was mad it seems like he wants to be your boyfriend and more than fwbs. Either way, he shouldn't be mad that you were just going out for dinner with a friend to catch up. If he won't admit that he was actually mad just drop it, and if the problem persists be sure to confront him about it!