Am I wrong to be mad?

I have a 2 month old son. Everytime I go to my sons parents house his grandfather (my boyfriends dad) calls my son a sissy when he cries and says he will beat him to show how not to be a sissy and that when hes older he will give him a fat lip to show him how to be a man. My boyfriend says that hes joking which I know he wouldnt reallly do that but I still think its wrong to say things like that and to call him a sissy. He calls his other grandson who is 3 years old a sissy and I do not want him doing that to my son. My boyfriend says im over reacting that hes joking but its everytime we go there..its so negative he can never have anything nice to say and I dont like it..Am I wrong?