Joe is upset because…

the other men are teasing him about his wife's affair with Spunk.

the other men keep trying to talk to him, when he just wants to relax.

the other men repeatedly ask him about his wife, when he doesn't have one.

the other men know he is jealous that Spunk married the woman Joe loves.