So here's a rule of thumb: if someone gets called out on a bigoted "joke", and your response is either "it's just a joke" or "don't make such a big deal out of it", then you are basically *guaranteed* to be on the wrong side of the argument, regardless of what the joke was.
And to be more explicit about this: if you do not have something supportive to contribute to the victim of the joke, then SHUT. UP. Your arbitration of merit is not required nor wanted.