That was one of my competing interpretations of that scene. The other was that Jim was annoyed at Vera for being pissy about Blair and was standing up for him by giving her a catty backhanded compliment. But I guess don't really think he has enough interpersonal finesse to pull that off.