Welcome, Yoko Msps.
First, it's a personal statement of principles for me that no one should feel uncomfortable at my gaming table. This means, among other things, that we will keep the tone of the game at a mutually acceptable level. I say this to argue this point: I share the opinion of your instinct that something is wrong (because of your discomfort and that of other players) and must be corrected.
Secondly, I think you did at least an extra mile with this player: you set out your guidelines, you saved and rerun your guidelines process to include it, you got it okay, and it was Violated again. It does not look like minor offenses.
Third, its justifications are a pure defense "My Guy", a variant that only covers anti-social behavior. If your instincts should ignore this defense, these instincts are also good instincts.
But in my experience, punishing players almost never works. This leads mainly to a conflictual position between the GM and the player. Indeed, the larger the offense, the more the sentence tends / must be strong, but this simply tends to deepen conflicting positions. I can not easily remember a situation in which a GM has managed to break the will of another player and I can not imagine that it would be so satisfying.
I guess you can let the world of the game punish the character (rather than punishing the player directly) if the character takes criminal, anti-social, or dubious actions in the game. But I have to be honest, it usually works the first time or everything goes wrong, according to my experience. Since you have already warned the guy, I do not see this as a great chance for success.
The only thing I can think of, if you really want to give this guy a second (or third) chance, is to let him ("leave") make a new character that is not so disturbing. At some point, though, you must be prepared to use the last resort, to get the guy off the table.
Finally, to answer your question, no, I do not believe that the fault is your fault for lack of severity. You should not be forced to do like a chariot just to make a point on the civility and standards of the table.