You only remove penalties on shield attacks
Your understanding of the FAQ on the second answer is wrong, he says that it removes the penalties in case of attack with a shield, not when you attack by brandishing / carrying a shield. To clarify, let me quote again:
Shield Master: When the Shield Master says, "You do not incur any penalty for attack rolls made with a shield when you use another weapon," it seems that in the context, that means you do not take the penalty for two-weapon combat, but simply says "all penalties" so that it is not clear. What penalties does the exploit allow a character to ignore?
Shield Master allows a character to ignore two-weapon combat penalties the rollers of attack with a shield while brandishing another weapon, but no other penalties.
When you attack with a long sword, you are do not attack with a shield. These are weapons and separate attacks. Two-weapon combat penalties normally apply to your attacks by long swords, but not to those of your shield (thanks to the Shield Master).
This was clarified by Jason Bulmahn (lead designer at the time) before this FAQ and the feat errata, which said:
Does Shield Mastery remove penalties for all attacks if you use a weapon without a shield and a shield and two-weapon combat? Or does it only remove the sanctions for the attack of the shield?
Shield Mastery only removes the penalty for the two-weapon combat on the shock of the shield itself, it does not remove it for a weapon other than the shield in your other hand.
This is only mentioned here and there, but the link to this link was broken when Paizo updated its website over the past decade. There is therefore no direct link to this quote and it is impossible to find it during a search on their website. So, I think this has been archived or deleted and can no longer be found.