Yes, the reaction is triggered by the casting of the spell
As you already mention in your question the feat does not require the player to see the spell being cast, and so by the rules, the player is allowed to take the reaction regardless of what type of spell is being cast.
I was not able to find any rulings that would contradict the rules as written in the feat.
I can see why some people would see the ability to detect a spell being cast without any visible components strange but that is after all why the character trained hard to become a mage slayer in the first place.
You could decide that the feat works only on spells with visible casting or effects but that is something best agreed on and house-ruled by the group/DM as it is not mentioned in the rules.
For example, Jeremy Crawford, Principal rules designer of Dungeons & Dragons mentioned on twitter that in a similar situation (subtle spell vs. mage slayer) he would rule that
the person with the Mage Slayer feat needs to be aware—by sight or sound—that a spell is being cast
This is not an official ruling and does not effect rules of the game, however it seems like a reasonable solution to the dilemma.