Admittedly, Hypnotic Pattern is an unusually powerful spell and one that can tip a combat, especially if the PC’s outnumber the monsters. Before you resort to changing the fundamentals of that spell, though, or “giving immunity” to monsters, make sure you are running the PC’s spells RAW. If you are not paying attention to things like range and timing, but just allowing the players to use spells because they have them, spells like Hypnotic Pattern are going to come off as even more powerful than they already are.
The sky is big – have you checked RAW range?
Unlike a “dungeon” or indoor setting, where most of the participants remain in spell range for the entire combat, the sky is a big, open space.
Mind Sliver has a range of 60 feet.
Polymorph has a range of 60 feet.
Hypnotic Pattern, for some reason, has a range of 120 feet.
But the roc has a flying movement of 120 feet! This means that it has a reasonable chance of remaining out of spell range until its turn, and then moving into range and making its attack, so long as its target is not immediately adjacent to the caster. For example, consider a Hypnotic Pattern caster and an allied PC standing on deck 20 feet away. If the roc approaches from the direction of the ally, it can start its turn at 140 feet from the caster (out of range of the Pattern) and still reach the ally to attack by the end of its turn. If it is a big ship and the PC’s aren’t clustered together, that may even give the roc the opportunity to both attack a PC and get back over the rail before the caster can respond. With a +13 to hit and an automatic grapple, the roc should be taking a PC with it. That should give the players pause – once their companion is over the rail in the roc’s talons, hypnotizing or polymorphing it may mean that a PC will be joining the roc in plummeting a thousand feet.
Casters can, of course, ready actions to cast spells when the roc comes within range. But remember that readying a spell requires casting the spell and then holding the energy, which dissipates at the end of the round. You can have the roc circle the ship a few times before it moves in. Once the casters have burned through a few third and fourth level slots readying spells that are never cast, they may reconsider.
Also, is the ship moving? If it is not hovering in place, but moving at a certain speed and direction, the roc may be able to use that. By grappling the aftmost PC, for example, the ship might move ‘out from under’ the PC on its turn.
Flying combat takes place in three dimensions
Currently 5e movement rules largely indicate two dimensional thinking. But the roc is a creature of the air. What is to prevent it from approaching the ship from underneath, so that the casters cannot see it until right before it bursts on deck and grabs someone? What is to prevent it (besides masts – I’m not familiar with the design of the airship) from approaching the ship from above, and dropping from outside of spell range to the deck, taking some fall damage itself but possibly doing considerable damage to everyone it lands on?
Finally, not RAW but pretty reasonable and supported in earlier editions – the roc could start its turn at a higher elevation than the ship and gain speed by dropping in elevation as it traveled toward the deck, allowing it to move from outside of spell range into its attack range and back off the ship in one turn.
The GM describes the environment
In general, you should avoid changing the details of player’s spells when you find them challenging to deal with. Players often feel a GM is being antagonistic rather than fair if they ‘nerf’ the player abilities – the one thing the players have going for them and can count on. However, it should be expected that the GM describes both the environment and its effects on the combat. Is the airship noisy and the roc a silent glider? Then it can likely sneak up on the ship at night, in a storm, or in a cloud – getting within striking range perhaps in a surprise round. When a roc falls, is it being buffeted by winds or just the air resistance of its massive wings? That could reasonably be considered something that would “shake the creature out of its (hypnotic) stupor” before it hit the ground. Have the players polymorphed an enemy flier into a chicken and thrown it off the ship? Chickens can fly – perhaps not well enough to glide a thousand feet to the ground – but certainly well enough to spiral, collide with the hull, take a point of damage, and resume their original form long before they hit the ground.
Rather than trying to change rules to limit the characters’ spells, try to rigorously enforce pre-existing RAW limits, expand the strategic options of the monsters, and describe the consequences of the environment in ways that don’t favor the players.