dnd 5e – What happens when a caster targets an object that looks like a creature with a spell that targets only creatures?

There are 2 ways to resolve this that have been discussed by WOTC designers in an official capacity: An older option on a Sage Advice segment of one of their Dragon Talk podcasts from January 2017 (specifically at about 13:12 for ~7.5 minutes), and a newer optional rule provided by Xanathar’s Guide to Everything‘s Dungeon Master’s Tools chapter.

Sage Advice: Intent is that spell effect doesn’t happen, action cost applies, and DM judges whether spell slot is used

To summarize Jeremy Crawford’s statements in the January 2017 podcast, “illegal targeting” is a gap in the written rules (as of the date of the podcast) and it’s mostly open to DMs to choose how to handle it. That said, Crawford says the intent for how it should be handled is that the spell should still take up the casting time but the spell effect will not occur and not consume a spell slot (I.E., option 2 listed in the question).

There are enough corner cases with this solution at the time of the podcast that Crawford still recommends that a DM adjudicate each individual occurrence on a case-by-case basis until there is eventually an official printed rule. As an example of why, he says spells such as those which require a spell attack probably should still consume their spell slot since there’d be some dissonance with the fact that those spells can miss, unlike saving throw spells which always “hit” but the target can resist their effects.

(He does not clarify what should happen if something like Eldritch Blast, which targets only creatures, actually hits a non-creature in this case.)

The flavor reason for this is that he views spells as essentially trying to make a magical connection between the caster and target. When that connection is established, the energy of the spell is consumed in producing the spell’s effect – but if the target isn’t one the spell can make a connection to, nothing happens and that energy isn’t expended.

He views spell attack spells as a different category; if I had to guess why, it’s likely because they mainly produce some effect that then follows standard attack rules in trying to reach the target.

Xanathar’s Guide to Everything (optional): No spell effect, action cost applies, and spell slot is expended

As of Xanathar’s Guide to Everything‘s release in November 2017, the (optional) rule for resolving invalid spell targets states (p. 85-86):

If you cast a spell on someone or something that can’t be affected by the spell, nothing happens to that target, but if you used a spell slot to cast the spell, the slot is still expended. If the spell normally has no effect on a target that succeeds on a saving throw, the invalid target appears to have succeeded on its saving throw, even though it didn’t attempt one (giving no hint that the creature is in fact an invalid target). Otherwise, you perceive that the spell did nothing to the target.

This is in slight contrast to Crawford’s earlier statements on the topic above, in that the spell still occurs and consumes a spell slot with no apparent effect.