encryption – How to ensure identity for server-to-server communication over IP

How would one go about ensuring that a server at a given IP-address is who it says it is? I need to send confidential data to another server at another IP-address. If the server had a public domain it is clear that a SSL certificate would be the right solution, but is this also the case for IP-addresses?

I control a specific piece of software running on both servers, where server one will send confidential data to server two. Server one lives in a cloud I can control, but server two can live on any machine around the world as long as it is connected to the internet. At server one I have a registry of all server twos that it can connect to.

I have thought about using a RSA key-pair, where the private key would live on the server two and the public key would live in the registry on server one alongside the server two IP-address. Then before sending the confidential data, I could present server two with the challenge of decrypting a random string and echoing back the result and check if it matches on server one. The confidential data will of course also be encrypted with the public key, so an eventual man-in-the-middle attacker spoofing the IP-address would get pure nonsens, but I don’t like the idea of someone eavesdropping on the communication.

TLDR; How do I make sure that a server at a specific ip address is not a man-in-middle attacker and is in fact a server that can be trusted with confidential data?