There are no rules, at least no strict and fast rules. Following the research to the letter can lead to the creation of an ugly website – the blue links work better, but the blue links on a red background are hard to read! Research can however guide you to "best practices" and maintain your sense of good aesthetics so that you know when to break with the convention.
Although there are no rules, there are a lot of guidelines; some are obsolete and others are very good.
Old link appearance messages are Nielson Alertbox messages:
View links and Change the color of visited links
People get lost and go around in circles when websites use the same link color for visited destinations and new destinations. To reduce confusion in navigation, select different colors for both types of links.
These publications give a good basic summary, but they are quite old. The big problem to remember is that links must be a single color, always chosen and the links visited must be of a clearly different color and systematically chosen. However, do not put too much stock in statistics because the web has changed a lot in 7 years. We will then see the current statistics.
The most important search on links is probably a Google experiment showing that blue links are more clicked. We click less on the non-blue links, so that the color scheme must take this into account. For this reason, blue should be avoided, as well as underlining.
Regarding the links visited, here is an excellent and up-to-date article on the status of Kevin Simons' web links; Where did all the visited links go?
Simons notes that the use of the color of the visited links has dropped sharply – I would even say that in 2004, most sites have probably never bothered to stylize their links, which has resulted in a change of default color of visited links. Simons points out that the links visited are mainly used to help a user know where you went:
Pay particular attention to the cognitive efforts you have to make on sites without a visited link color to find out where you are. You are sure to develop a new appreciation of the visited link.
Does the user need a reminder that he has already navigated to this link? If they do not, the link style probably should not change to all. Note that for top-level navigation items, a visited link color may be unnecessary and visually ugly (do I really SEE visually that I have never used the Contact Us section?).
For dynamic pages, a visited link color may be downright misleading; Nowadays, many web applications have dropped the convention on visited links because "visited" does not mean much in the context of an application or dynamic pages.
If you need to distinguish between visited links and unvisited links, Contrast is King.
As noted by Leslie Jensen-Inman of A List Apart:
The effective use of contrast not only differentiates your design from others, it is the essential ingredient that makes the content accessible to all viewers.
The article explains in detail the importance of contrast and its effective use; I strongly suggest you read it carefully.
They point out that, for reasons of accessibility, you should compare your colors with common colorblind simulators to make sure that the contrast is suitable for color-blind users, as well as for non-colorblind users. Remember to tint your colors to see if they work better.
They also offer CheckMyColors.com, an excellent site that will allow you to check the background / foreground contrast of all elements of your site at a time.
Note that the foreground / background contrast do not tells you if two pieces of text in the foreground have enough contrast! That's why you can not just use the Color Contrast Checker as some have suggested and call it a done deal.
Here's an excellent guide to typographic contrast that will help you bring together all forms of contrast to create a fluid and enjoyable site.
In the end, the contrast is great for differentiating things, but it must be used appropriately. There is really no reason to differentiate the links that are visited and unvisited, do not. If the maximum contrast or standard link colors are too ugly with your current design, try something different.
Only real users can check if your links are separate and functional.