I've thought about this question many times as well when trying to decide between various dedicated server specs. This should be all based on the intended use of the server. Is it being used for hosting clients? Is it being used to host your own sites? Do any of the sites being hosted have the potential for using large amounts of CPU, or perhaps bandwidth? These are the types of questions you should be asking yourself when deciding which server you should select. For example, if you're using your server to host an image hosting or file hosting website, then those are known to grow very quickly. Within a few days your hard drive space, and bandwidth being transferred can literally go through the roof. The next thing that occurs is your server starts to slow down, and the other sites hosted on the server also slow down, and you have to be looking to upgrade the server.

It's usually best to try for a medium. Not too much and not too little. It doesn't make sense spending $300/month for a server and your hosting a site that transfers little to no bandwidth, and doesn't use much of the resources. If that's the case then you should consider a Virtual Private Server rather than a dedicated server. On the other hand if you're going to be hosting multiple active websites then by all means choose a server that's good from the start that exceeds your current needs which will allow it to grow without issues.

I should note that if your running MySQL, Mail, DNS, and Apache or IIS all on the same server, then it's wise that the server has a healthy set of ram, hard-drive and a powerful CPU. Each of those applications can be very intensive and will try to dominate each other for the resources. Do not expect to purchase a server with only 256 MB Ram and try to run all those applications on the server. You'll need a lot more than that to keep the server stable. This goes out mainly to anyone looking to start hosting client's websites using the server. You can have many angry clients if your server slows down tremendously based on packing too many applications on a server with poor specs.

In any event, be smart. Only you will know the reason why you're looking for a dedicated server. Key in on the question points I mentioned. Weigh the pros and cons, and you'll make the right decision.