Many domains fail almost every one of the reliability tests. They have two nameservers (the minimum), but often on adjacent IP addresses, say x.y.z.1 and x.y.z.2. These are certainly on the same IP network, and therefore in the same AS. They’re probably in the same rack, one sitting on top of the other. There is a large number of common failure modes that can make them both temporarily unreachable.

You can use the whois command (or any WHOIS-lookup web site) to look up the NS records by the name of your domain. It should tell you both their names and their IP addresses.

Here’s a very bad example, but very typical:

$ whois example.com
(…)
Domain servers in listed order:

A.EXAMPLE.NET 192.0.34.43
B.EXAMPLE.NET 192.0.34.44

If there are only two of them, and their IP addresses are identical except in the last number (as in the bad example, above), you have a problem.

If they’re not as close together as the example, finding out for sure whether they share an IP network takes local knowledge of the routing topology, which you probably can’t determine on your own. Ask your DNS provider for details.

Finding the AS path for the IP addresses is easier, but the output takes some expertise to interpret. http://nitrous.digex.net/ has some Looking Glass web pages that will let you look up available routing information for any IP address. (Pick a site and choose a BGP Query.) If in doubt, ask your DNS provider to describe their redundancy in terms of IP networks, physical space, and AS diversity. If they don’t even know what you mean, escalate. If they refuse to “disclose” this to you, vote with your feet.

You bet it is!!!

Your email delivery is dependent upon the wide and reliable visibility of your MX record. In order to assure this, your zone file should be published on at least two DNS servers, or nameservers, and preferably more — four is ideal. This is so that at least one of them is reachable and working correctly, from the standpoint of someone trying to send you email.

If all of your nameservers are unreachable from a given point on the Internet (your correspondent’s originating mailhost), at a given time (when they hit “send”), your domain will appear not to exist, and mail will likely bounce immediately with:

HOST UNKNOWN

This is what we most want to avoid. If your nameservers are up, even if your mailhost is temporarily offline, mail will queue at the origin, rather than bounce, and this is much to be preferred. It gives you a while to fix your mailhost before the senders notice anything is wrong.

Your nameservers should be separated widely, in several ways, to avoid so-called common-mode failures that would otherwise affect them all. They should be:

On multiple hosts - Not on a single host that may be down.

This shouldn’t need much explanation. Computers do not run without incident for long times. Even with no errors and no equipment failures, they need maintenance and upgrades from time to time. DNS requires that every domain have at least two nameservers. Host redundancy is built into the protocol.

Separated in space - Not in the same city, let alone the same building, or the same rack.

To protect against local physical accidents: fire, flood, tripping over the power strip, meteor strike, et cetera.

On separate IP networks - Not on the same IP network or physical (Ethernet) segment.

To protect against common-mode network failures, such as routes being dropped, or border router or LAN switch equipment failures. This is the most commonly violated principle.

In separate Autonymous Systems - Not on networks sharing major routes from the Internet.

Here we get into the core of routing in the Internet. An AS is a group of IP networks run by a single organization, such as an ISP. Where ISPs’ networks connect to one another, they exchange routing information about their networks, but they summarize the exact path that a packet will take. They just advertise the so-called AS Path. It will get your packets to the right ISP, and then a more-detailed system takes over and routes the packet to the exact router and host.

If you have four nameservers, in four cities, on widely different IP networks, but all these networks terminate in the same AS (for example, data centers all run by one ISP), then there is still an important, but subtle, common failure mode. That ISP (that AS) can have a widespread failure that makes all of their networks unreachable. If at least one of your nameservers is in another AS entirely, you are protected against even this.

Should they be on separate nations/continents/tectonic plates?

All the world is not the USA. Though of course there is a point of diminishing returns.

“Imagine you run a large technology company not named Apple. Let’s say you’re Steve Ballmer, Michael Dell, Meg Whitman, Larry Page, or Intel’s Paul Otellini. How are you feeling today, a day after Apple CEO Tim Cook unveiled the new iPad? Are you discounting the device as just an incremental improvement, the same shiny tablet with a better screen and faster cellular access? Or is it possible you had trouble sleeping last night? Did you toss and turn, worrying that Apple’s new device represents a potential knockout punch, a move that will cement its place as the undisputed leader of the biggest, most disruptive new tech market since the advent of the Web browser? Maybe your last few hours have been even worse than that. Perhaps you’re now paralyzed with confusion, fearful that you might be completely boxed in by the iPad—that there seems no good way to beat it.”

http://www.youtube-nocookie.com/embed/LabqeJEOQyI?rel=0

“It is unacceptable at this point in time to not know the true cost of a service and the real exit costs from those services: the costs commercially, technically and from a business de-integration standpoint. So, how do we untangle our way out of a particular product or service. I can’t tell you how many times I’ve had the discussion that says, we need to get away from that, and we can’t because of the complexity of getting out from where we are, and of all the things that are hanging on to that particular service, that we can’t disentangle ourselves from.

I think it’s completely unacceptable at this point in time to enter into contracts for longer than 12 months. I can’t see how we can sit in a world of IT, and acknowledge the arrival of the iPad in the last two years, and yet somehow imagine that we can predict what we’re going to need to be doing in two or three or five or seven or ten years time. It’s complete nonsense.”

It fills me with hope that there are people out there working to highlight that the command-and-control approach to technology and systems no longer holds up. Can the few break out of the force field of the incumbent suppliers? I really hope so.

Full audio of Chris’s piece is available here.

“ The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong questions.” Peter Drucker