The day IPv6 broke my DC

Last week I worked a case where an AD DS/DNS server couldn’t resolve the domain name using IPv6, which was resulting in various problems and errors. Corner case? Maybe, but there are tricks to be learned from the troubleshooting and it serves as a cautionary tale for how important it is to plan and test your IPv6 implementation.

First I’ll review the situation and walk you through what we checked and what was working. This ultimately led to what was done incorrectly and why it broke.

This particular domain happened to have two 2012 Server DC’s, both running Active-Directory integrated DNS, and both on the very same subnet.

Symptoms:

  1. Running “nltest /sc_verify:domain.local” on the domain controller yields DsGetDCName failed: (error code) ERROR_NO_SUCH_DOMAIN.
  2. Nslookup on the DC lists the default DNS server as “Unknown Name” with an IPv6 address. Resolution of the domain name failed.
  3. Nslookup against the IPv4 address of either domain controller causes resolution of the domain name to succeed.
  4. Active Directory Users & Computers throws an error at open about not being able to contact a global catalog. I wish I had captured the error, but unfortunately there wasn’t time.
  5. Dcdiag fails the Advertising test with the message “Warning: DsGetDcName returned information for \\DC1.domain.local when we were trying to reach DC1. SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.”

What not to do:

In a related post on this very website, which can be found here – https://windorks.wordpress.com/2014/02/24/known-issues-with-disabling-or-unbinding-ipv6/ , we DO NOT want to disable IPv6 on the network stack. Disabling IPv6 would be a short-term fix that causes long-term problems. We’re not troubleblasting here, we want to find the root cause or it’ll just bite us down the road.

Troubleshooting:

To me the thing that stuck out the most was nltest saying it couldn’t query the domain and nslookup failing over IPv6. This wreaks of DNS resolution issues so we approached the problem from that standpoint. Here’s a list of what we checked.

  • Verify each domain controller has the correct DNS settings on its own stack: Namely have Primary set to the other DC’s private IP, Secondary set to its own private IP, Tertiary set to loopback.
  • Make sure there’s a reverse lookup zone for each domain controller IP range. Create if needed.
  • Remembering that IPv4 was resolving the domain but IPv6 was not, we verified the binding order on the networking stack. The logic being if we could get the server to default to IPv4, life would be peachy. This is done by opening ncpa.cpl and going into the Advanced Settings. In this case, IPv4 was prioritized above IPv6 which is what we had hoped to see. Also if there are other network interfaces in the top pane of the window, we’ll want to ensure the primary nic to be set at the top. As you can see this is already the case in the screenshot below.
    ncpa advanced
  • Unbind IPv6 from serving DNS requests. By default, a DNS server will listen on all bound IP’s on all enabled nics. To simplify, we ensured this was set to the known good IPv4 address for the DC, which would look similar to the following.DNS binding
  • If any of the DC’s are multi-homed – which I don’t recommend in the first place – make sure that only the nic running on the corporate private network (meaning the network interface of the DC you want servicing requests) is set to “Register this connection’s addresses in DNS.” Put differently, make sure this checkbox is cleared for any other interfaces on the domain controller. This seems like a no-brainer but here in support we’ve seen VPN connections, Wi-Fi connections, secondary Ethernet cards, and all other manner of sins. We only want to have a DC register it’s private IP in DNS. Here’s a screenshot of the setting.
    register this connection

After checking all of this and fixing where needed, we restarted the DNS service on each DC and flushed the DNS lookup catch by running “ipconfig /flushdns”. This clears the mechanism for accurate testing.

Did it work? Nope. Nslookup on the domain name still failed, the default DNS server was still showing “Unknown Name” and an IPv6 address instead of IPv4. No joy.

The Smoking Gun

It was time to look at the IPv6 configuration because whether we liked it or not, that’s what the server was defaulting to. Typically a domain controller will set itself (via loopback) as the DNS server in the IPv6 config. Here’s a screenshot of that default.

ipv6properties

In our case, the DC’s had this set to “Obtain DNS server address automatically” which we all know means “dhcp-assigned.” This in and of itself actually isn’t terribly alarming. In fact, a typical workaround to get rid of the “Unknown Server” message in nslookup is to change this setting to dhcp-assigned.

We then opened the network interface in ncpa.cpl and clicked the Details button for the network interface resulting in a screen similar to the one below.

ipv6 details

On the DC in question however, the Default Gateway and the IPv6 DNS Server were identical IPv6 addresses. Why would the gateway, otherwise known as the router, have the same address as the DNS Server??? Answer: Because the router is configured to hand out DHCP!!!!

This is fairly common in the SMB world. DHCP was unfortunately configured to hand out IPv6 addresses and the scope options were set for the clients to use the router as its DNS server. As IPv6 dhcp clients, no wonder the DC’s couldn’t resolve the domain over IPv6: they were trying to resolve against the router, which has zero information about the domain!  All valid DNS information is in AD-integrated DNS, not the router.

The options at this point are either 1.) change the dhcp configuration on the router, or 2.) set the DC’s back to use ::1 (loopback) for IPv6 DNS resolution as in the screenshot above. In the short run, we chose option 2.  In the long run, they need to fix IPv6 by either temporarily turning it off or ensuring the scope options are accurate.

The point

Pay good attention to IPv6 especially when it comes to your DNS servers. It’s easy to think it just takes care itself. As we’ve mentioned previously, disabling IPv6 isn’t the answer. Implementing IPv6 takes careful thought & planning, and if you do that work up front hopefully you won’t run into any problems like this.

Advertisements
Tagged with: , ,
Posted in IPv6
6 comments on “The day IPv6 broke my DC
  1. Bjorn Houben says:

    Good article. Thanks a lot.

  2. chucko says:

    IMHO, the root of the problem is really that Microsoft has configured all of their desktop and Server OS to prefer IPv6 over IPv4. It’s been that way since Vista and Server 2008. This can be changed in the registry.

    Microsoft even has a set of Fixit tools to automate the change.

    https://support.microsoft.com/en-us/kb/929852/en-us

    Use Fixit 50410 to set and Fixit 50441 to revert (if needed). Nothing is disabled by using Fixit 50410, it just changes the preference. This is more than binding order.

    In my area, Comcast is providing new boundary devices for business clients that by default have IPv6 and IPv6 DHCP enabled. Setting client servers to prefer IPv4 and disabling the IPv6 DHCP seems to keep things in order.

    You can tell it works by pinging the servername from a command prompt on the server itself. In default mode you’ll get an IPv6 address returned when you ping, but after the change, you’ll get an IPv4 address.

    The Fixits also work on Desktop Windows if needed.

  3. Earthrunner says:

    I had a DNS issue mysteriously appear after months of working (2012R2) with ::1 always appearing first despite me turning it off in DNS settings. After reading here I checked IPv6 settings and found ::1 as preferred DNS, removed it and bingo, 🙂

    Thanks to you blog here I was able to figure out the issue. Thanks very much 🙂

  4. tkardinal says:

    We had a similar problem in our recovery environment, especially not finding any DNS servers in our AD Sites. It especially came up when we were restoring Exchange and it claimed there were no GC’s in the site. Ceasing to listen on IPv6 solved it. It also improved DNS resolution speed dramatically. I’m considering doing the same in production.

  5. Thanks, having this pop up on a client site and this was my suspicion.
    Again, thanks for confirming.

  6. Perrin says:

    Thank you so much. Have a personnal infrastructure at home.Changed my ISP, deactivate DHCP on the new router provided by this new ISP. Distributes Gateway and DNS IPV6 addresses anyway !

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: