I’ve been mostly happy with Road Runner cable. It’s fast and stable, and unlike Verizon, there is no public record of Time Warner giving out my browsing habits to the Fed on a whim.

I’m still switching to Verizon DSL. Fuck you very much Road Runner, you treat your customers like they are idiots, and they’ll abandon you.

Update: I returned to Pullman and set up the Verizon DSL that was waiting for me. It wasn’t as easy to get working as RR was, but now that it does work, I notice connecting to panix works very fast now. Just like I suspected it would.

It all started a few weeks ago, or perhaps just a bit longer.

My interactive sessions to panix.com, my oh-so-brilliant hosting provider started taking a while to start. A really long while. It should take about 3 seconds, and it was taking 10-15.

Yesterday, I finally grew tired enough of it to try to figure out the problem. My Google-fu was strong, and I made the discovery:

My IP is (or at least was) 76.178.157.111. All IPs should have a name associated with them, for example, ‘www.panix.com’ is several machines, one of which is ‘166.84.62.125’. They also should have what is know as reverse DNS, that is, a way to get the name from the IP.

76.178.157.111 in fact has such a name. It’s name is cpe-76-178-157-111.natnow.res.rr.com. However, because Road Runner has misconfigured DNS servers, it is impossible for a normal DNS server on the internet to know this.

Instead, when a server asks “what name is associated with the ip 76.178.157.111”, it gets “uhhh, I dunno”.

I was able to find exactly the problem, and even know how to fix it.

Tthe command “dig” contains an option that will force the client (dig) to iterate through the DNS server chain to find the name, basically doing what a DNS server normally does behind the scenes.

Here’s dig running against Road Runner’s DNS server, attempting to resolve the name:

; <<>> DiG 9.4.2-P2 <<>> @76.85.229.110 -x 76.178.157.111
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 40285
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;111.157.178.76.in-addr.arpa. IN PTR

;; Query time: 102 msec
;; SERVER: 76.85.229.110#53(76.85.229.110)
;; WHEN: Mon Dec 15 10:35:46 2008
;; MSG SIZE  rcvd: 45

[hephaestus:~]% dig @76.85.229.110 -x 76.178.157.111 +trace

; <<>> DiG 9.4.2-P2 <<>> @76.85.229.110 -x 76.178.157.111 +trace
; (1 server found)
;; global options:  printcmd
.        172406   IN NS C.ROOT-SERVERS.NET.
.        172406   IN NS D.ROOT-SERVERS.NET.
.        172406   IN NS A.ROOT-SERVERS.NET.
.        172406   IN NS F.ROOT-SERVERS.NET.
.        172406   IN NS G.ROOT-SERVERS.NET.
.        172406   IN NS E.ROOT-SERVERS.NET.
.        172406   IN NS M.ROOT-SERVERS.NET.
.        172406   IN NS L.ROOT-SERVERS.NET.
.        172406   IN NS I.ROOT-SERVERS.NET.
.        172406   IN NS H.ROOT-SERVERS.NET.
.        172406   IN NS B.ROOT-SERVERS.NET.
.        172406   IN NS K.ROOT-SERVERS.NET.
.        172406   IN NS J.ROOT-SERVERS.NET.
;; Received 504 bytes from 76.85.229.110#53(76.85.229.110) in 97 ms

76.in-addr.arpa.  86400 IN NS X.ARIN.NET.
76.in-addr.arpa.  86400 IN NS Y.ARIN.NET.
76.in-addr.arpa.  86400 IN NS Z.ARIN.NET.
76.in-addr.arpa.  86400 IN NS CHIA.ARIN.NET.
76.in-addr.arpa.  86400 IN NS DILL.ARIN.NET.
76.in-addr.arpa.  86400 IN NS BASIL.ARIN.NET.
76.in-addr.arpa.  86400 IN NS HENNA.ARIN.NET.
76.in-addr.arpa.  86400 IN NS INDIGO.ARIN.NET.
;; Received 200 bytes from 192.228.79.201#53(B.ROOT-SERVERS.NET) in 51 ms

157.178.76.in-addr.arpa. 86400   IN NS DNS6.RR.COM.
157.178.76.in-addr.arpa. 86400   IN NS DNS5.RR.COM.
157.178.76.in-addr.arpa. 86400   IN NS DNS2.RR.COM.
157.178.76.in-addr.arpa. 86400   IN NS DNS1.RR.COM.
157.178.76.in-addr.arpa. 86400   IN NS DNS3.RR.COM.
;; Received 146 bytes from 199.71.0.63#53(X.ARIN.NET) in 40 ms

157.178.76.in-addr.arpa. 600  IN NS device-dns1.rr.com.
157.178.76.in-addr.arpa. 600  IN NS device-dns2.rr.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 135 bytes from 24.30.199.7#53(DNS3.RR.COM) in 109 ms

111.157.178.76.in-addr.arpa. 604800 IN PTR   cpe-76-178-157-111.natnow.res.rr.com.
;; Received 122 bytes from 65.24.6.70#53(device-dns2.rr.com) in 100 ms

Hey, look at that, it worked!

Well, not quite. See, there is that nasty note about BAD (HORIZONTAL) REFERRAL. When these things happen, DNS servers stop following the chain and return an error, because this sort of thing isn’t allowed.

Essentially, what is happening above can be described as a sequence of questions:<

  1. dig first asks the DNS server 76.85.229.110 “hey, where would I find the name for 76.178.157.111?”
  2. 76.85.229.110 says to me “I don’t know, but the any one of C.ROOT-SERVERS.NET, D.ROOT-SERVERS.NET, etc would!”
  3. OK, so then dig moves on, in this case to B.ROOT-SERVERS.NET, and asks “Hey, where would I find the name for any IP that starts with the number ‘76’?
  4. B.ROOT-SERVERS.NET says “You can find that at X.ARIN.NET, Y.ARIN.NET, etc”
  5. dig then asks X.ARIN.NET, “Hey, who could tell me about an IP that starts with the number “76.178.157”
  6. X.ARIN.NET says “you can find that at DNS1.RR.COM, etc”
  7. dig then asks DNS3.RR.COM “Hey, you know all about the IPS that start with ‘76.178.157’, what’s the name for ‘76.178.157.111’
  8. DNS3.RR.COM says “I don’t know, you’ll have to ask device-dns1.rr.com or device-dns2.rr.com
  9. dig then says “well, that doesn’t make any sense, device-dns1.rr.com and DNS3.RR.COM are both in ‘.rr.com’, so I’ll complain; a real DNS server would quit here.
  10. dig finally asks device-dns2.rr.com, “What’s the name for ‘76.178.157.111’?’
  11. device-dns2.rr.com answers “‘76.178.157.111’ is ‘cpe-76-178-157-111.natnow.res.rr.com’

There’s an important thing to note about step 9. The process fails because “device-dns1.rr.com” and “DNS3.RR.COM” are in the same domain. If, just for example, the server that held the information about the IP was on a machine named ‘device-dns1.natnow.rr.com’, that would be a subdomain of rr.com, and would be legitimate (natnow.rr.com is not the same domain as rr.com)

There are two simple solutions to the problem:

  • rename ‘device-dns1.rr.com’ and ‘device-dns2.rr.com’ to ‘device-dns1.res.rr.com’ and ‘device-dns2.res.rr.com’. This creates a subdomain, which will eliminate the horizontal referrer.
  • set up DNS1.RR.COM, etc to be authoritive hosts for the zone hosted by the device-dns servers, so that they can answer the question directly without referring to the device-dns queries.

Anyway, RR refused to allow me to talk to somebody that would understand this process, and wasted my time for over an hour. Instead of fixing, or even acknowledging the problem, they simply refused because I was not a customer with their expensive, too much service for my needs, business cable customer.

So fuck em, I’m switching to DSL. Verizon may have it’s problems, but they at least understand basic networking.