Here’s what happened …

1. Lightning struck near our building, sending a surge into the building by way of the COAX broadband Internet cable
2. The strike blew the cable modem
3. Charter was called and they brought us a new cable modem.  The tech seemed to be unsure of how to provision it but seemed confident that he had it correct when he left. The initial tests of Internet connectivity seemed to confirm this.
4. After the cable connection was revived, I quickly discovered that the current from the lightning strike followed the Ethernet cable out of the back of the cable modem and into the firebox firewall that provided Internet access to a critical web server and a number of other shop floor devices.  The device would appear powered on but would not route traffic.
5. This was the backup device that was put into place because it’s predecessor had died a couple of weeks earlier.  This left us with no device to fail over to.  we were forced to put the web server back onto the TZ170 router that it was removed from in the past due to performance reasons.
6. I configured the Sonicwall TZ170 for the NAT and performed a connectivity test.  The speed tests on the TZ170 were equal to the old firebox.  Everything looked great  … at first …
7. Over the weekend, the TZ170 stopped routing published traffic to and from the servers.  A reboot of the router seemed to have cleared it up for a while
8. Reports were pouring in that customers could not access the web and our remote staff could not access the remote terminal server reliably.
9. This morning, I started to investigate these reports and believed that I had found the root cause. I thought that it was because the cable guy just forgot to provision the correct static IP addresses for our router.
10. I called Charter again and they assured me that the correct addresses were provisioned correctly.  We rebooted the cable modem and the TZ170 and everything seemed to work for the duration of the support call.  We hung up.
11. 15 minutes later, the servers were inaccessible from the outside again.
12. I connected my MacBook Pro directly into the cable modem and assigned it an available static IP address from our public pool.  I got nothing.
13. I removed the static IP from the macbook and set it to obtain via DHCP and … It was gived an address by the modem!  This was bad!  Really bad!  But really good at the same time because now I knew exactly what the problem was without a doubt!

The cable modem should NEVER be giving out automatic dynamic IP addresses because everything is statically assigned to published servers.  We only have a small range of addresses assigned to us and all but two are taken by servers and services.  The address that the modem gave to my MacBook Pro was actually the address that the TZ170 was statically assigned.

So, this whole thing was cased by an IP address conflict caused by a misconfiguration that left the DHCP server running on the modem.  The Charter guy should have known better.

I called Charter back and let them know what I found.  They logged into the router, disabled DHCP, and everything has been working since.

To add to this story, we performed a Pretty massive server room consolidation project over the weekend that went off without a hitch.  Instead of getting praise for the awesome job that we did, this Charter screwup made it look to the company like our move caused the Internet servers to go off line for 2 days off and on.

They will never understand the root cause … and this is the thankless life of an IT Admin.

]]> http://www.bostedor.com/2009/06/29/charter-business-internet-has-been-making-my-day-hell/feed/ 0