posterchild for a broken company
My local cable company has a virtual monopoly on the local broadband market, and they know it. The phone company is not able to compete because their hardware is ancient, even by telco standards. As a data point, last summer when I was looking to upgrade my connection the best DSL I could get here was 'up to' 768k bidirectional, these were the same plans they had been offering for years, and I'm told that in our neighborhood the 'up to' meant "you'll be lucky to get 1/2 that at best". The cable company by comparison was offering both 8m/2m and 10m/1m service options, and added 16m service just months after I locked into a contract. The cable option was only a few dollars per month more expensive. So it's pretty clear the local phone company wasn't even trying to compete, they had given the market up as a lost cause I'm sure.
Now in reality, I've been a data customer of the local cable company for over 10 years now, and 95% of the time, I've been a very happy customer. There was a long period where they had oversold the network, and their links out of town just couldn't accommodate the traffic, and there was a period where their peering onto the backbone was very poor. Both of these drug on for many months before they got resolved. Then there was the snmp policy change, when they locked users out of our modems so we can't query for signal levels or error counts or anything useful. That's still annoying me, especially since they also recently locked some of us out of the web interface to the modems. But they're collecting the stats every 15 minutes so at least you can call in and they can query the stats and find out what's going on.
But recently, they've really demonstrated a level of service that boggles the mind.
First, some background. I'm a business customer, I pay for a 24x7 SLA with them, I have a priority escalated number to call for technical issues. Typically, I can get a live human being, located in the US, with a strong technical ability and all the tools needed to trouble shoot in less than 2 minutes from picking up the phone, middle of the day or night. Last week, around 1am Friday morning, my cable modem started to fall off line about every 15 minutes to an hour. I didn't feel lie dealing with them at that hour, so I called in Friday morning after getting some sleep, and they dispatched a tech for that afternoon.
Tech arrived a bit after 5:30, apologized for being late and proceeded to do everything he could to find the problem. An additional (more senior) tech arrived about 6:50 and he give him a few more things to check and try, you could almost see the relief on the younger tech's face when the other guy showed up. They did a bit more, but couldn't solve the problem. Finally, they tracked it back to something upstream of our pedestal. At the point that they had their meters connected to the pedestal and decided the call needed a line tech, they got on their walkies and got the line tech on duty. By now it was about 7:45pm Friday night. They were unable to get the line tech on duty to get in his truck and come out here to fix the problem. Lots of back and forth with him, lots of "it's just that one modem, no one else in the neighbor hood is having problems... so it has to be on his side of the connection" type of exchanges. They replaced cables, and connectors, and a bunch of other stuff. Pretty much every thing they COULD replace, they did. Finally when they had done all they could, they left, pretty sure that the line tech was in his truck driving out here. This was because he wasn't answering them on the walkies any more. So they said "if it's still broke in the morning, call back in, we start at 8am." and took off. There had been some talk that even if he had to do any repairs upstream, it couldn't happen until after midnight, because they weren't allowed to intentionally take anything offline until then now that they also provided telephone support. So ok, the line tech knows there's a problem, he'll solve it tonight.
Saturday morning rolled in with even worse connectivity than before. We set a record low uptime of 3 minutes at one point. 5-7 minutes to retrain and re-estabish connectivity every outage, and it was doing that after 3 minutes of use. Saturday's call center volume was a bit high... it took a whopping 30 seconds on hold waiting for a tech instead of the usually instant transfer to a live person. Virtually identical conversation with the support guys as the day before, explain what I'm seeing, they jump into the modem and look at the logs which tell exactly the same story, they dig a bit more and see the same thing the line tech reported the night before on the walkies... my modem's tx power levels is going from the 30s to the 50s. And it was a little flaky from about Tuesday forward, then really went to hell in a hand basket Thursday night into Friday morning, where it's been since. A tech was dispatched.
He got here in the afternoon, saw the new cable from the pedestal to our house, immediately knew he wasn't the first one here. We talked over the situation, he called in for a line tech and prepared to wrap it up. Nothing he could do. Being a contractor, he wasn't allowed to directly call the line techs, best he could do was put in a maintenance request... which he did... and they told him first available would be 10-12 on Tuesday. Now after he did that, he did look at what the previous techs had done... and he cleaned a couple things up, things that the in house techs hadn't done, and rarely do, but are supposed to. Things that would a gotten him fired if a line tech came out and saw he had left them like that. But they didn't fix anything, in fact they barely changed anything. He left after assuring me that a line tech was the right solution and that one would be out between 10-12 on Tuesday.
Well, at almost 1pm today, I got a call from them. Another tech... not a line tech... another customer tech. He said "I got a work order that just says "customer would like a static modem". got some more background for me?" So I explained the history to him... the modem was the last thing that hadn't been replaced because none of the 3 prior techs that had been here had the right kind of static modem in their trucks. (Business class service can come with a static subnet of IP addresses, for consistency, they want all of those accounts on the same type of modem.)
Now I had been sorely tempted to call up and cancel that call. The reason being that at 2pm on Sunday everything started working like normal again. But I wanted to know what had happened, so I left the call scheduled.
After giving the tech the full history of the issue, he said he'd look at the logs in the modem and their system, then grab a new modem just in case from the warehouse and head out here.
So here's where I get really pissed off.
It turns out that Sunday afternoon the regional technical support center for the midwest paged the line tech on call for the weekend. The very same one that had been looking at my modem levels, and the ones for the whole neighborhood. And he came out and replaced the upstream repeater board that services our area. It had started to get flaky. Now this call came from the cable side of the company... and they're only going to make a weekend page if there's "sufficient volume" to warrant it. Now there are only two things that use the upstream links that would generate a customer noticeable problem if they couldn't talk to the head end. One is video on demand, the other is pay per view.
The tech that visited today saw in the logs when this maintenance happened, he saw it on my modem, and every other modem in the area. There was an "obvious pattern" of TX power levels going up and down for a couple weeks, getting more pronounced over time, until modems started being unable to keep up with the out of spec voltages and fell off line.
So in hind sight, the techs that were here Friday night and radioed the line tech were on the right path, they had the problem nailed down, dead to rights. The line tech didn't do his job to support a $100/month business customer... but when Joe-Six-Pack couldn't get his pay per view wrasslin' on Sunday afternoon... boom he was off like a shot and had the problem isolated and fixed in an hour.
Way to go Charter.
Now in reality, I've been a data customer of the local cable company for over 10 years now, and 95% of the time, I've been a very happy customer. There was a long period where they had oversold the network, and their links out of town just couldn't accommodate the traffic, and there was a period where their peering onto the backbone was very poor. Both of these drug on for many months before they got resolved. Then there was the snmp policy change, when they locked users out of our modems so we can't query for signal levels or error counts or anything useful. That's still annoying me, especially since they also recently locked some of us out of the web interface to the modems. But they're collecting the stats every 15 minutes so at least you can call in and they can query the stats and find out what's going on.
But recently, they've really demonstrated a level of service that boggles the mind.
First, some background. I'm a business customer, I pay for a 24x7 SLA with them, I have a priority escalated number to call for technical issues. Typically, I can get a live human being, located in the US, with a strong technical ability and all the tools needed to trouble shoot in less than 2 minutes from picking up the phone, middle of the day or night. Last week, around 1am Friday morning, my cable modem started to fall off line about every 15 minutes to an hour. I didn't feel lie dealing with them at that hour, so I called in Friday morning after getting some sleep, and they dispatched a tech for that afternoon.
Tech arrived a bit after 5:30, apologized for being late and proceeded to do everything he could to find the problem. An additional (more senior) tech arrived about 6:50 and he give him a few more things to check and try, you could almost see the relief on the younger tech's face when the other guy showed up. They did a bit more, but couldn't solve the problem. Finally, they tracked it back to something upstream of our pedestal. At the point that they had their meters connected to the pedestal and decided the call needed a line tech, they got on their walkies and got the line tech on duty. By now it was about 7:45pm Friday night. They were unable to get the line tech on duty to get in his truck and come out here to fix the problem. Lots of back and forth with him, lots of "it's just that one modem, no one else in the neighbor hood is having problems... so it has to be on his side of the connection" type of exchanges. They replaced cables, and connectors, and a bunch of other stuff. Pretty much every thing they COULD replace, they did. Finally when they had done all they could, they left, pretty sure that the line tech was in his truck driving out here. This was because he wasn't answering them on the walkies any more. So they said "if it's still broke in the morning, call back in, we start at 8am." and took off. There had been some talk that even if he had to do any repairs upstream, it couldn't happen until after midnight, because they weren't allowed to intentionally take anything offline until then now that they also provided telephone support. So ok, the line tech knows there's a problem, he'll solve it tonight.
Saturday morning rolled in with even worse connectivity than before. We set a record low uptime of 3 minutes at one point. 5-7 minutes to retrain and re-estabish connectivity every outage, and it was doing that after 3 minutes of use. Saturday's call center volume was a bit high... it took a whopping 30 seconds on hold waiting for a tech instead of the usually instant transfer to a live person. Virtually identical conversation with the support guys as the day before, explain what I'm seeing, they jump into the modem and look at the logs which tell exactly the same story, they dig a bit more and see the same thing the line tech reported the night before on the walkies... my modem's tx power levels is going from the 30s to the 50s. And it was a little flaky from about Tuesday forward, then really went to hell in a hand basket Thursday night into Friday morning, where it's been since. A tech was dispatched.
He got here in the afternoon, saw the new cable from the pedestal to our house, immediately knew he wasn't the first one here. We talked over the situation, he called in for a line tech and prepared to wrap it up. Nothing he could do. Being a contractor, he wasn't allowed to directly call the line techs, best he could do was put in a maintenance request... which he did... and they told him first available would be 10-12 on Tuesday. Now after he did that, he did look at what the previous techs had done... and he cleaned a couple things up, things that the in house techs hadn't done, and rarely do, but are supposed to. Things that would a gotten him fired if a line tech came out and saw he had left them like that. But they didn't fix anything, in fact they barely changed anything. He left after assuring me that a line tech was the right solution and that one would be out between 10-12 on Tuesday.
Well, at almost 1pm today, I got a call from them. Another tech... not a line tech... another customer tech. He said "I got a work order that just says "customer would like a static modem". got some more background for me?" So I explained the history to him... the modem was the last thing that hadn't been replaced because none of the 3 prior techs that had been here had the right kind of static modem in their trucks. (Business class service can come with a static subnet of IP addresses, for consistency, they want all of those accounts on the same type of modem.)
Now I had been sorely tempted to call up and cancel that call. The reason being that at 2pm on Sunday everything started working like normal again. But I wanted to know what had happened, so I left the call scheduled.
After giving the tech the full history of the issue, he said he'd look at the logs in the modem and their system, then grab a new modem just in case from the warehouse and head out here.
So here's where I get really pissed off.
It turns out that Sunday afternoon the regional technical support center for the midwest paged the line tech on call for the weekend. The very same one that had been looking at my modem levels, and the ones for the whole neighborhood. And he came out and replaced the upstream repeater board that services our area. It had started to get flaky. Now this call came from the cable side of the company... and they're only going to make a weekend page if there's "sufficient volume" to warrant it. Now there are only two things that use the upstream links that would generate a customer noticeable problem if they couldn't talk to the head end. One is video on demand, the other is pay per view.
The tech that visited today saw in the logs when this maintenance happened, he saw it on my modem, and every other modem in the area. There was an "obvious pattern" of TX power levels going up and down for a couple weeks, getting more pronounced over time, until modems started being unable to keep up with the out of spec voltages and fell off line.
So in hind sight, the techs that were here Friday night and radioed the line tech were on the right path, they had the problem nailed down, dead to rights. The line tech didn't do his job to support a $100/month business customer... but when Joe-Six-Pack couldn't get his pay per view wrasslin' on Sunday afternoon... boom he was off like a shot and had the problem isolated and fixed in an hour.
Way to go Charter.
Labels: whine
posted by cabbey at 2:38 PM
1 comments
