Tracking down rogue bandwidth: a story of Comcast data caps and offsite backup

tl;dr: if you use BackBlaze and are subject to a data cap, you should keep an eye on BackBlaze to make sure it doesn’t quietly use up all your data allowance. I’ve switched to CrashPlan, which has better retention, encryption, and backup policies.

Beyond sharing this warning, I wanted to do a longer write-up of what I experienced because I found some pretty interesting things along the way – how to get true visibility into what’s happening on your local network, what’s up with Comcast’s usage meter, and what happens when something goes wrong with offsite backup with BackBlaze.

Earlier in the year Comcast announced they were introducing 1 terabyte data caps in 27 markets across the US (of which 18 previously did not have enforced caps). This is a pretty consumer-unfriendly move, and being in California this meant this was happening to me – but personally I had never come close to exceeding the cap, so wasn’t overly worried (and won’t be until 4K video becomes more common). In the second week of November 2016 I got a notification in my browser telling me I’d used 90% of my cap. This was surprising in two ways – one, I didn’t expect Comcast to hijack an http request to do this, and two, it was saying I’d used over 900gb of data in just over a week. I was about to head overseas for 8 days so I decided to shut everything down while I was away and figure out what was using the data once I got back.

While I was away, each day I saw my data continue to increase by approximately 80gb a day. I had a left my Mac Mini server running, a Dropcam, and a Nest and that was it. I checked remotely to ensure there were no other wireless clients running on my network, but couldn’t see anything. I used Activity Monitor to keep a track of how much bandwidth each process was using on the server, and saw BackBlaze (bztransmit) had transmitted ~50gb, but my uptime was 3 weeks at that point, and so that seemed about right. I was completely baffled – nothing else was being generated by the Mac Mini, and the amount of data uploaded by the Dropcam was also minimal, yet each day it kept increasing.

With no obvious culprits on my network, it seemed suspicious for me to be exceeding my cap just as they introduce caps, so my attention turned to whether Comcast’s usage reporting was accurate.

Comcast gets a lot of heat for their data caps, and in particular for their usage meter accuracy. When trying to use the meter to diagnose bandwidth issues, Comcast itself says it should not be relied on. They have a disclaimer on the meter saying it is “delayed by up to 24 hours” (and on the phone I was told it could lag for weeks). By the end of figuring all this out, I actually found their usage meter to be accurate in realtime, but others online report usage changing after the fact. I also found numerous people online complaining that their usage meter did not match their self-evaluated usage (some measured using tools such as using Gargoyle firmware on their router). However, most of these quibbles were off by 10% or so, not the orders of magnitude I was seeing. I was also unable to find well-proven cases of their meter being wildly off except for one instance where a MAC address had been entered with a single character typo. There was only one other example of Comcast rolling back fees for overages – but it only happened after the media became involved, and nobody technically proficient actually checked the network.

Given the likely accuracy, I was keen to find an inexpensive (and preferably software based, so I could do it remotely) method for measuring what was actually going through the cable modem. Most of the advice I found suggested buying a router that supported Gargoyle, but I also discovered that the Motorola SB6121 cable modem I use reports number of codewords per channel, aka bytes.

In the above example, you can see the aggregate codewords for the 4 channels is 275,455,328,290, which is 275.45 gigabytes. This was exactly as I would’ve expected. I remotely reset the cable modem (another quirk – Comcast removed the ability for me to do that myself on the cable modem through a modem update – I actually own the modem too) using the mobile app, and after doing so my traffic was measured in megabytes per day, which was what I expected. At this point I was baffled, as advice I’d seen repeated a few times online is to use the codewords as the source of truth, but they do NOT include uploads, which Comcast counts towards your cap.

At this point I didn’t know that, and since I had no idea how they were measuring so much traffic, I kept escalating with Comcast, eventually being told a level 3 support technician would contact me. That was a couple of weeks ago, and I never heard anything more from Comcast.

To be completely sure it wasn’t something on my end, my next step to diagnose this was to use SNMP logging to verify the amount of data leaving my network via the ethernet connection to the cable modem. SNMP is “simple network management protocol”, and is a standardized way of reporting and collating information about managed devices on a network. Routers with SNMP logging are able (amongst other things) to report exactly how much bandwidth is being consumed on any of its many interfaces (which are unhelpfully named, but a good description of each on an Airport Extreme is available here). Helpfully, mgi1 is the interface specifically for the WAN connection – i.e. I was able to measure very specifically every bit of traffic going to and from the cable modem. Unfortunately, Apple took this feature out of the 802.11ac Airport Extreme, which I used as my main router. As such I reconfigured my entire network to use my older router and an old Airport Express (which also includes SNMP logging).

At this point I had SNMP logging all my internal network bandwidth, and to visualize it I used PeakHour 3, which did a great job of making it very easy to see what was happening. At this point I finally had proof my Mac Mini really was uploading a LOT of data, and matched the Comcast usage. But Activity Monitor still did not show me that anything was out of sorts, so I still didn’t know WHAT was causing all the usage.

I researched what other tools I could use to monitor network traffic by process, and I found Little Snitch. Little Snitch lets you police all network requests, and approve or deny them, which is pretty nifty, but all I needed was the monitoring tool. This let me see that bztransmit was happily uploading at 5-10mbps in bursts every few minutes, and the cumulative from this process matched exactly the upload traffic seen in the SNMP logging. Leaving it for even just a few hours, it was clear this was the culprit.

I throttled BackBlaze using their preference pane to 128kbps (it claimed it would upload “approximately 1gb a day”, compared to the 4gb/hour it was doing at the time) and contacted their customer support. While I waited to hear back from them, I started reading through the BackBlaze log files, and saw it was uploading the same amount each day, and then a little more, e.g.:

server:bzreports_eventlog admin$ cat 10.log
 2016-11-10 04:55:40 - END_BACKUP: Backed up 2801 FILES / 40617 MB

server:bzreports_eventlog admin$ cat 17.log
 2016-11-17 05:02:54 - END_BACKUP: Backed up 2933 FILES / 51489 MB

They helpfully have a log of the last files uploaded, located in /Library/Backblaze/bzdata/bzlogs/bzreports_lastfilestransmitted. The same files were being uploaded each day.

Another thing I noticed in the log files is that it appeared that BackBlaze was downloading updates and/or reinstalling itself basically every day. As far as I can tell, this was resetting the usage in Activity Monitor and why that was not a reliable measure.

While I was investigating these logs, I continued monitoring my network, and noticed that while BackBlaze was “throttled”, it ended up uploading nearly 4gb of data in 24 hours. The bztransmit process was using just under 1mbps of bandwidth, approximately 8x the promised throttle limit. To be clear, I have been using megabits for all my bandwidth measures in this post. I have to assume that there is a mistake in the conversion somewhere, which would perfectly explain why it was uploading at 128 KBps rather than Kbps. Their annotation in the UI and their documentation is ambiguous as it’s all lowercase and abbreviated, however, their estimates match that the numbers shown should be kilobits (128 kbps = 16 KBps = 1,382,400 kbytes = ~1.3GB/day).

BackBlaze got back to me the following day and asked for a copy of my logs – all of them. They gave me a unsigned tool which gathers all these logs, as well as a full system snapshot – approximately 50mb of log files in my case. At this point I wasn’t very comfortable about this. BackBlaze encrypts your files with your private key before uploading them, and according to an employee on Reddit, they can’t even see the filename. I really liked this feature – having all my private documents in the cloud is a scary proposition from a security perspective, and even if the contents are encrypted, the filename themselves leak entropy (e.g. financial documents, photo folder names, etc). I wasn’t particularly keen to send that over, as the logs are very chatty about your system and the files it’s working on.

At this point things with BackBlaze broke down. The customer support person I was communicating with ignored my request to surgically provide logs rather than send all of them. And by ignore, I mean they stopped updating the ticket and ignored my updates to the ticket. I was only able to re-engage them by pinging the BackBlaze Twitter account. He then refused to escalate further without the logs, and ignored my report of the throttling bug.

Without BackBlaze making any good faith efforts to remedy the situation (I should also note they were never apologetic about any of this, including just ignoring the ticket), I investigated alternatives, and have switched to CrashPlan. They offer a variety of better features compared to BackBlaze, including file versioning (including allowing deleted files to stay backed up – BackBlaze will delete after 30 days), 448-bit file encryption (versus 128-bit for BackBlaze), and allow NAS backups too. They are $60/year compared to $50/year for BackBlaze.

I had been using BackBlaze for 6 years. I had been an evangelist, recommending it to family and friends, and likely referred at least a dozen new customers to them. To say the least, I was very disappointed at this experience. After this happened, I did a quick look around to see if this had happened to others. What I found was a pattern of issues with customer support, slow restores, other bugs (folks missing files showed up several times), and backups being unexpectedly deleted from the server (e.g. someone goes on vacation and leaves their external drive at home, and while away the 30 day trigger hits. This also means if you do have a catastrophic data failure, you have 30 days to get your computer set up again to do a restore). It was interesting the different approaches people took to try to vent their issues with the company, including Amazon reviews, Facebook, the BBB and even CNet.

Most tellingly I found someone on Reddit reporting the same issue I encountered – a year ago – with other users in the thread reporting the same problems.

At the end of all this, I’ve spent somewhere in the vicinity of 20-30 hours of my time diagnosing this and talking with customer support, I’ve gone over my quota for two months (October and November – with the latter hitting 2TB in the end), leaving me a single courtesy month with Comcast (after which I never get any courtesy months again). On the plus side, my network is now more secure, and I learnt some interesting stuff along the way, and I was able to diagnose the cause literally 1 hour before midnight on November 30th, preventing me from going over again. The bummer is that there are two companies who have shown no interest in fixing issues for the consumer and puts all the onus on them: BackBlaze need to fix these bugs, and Comcast needs to provide their customers with better tools for diagnosing and monitoring network traffic if they’re going to institute caps for everyone.

Skype will never succeed as a replacement for your home phone…

…so long as they’re happy to just “block” people’s accounts and prevent them from using the service.

Last night I tried calling my wife and I kept getting “call failed”. So I tried from my computer, thinking my Skype phone was screwy. I got the dreaded “account blocked” message.

I rely exclusively on Skype for my phone. I have a San Francisco SkypeIn number, a Brisbane SkypeIn number, a Skype home phone and I also rely on Skype To Go to make international calls from my mobile phone. Aside from my T-mobile bill, they now get all of my telecommunications money.

Regardless of my long-term account which has always been dutifully topped up using their auto-billing, for a reason they never both communicating to me, they have completely blocked any internal or external calls aside from Skype to Skype.

The worst aspects to this are:

  • No notification of the blocking event
  • No explanation for the blocking
  • No customer support to get the account unblocked

I had to file a support ticket requesting they unblock my account.  There are no standard mechanisms for doing this – I had to go to support and select “Other help”.  I emailed a few people at Skype directly, but haven’t heard back from them either.  Other people in the same predicament reported between 4-10 days to get their account unblocked.  In the meantime I am completely without home phone service or international calling.  I am surprised I hadn’t heard of this before it happened to me — I can only imagine as they keep doing this to people they will continue to drive away customers to other VOIP solutions or cell phones.

Update 09/25/08:

Well, first of all I found out there is a dedicate help section for unblocking your account, but it amounts to the same as selecting other help.

Secondly, thank you to Peter in the comments for offering to help out which is very nice indeed.  After filing two tickets, I got in contact with Peeter Mõtsküla who was gracious enough to contact support for me and help resolve the problem.

I also found out the reason my account was blocked.  A couple of months ago I had some fraudulent activity on one of my credit cards.  As such I was issued a new card, but I decided to close the account.  As such when my autobill went through it got rejected and Skype emailed me to let me know that on August 16th.  Either way I think this would have been triggered since the original credit card number was no longer valid.

The weirdest thing is that it wasn’t until September 18th that my account was blocked due to the credit card rejection.  No ‘invalid’ funds had been withdrawn, and everything was fine otherwise.  Pretty frustrating stuff, but good to know Skype has some great people working for them!

How not to run customer support

I just got back from vacation last week.  It was a whirlwind tour of Norway, Denmark, Belgium and the UK.  I had a great time, sampling a lot of beer and seeing a lot of cool places.

At the start of the trip, I was connecting from London Heathrow to Oslo.  Given I had around 5 hours to kill in Terminal 5, I decided to pony up for some airport wifi.  They have several providers in T5, and I went with the brand name I knew, which also had a decent price.  Boingo.

Everything was just peachy when I signed up. Then I tried to log in.  Complete brick wall – everything started timing out, and authentication just wouldn’t happen.  No biggie, I signed up for another provider and sent an email to them letting them know their wifi wasn’t working at that location and if I could get a refund.

Hi there,

I just tried to sign up for Boingo and it worked just fine for taking my credit card details.  However, after this I was unable to surf the internet – the client I downloaded could not authenticate me, nor did the online authentication work.  The online authentication seemed to work momentarily and then sites began timing out.  I tried restarting the computer and reconnecting, but even the unauthenticated pages wouldn’t work.  I had plenty of wireless signal.  I have screenshots of the problems if that helps.

I needed to use the net in a hurry and ended up having to buy access through another wifi provider.  Since Boingo could not provide the service I paid for, could I please get a refund?  My username is <xxx>.

Thanks,
Tim

After three days, I hadn’t heard anything.  Hmm, not a good sign.  This time I CCed service as well as support.

Hi,

I haven’t heard anything back yet, and was hoping someone could help me with this?

Thanks
Tim

This is the response I got (literally the whole email):

Thanks for contacting Boingo Wireless.

What is your username?

Irritating and unhelpful, but nevermind – I can understand that CSRs are usually busy and overworked.

Hi there,

As per my original email, it is <xxx>.

Tim

A day later I got another response…

Thanks for contacting Boingo Wireless.

It could be any number of things and your account status is active.

So that we can better trouble shoot you please call when you are at the hotspot location.

I didn’t think they could be serious. This is how you’ve trained your CSRs to help customers?  For a company that is based entirely on wireless hotspots?  You ask your travelling customers to go back to the hotspot they had trouble in and to troubleshoot from there, days after their missed opportunity?

Finally, I requested a refund again:

Hi,

I was at Heathrow Airport for a period of 5 hours, and needed the service then.  I am in the middle of a trip, and I will not be back at the service location.

This is getting very frustrating – I would really like a refund, otherwise I will need to request a chargeback from my credit card company.

Thanks
Tim

This was the response:

We were unable to locate an account for you, with the info provided below. You should not be charged.

At this point (a week after first requesting the refund), it seems like a customer care supervisor saw our back and forth, and thankfully intervened.  I can happily say I then received the refund:

Dear Tim Cederman-Haysom,

Thanks for contacting Boingo Wireless.

This is to confirm your refund of 5.95 to your credit card and we do apologize for this inconvenience.

If we can be of any additional help, please don’t hesitate to contact our Customer Care team.  We’re available 24/7.

Warm regards,
Brenda Cooper
Online Customer Care

It is safe to say that I stuck to Boingo alternatives for the rest of the trip.  The worst part about providing sub-par support like this is you don’t lose a single customer: you lose the word of mouth from them as well.  I’m still not sure why people skimp on providing excellent customer support when companies like Amazon, Fog Creek and Zappos have used customer support to as a way to get talked about, in a good way, and this word of mouth brings an incredible amount of business.

Incidently I had great service from The Cloud, the alternative I used, and T-mobile wasn’t bad either when my wife used it with her roaming account.

Spamming users

One of the great things about Australia is we have a very strong department in the government called the ACCC.  The Australian Competition and Consumer Commission does a great job of keeping things fair in Australia between businesses and consumers.  While this can be seen as hampering free trade and an open market, they actually do a great job of keeping a “treat people fairly” mentality prevalent, and in practice there is great competition in Australia.

The ACCC help support other branches of government such as the Australian Communications and Media Authority with things like the 2003 Spam Act.  As per the ACCC, “Under the Spam Act it is illegal to send, or cause to be sent, ‘unsolicited commercial electronic messages’ that have an Australian link.”

What this means is even if you have a prior business relationship, if you haven’t explicitly stated “send me emails about stuff”, businesses are in breach of the act if they send you anything to do with a commercial site at all.

Since moving to the US I’ve noticed that on almost every site I use, if I give my email address I can expect to start receiving a decent amount of crap from that company.  For a lot of businesses it ends up losing them income in the long run by alienating power users who would otherwise use word-of-mouth to promote that business.

Lately I’ve noticed something somewhat sinister.  I’ve been trying to unsubscribe from websites and regardless of what I do, I remain on the lists.  Sometimes it’s because the company obfuscates the removal process (hi Mint – by the way, thanks for sending super-confidential details via email without asking me first!  Shame your site is so pretty, so I forgive you), but I’ve seen several examples of late where the unsubscribe is just plain broken.

So let me name and shame some people.

The worst two:

Lee Jeans is a shocker.  Unsubscribe link that does nothing at all.  I had to add them to a deletion filter, as numerous emails to members of staff did nothing to resolve this.  Even mention of the Spam Act did nothing to help.

JobFox.  Ahh, JobFox.  I tried everything I could to unsubscribe from JobFox.  I edited all my preferences, I clicked on links, I emailed the helpdesk, and then I even emailed individual members of their team.  Nothing.  Also added to the deletion queue.

Then there are a whole bunch of smaller sites (Hi DavinciTeam).  Thankfully some startups at least listen when you write to them.  I got a very impressive response from Mixx via the Director of Product Management, Will Kern:

I wanted to let you know that this has been taken care of.  You will no longer receive marketing e-mails from Mixx.  Let me know if there is anything else I can help you with.

Me: Thanks very much – and thanks for letting me know too (and on a Saturday no less!).

Will: You are most welcome!  Saturday, Sunday, Tuesday, who keeps track ;-)

Very pleasant and prompt.  You guys are definitely back in my good books!

Workology also had a similar bug, but again, I got a prompt, helpful response which was great.

Finally, I wish I could remember the name of the site who had an unsubscribe link to nowhere.  Checking back a couple of weeks later and there was a page but with no options on it.

Update: Just remembered.  Stumbleupon.  I never did get a reply from your customer service team either, although thankfully the emails stopped.

All of this begs the question, why do so many companies have broken systems?  Is it a deliberate thing?  Is QA behind the ball?  Am I just unlucky?  Inquiring minds wish to know.

All I do know is it really hammers home just how underappreciated the asynchronous user experience is.  Incorrect or poorly timed emails, slow-to-arrive confirmations, sensitive information, spam, and poor control of all of this can have a huge effect on the user experience of the site.  While this part of design for a new application usually comes late in the process, it doesn’t mean it should be treated as an afterthought or not part of the user experience.

The Consumerist should stop giving a voice to the entitled.

I love the Consumerist. Honest I do. But all too often the ‘problem consumers’ are making it their soapbox. I’m all for the little guy getting a leg-up against corporations, but there are people who try to achieve what is right when fighting with businesses, and there is people who try to get what they think they are owed.  In this case, the consumer in question isn’t trying to get a freebie, but basically an Apple store exercises its right to refuse a sale because  they suspect the customer is going to unlock the phone.  The customer responds by having a tantrum.

I still don’t understand the volume of commenters at the Consumerist who side against the consumer, but here is one of the few occasions I will join them:

Now I am about to lose it.

“First off, you have NO right to dictate to me what I do and do not do with a product I purchase. If I pay FULL RETAIL PRICE for something, I can smash it with hammers and throw the pieces off the Grand Canyon if I so choose!”

Purchasing an iPhone at full retail price carries with it no commitment to anything, and to make assumptions to what I plan to do with the phone is complete discrimination. What if I wanted to give it as a gift to my girlfriend and she would be able to activate on her own accord???

I explain all of this, calmly mind you, and then ask them to call their corporate office. Mrs. CSR #1 says “We are corporate”

So now they’ve lowered themselves to LYING to my face. Awesome.

“You’re corporate? I make more money in one day then you make all week Mrs. Corporate

Now THEY’RE really pissed and tell me they are calling security. I tell them go ahead! I’m not stealing, Im not breaking things, Im not wildly running around the store naked & screaming “The iPhones are your demi-god! OBEY!” I am trying to PURCHASE a product and agree to the terms of their contract!!!!!

“We are denying to sell to you”

I fold my arms and wait. Three security guards show up and I take the ‘leader’ aside and explain. He has NO idea what to do. He actually calls the main office, his boss, and asks what to do in this kind of situation. The security boss says the store CAN deny me the sale and asks the security guard to pass along that message to me.

Passengers to start getting fingerprinted at Heathrow

In another life I was an information security tutor. Security is such a fascinating topic, both to teach and learn. I can’t get enough of it, and routinely end up taking part in security decisions at my company, just because it’s a startup, and I can. I have a background in usability and user experience research, and I love trying to solve the problem of the balance between user frustration and protection.

Anyway, I digress. Security theatre is such a problem at the moment, 6.5 years after September 11th. Cory Doctorow from BoingBoing has posted the latest ridiculous element of “security” to be enforced, with some excellent commentary.

Britain’s breaking new ground in the slide into a total surveillance state: as of the end of this month, domestic passengers at the new Heathrow Terminal 5 will be fingerprinted and photographed twice, to “ensure the passenger boarding the aircraft is the same person.”

Well, I suppose that if you’re the kind of lazy suicide bomber who believes in dying for the cause — but not if it means rebooking your ticket or, you know, driving to Stansted or Gatwick or East Midlands or Manchester, this’ll work. And that sounds like a pretty good adversary analysis. We all know how easily dissuaded suicide bombers are.

Virgin America

As a frequent reader of BoingBoing I was sold on their glowing recommendation of Virgin America. I later discovered business relations existed, but it did sound pretty cool. So based on said recommendation I purchased tickets to fly to Seattle in May. Virgin’s website recommended that you buy as soon as possible, as they offer all the cheapest fares first. In fact, the exact quote is:

To cash in on the savings, we encourage you to book as soon as possible so you can get the lowest rate.

So I did. And then the fares promptly went down. And then down some more. The tickets now cost $50 less than when I bought them. I had a similar frustration a few months ago with JetBlue. JetBlue then promptly credited me the difference.

VirginAmerica? Not such good customer service. I’ve so far received three template emails, all telling me I have to pay $40 to get a price difference – and one real one. After the third template email I wrote back to let them know of the JetBlue policy as I had seen it, but asked for a $10 credit based on their policy. The reply I got was:

Please call our Reservations line and they will be able to assist you from there.

I just wanted to give you an update. I just called JetBlue and the $40.00 change fee was implemented a month ago. …Just so you know if you choose to fly them again!

Wow, great way to build customer relations Virgin America! A supremely unhelpful email, and I did like the way you rubbed my nose in it. Kudos.

Fighting for consumer rights

I’m a big fan of The Consumerist (although sometimes the commenters are somewhat rabidly against the consumer – what on Earth are they doing at a site called The Consumerist?), but seeing as they didn’t post my great experience with Netgear, I’ll post it here. The long and the short of it is that when my phone was playing up and regular support wasn’t helpful, a well placed email to the executives at Netgear did wonders. They could have just ignored me, but I can’t praise them enough for listening to a frustrated consumer and going to such lengths to sort out my problem.

I just had an interesting experience with Netgear. I bought a Skype phone (SPH200D) from them a year ago and it’s been somewhat buggy since I got it. I spoke with an engineer soon after getting it about the problems I was having, and this initial contact was a great experience and we resolved things well enough with workarounds.

Fast forward to last month and the phone was having call quality issues as well as the bugs. I tried to contact Netgear to resolve it and went through customer support hell – people not reading my requests properly and sending out template responses that weren’t relevant (my biggest pet hate of customer support), support tickets getting ignored, and on top of that a repair/replacement program that I had to pay for, which would leave me without a home phone for 2 weeks.

Using tips from the Consumerist of emailing the executive team, I sent an email to the Senior VP of Worldwide Sales and Support yesterday, simply stating what had happened. I wasn’t trying to get a “freebie” or “compensation”, I just wanted to let him know there were problems with their support program. Ultimately it ‘worked’ and I was getting a replacement phone but the process was very frustrating and did not get any initial resolvement.

Today I received a call from an engineer at their Santa Clara headquarters. He apologised for the difficulties and discussed all the problems with me. He then organised an immediate replacement, with free shipping directly to him so he could analyse the problems first hand. This is a great response. I didn’t expect anything, so to get priority replacement treatment was nice, but the best part was that they actually listened to my problem and *are trying to fix it*. It’s very rare that companies will pay such close attention to a product a year old. Kudos Netgear.