Recent BroadVoice QoS issues

Louis pointed me in the direction of an open letter from BroadVoice's CEO David Epstein. It reads:

The last seven days have been very difficult for BroadVoice and many of our customers and Id like to sincerely apologize for the inconvenience. I also want to explain, briefly, why service has been intermittent or interrupted for some, and what we have been doing to solve the problems.

On Wednesday May 5th at 2 am, BroadVoice began a major infrastructure upgrade of our core systems. This upgrade, which initially went as planned and had been tested in our lab, represents a substantial investment in future service enhancements that will allow us to add more features, new CODECs and additional user capacity. In retrospect, the upgrade could not have happened at a worse time.

On Thursday May 6th at 9 am a 12-month long dispute with a telecom carrier partner came to a head, when that carrier unilaterally broke off negotiations and interrupted some of our outbound calling services and all of the inbound calling services of over 7,000 BroadVoice customers. Even though BroadVoice has received bills from the carrier that inflated charges due by over 44% and, in some cases, reflected rates that are 13 times the contracted rate, BroadVoice has paid 100% of the undisputed charges. The heart of the dispute involves the appropriate classification of our users calls. BroadVoice believes these calls originate from the Internet which is global in nature. We further believe that the proper resolution of this open issue has significant implications for all VoIP service providers and end users.

Fortunately, we quickly found alternate carriers who have worked with us around the clock to restore service to the affected users. Unfortunately, the combination of the above upgrade and the need to technically interconnect with these new carrier partners on such short notice has exposed some significant bugs in the newly upgraded system that were not present in our earlier testing. Our team and the vendors involved are making substantial progress on stabilizing the system and removing the remaining issues but they are not done yet. We believe we have identified all the remaining issues and are working to fix them as soon as possible.

I am sorry this communication did not go out earlier but we have been 100% focused on resolving the problems and the team here is working non-stop to do just that. We have seen incremental improvements, and a number of issues are resolved, but there is more to do.

We all apologize for any interruptions you may have experienced, and we thank you for your loyalty and your patience. We will continue to work 24/7 to bring you the high-quality, feature-rich service you deserve, as we are committed to making BroadVoice the best VOIP service available.

When I called my BroadVoice number tonight I get my simple "hello world" style AGI-application that recites back the phone number I'm calling from.

So, I'm inclined to think the frustrations I was having Monday evening were more likely due to trouble on BroadVoice's side, not mine. I've been too sick to fiddle with it anymore since Monday, so if my side was broken then, it should have still been...

—Michael A. Cleverly
Thursday, May 12, 2005 at 18:43