June 9th, 1999

Hello again, all. As many of you know, on Friday, June 4th, I met with several officials again, from Rogers @Home senior management. Bill Lukewich, General Manager of Rogers Cablesystems for the Greater Toronto Area (my original contact); Vic Pollen, Director of Operations for Rogers @Home; Rob Boucher, Director of Network Operations for Rogers @Home; Susanna D'Arcy, GM of Rogers Cablesystems for GTA West; and Stephen Haynes, Director of Customer Care for Rogers @Home, all attended. Every attendee was issued an agenda for the meeting, which also included ratings on all of the corresponding Rogers and @Home backbone routers for Ontario, along with two newspaper articles written on MindSpring's customer service and the CRTC's open cable ruling. The purpose of this second meeting was to address all of the outstanding issues and obtain timelines for the resolutions of those issues. I believe this was achieved at the latest meeting. I will now go over each of the discussion topics and include the responses of senior management to each of those topics. Once again, sorry for not posting this any earlier. I didn't get a chance to put this all together over the weekend. It's a great deal larger than the report on my first meeting!

1. Official representation in the newsgroups- this is something that's coming soon. Management never provided an actual timeline at the last meeting; they only referred to the implementation date as sometime "within the next few months". This time I pressed them for a timeline and they responded that July is the target, with August being the month when we'll be seeing active participation from the representative.

2. Peering and routing issues- I first brought up the fact that there is a severe lack of peering with Canadian service providers, i.e. AT&T Canada, MetroNet, Teleglobe, UUNET Canada, Sprint Canada, Sympatico, ONet/CA*net and Toronto-area ISPs. In conjunction with this, I also made sure to mention that inefficient routing through Mae-West (San Jose, CA) and PAIX (Palo Alto, CA) results in poor performance and high latency to nodes located in Ontario and mideastern Canada. At my first meeting, only UUNET peering at PAIX was a major issue; unfortunately things since then have severely deteriorated. Basically now, if you want to reach any ISP in Canada, other than the few that can be reached through RNS or PSINet Canada/iStar, your packets will travel through either PAIX or Mae-West, in order to reach their destinations. This is a major problem and leads to poor performance for all Rogers @Home subscibers in Ontario, who are trying to reach domestic destination nodes. Mr. Boucher, the Director of Network Operations, explained to me that this had to do with a capacity issue on @Home's end and that it should be corrected soon. They were not able to provide a date, but hopefully their prediction will prove to be correct. Up to this point, @Home still only peers with UUNET on the west coast, so that also results in a performance hit for all @Home users on the east coast. I asked what the status was on CANIX peering, as they promised at my previous meeting that peering would be quite evident by the end of May. Again, the Director of Network Operations was not able to provide any dates on peering, so this issue is still totally unresolved. To be fair to management, though, there have been many problems at CANIX that might have thwarted their peering efforts. Mr. Boucher told me that they're looking to first get a DS3 (45 Mbps) to CANIX and expand that to an OC3 (155 Mbps) by fall. TORIX peering (this point serves to interconnect all Toronto-based ISPs) happens to also be in the works. Rogers is looking to deploy an OC3 to TORIX and expects quite a bit of traffic to travel over it. A new peering agreement with UUNET Canada is another thing they're working on. The Director of Network Operations stressed that after UUNET Canada and CANIX peering is in place, many others with prospectives like AT&T Canada, Bell Canada and IBM Canada, will follow quickly thereafter. They were not able to adequately explain why peering with affiliates AT&T Canada and MetroNet has yet to be established. Hopefully this will change soon.

Another interesting bit of news is that we'll be seeing the implementation of a second RDC (regional data center) in Ontario, sometime over the next couple of months. This will cut the current level of local traffic almost in half and the performance increase should be clearly evident. Creating servers accessible by region is something else that will be completed around the time of this second RDC. The response times of all the RDC servers, such as mail and news should be very speedy after these two changes. The last few things I spoke of under this current topic, were the Shaw Fibrelink interconnect with Rogers, the Sprintlink interconnect with @Home, the Rogers Ontario bordercore router (uplink and some of the illogical routing on @Home's end. The SFL interconnect is running much more smoothly, due to some very recent changes, so that didn't need to be addressed. The Sprintlink interconnect in NJ, with @Home, has gotten marginally better over the past two or three weeks. Whereas it used to drop at least 15% of all packets during peak hours, now it drops roughly 10%- still completely unacceptable. Just a week ago, that number was down to 6-7%, but it's now back up into the double digits. There's a huge amount of room for improvement, although the problem has been addressed somewhat thus far. As a great deal of our traffic travels across Sprint's circuits in the US, this problems affects most, if not all, of us. Next, the ON bordercore router is still performing badly; earlier tonight I logged average latency to be in the 100 ms range, although this device is located closeby in the Ontario RDC. Obviously this needs to be rectified and I trust the problem is being worked on. The Ontario bordercore router serves to interconnect with practically all Canadian providers, so it's key that it performs well. The illogical routing I mentioned, has to do with @Home routing traffic through east coast peering points, to get to west coast destinations and using west coast interconnects to get to east coast destinations. This makes little sense to me and ends up costing us all performancewise, so I made this fact clear at my meeting. I would expect to see some results in this area by the end of the month.

3. IO capacity expansion vs. new users- luckily, this is something that hasn't been a thorn in our sides for quite some time. At least the @Home interconnects on the east coast haven't been too bad, Sprintlink notwithstanding. Before, seeing 200 millisecond latency to east coast routers would be fairly commonplace, while now, seeing anything over 100 ms is a rare occurrance. The Director of Network Operations told me that the Ontario RDC is now running Gigabit Ethernet, to account for the current trafficload and that new HA (high availability) servers are being deployed in the RDC, to further bear the brunt of the increasing network traffic. The HA servers are able to practice active load balancing and other functions, that will contribute to the overall stability of the Ontario RDC. Time will tell if this turns out to be in our favour.

4. Block sync issues- this is an ongoing issue that affects people in many different regions across Ontario and BC. Work is continuing in the Kitchener-Waterloo and Brampton areas to tackle these problems. My RHUA contacts in these various areas have been instrumental in getting the names of affected subscribers, out to me and ultimately, contacted by Rogers personnel. Please, if you are experiencing block sync difficulties, don't hesitate to contact your regional RHUA representative. If you're in Toronto or Etobicoke, please contact me directly; Mississauga subscribers should contact Wojtek Zlobicki; Brampton users, please get in touch with Adrian D. Henderson; Ottawa people, just e-mail Buzz Haze; Kitchener-Waterloo subscribers, please get a hold of Scott Brown; London users please provide Mike Haggerty your relevant info; and you BC users experiencing block sync problems, just contact Ken Moren. Anyone else having these problems, please e-mail me directly. In addition to some of these individuals taking part in newsgroup discussions, their contact info shall be posted to the RHUA website over the next couple of days. I give you my personal assurances that upon hearing of your block sync problems, your concerns will be forwarded directly to the General Managers of your corresponding areas. You will be contacted and your problems dealt with on an individual basis. I also recommend people asking for full credit on their monthly bills, for every day that they experience the constant loss of block sync. The only way to do this is to provide clear evidence of the problem, otherwise you probably won't get more than 50% credit. I suggest logging ICMP queries to your gateway, for every day that the problem exists. I advise pinging your gateway 200 times per query, with packets no larger than 100 bytes each (I recommend 48 bytes of data per packet). I recommend using a prog like Netlab for this task. This and other such progs can be easily found at shareware websites like TUCOWS. People experiencing the constant loss of block sync, will see plenty of severe latency spikes, along with packet loss that exceeds 2%, all to their first hop, or gateway. I strongly advise people to seek credit once per month. Simply call or e-mail tech support your request. When they ask for evidence of your problems, simply zip up the current month's ICMP gateway queries and send it away. You'll get your credit, I guarantee it. If they give you any trouble, please let me know. Just for the record, I've lost block sync less than ten times in all, since I first signed on to the service back in October. I get an average 2 ms RTT (round trip time) to my gateway, drop perhaps 1 out of 500-800 packets and I rarely see spikes past 10 ms. People with healthy physical connections should be seeing similar numbers.
5. The state of technical support- this is a topic that probably all of you will know something more about than me. I repeated your frustrations over the past few months at the long hold times, non-responsive e-mail and often lack of knowledge of the personnel on current issues. Stephen Haynes and Vic Pollen told me that they keep tabs on the average hold times and that they were reduced dramatically to around 15-20 minutes after the beginning of May. As well, Mr. Haynes stressed that all e-mail has been answered within a 24 hour period, also since the beginning of May. I was notified that there was an average hold time of only one minute during the Victoria Day Weekend. They do concede, however, that hold times often range from 30 to 60 minutes, during peak hours, i.e. 7pm to 12am.

Stephen Haynes stated that we'll see some more big improvements, such as reduced hold times during peak hours, around the end of this month and that they now train their support staff around the clock. In addition to this, a service management center will be operational between the 15th and 30th of this month. This center will dramatically improve two-way communications between subscribers and tech support, so that things like trouble tickets aren't closed without the user's knowldege. As I said, over the past few months, I've had very limited communication with tech support, so I need as much input as I can get on this subject, from all of you. Please let me know if the state of tech support has improved as management claims, over the past five weeks. Remember, I'm only interested in your experiences with tech support, particularly the e-mail aspect, since May 1st. Believe me, I'm well aware of what happened before then.

6. Network status page- I suggested that this be implemented at my last meeting, but unfortunately haven't heard a peep about it since then. Mr. Pollen said it will be done in a July/August timeframe and that it'll either appear as a link off of the Buford portal, or end up as a separate site, entirely. I stressed at my last meeting that such a site can be put together for a low cost and within a few, short days. Because of that fact, I don't understand why it's taking management many months to follow through on such a simple idea: something that would benefit them, probably even more than us. I don't think I even need to explain how this status page would be able to reduce a significant number of calls to tech support, during outage periods. In any case, I expect that this status page will be well worth the wait.

7. DOCSIS modem standard/availability; coexistence with LANCity devices; notices sent to older subscribers to clarify monthly pricing, until such devices become available for retail- I lumped all of these together into one topic for simplicity, seeing as they're all related. Mr. Pollen explained that Rogers will be phasing in the Terayon TeraPro cablemodems over the next few months and that they'll coexist with the LANCity devices, without any problems. At some point in the near future, there shall be as many as three different brands of cablemodems in operation, in any one area. Management isn't anticipating any difficulties. At my last meeting, I had asked that the modified monthly pricing scheme be relayed to subscribers that signed on to Rogers @Home, after September of 1998. No clarification was ever given. Basically, the root of the issue lies in the fact that many people think that they have only six months, from the date of their first day of service, until they have to pay a monthly rental fee for their cablemodem. This simply is not true, but has never been officially clarified by management. Just for the record, no one will have to pay even one penny toward their cablemodem monthly fees, until such a time that DOCSIS-compliant devices are widely available at Canadian retail locations, for purchase. That's still a good six to eight months away, so you don't have to worry about any extra fees until then. I hope that we'll see this official clarification by the end of the month.

8. PST rebate- this issue was mentioned at my last meeting and was promptly resolved. It had to do with people receiving monthly credit, but not seeing the proportional amount of tax credit on their bills. I have heard absolutely no complaints from anyone since March.

9. Zenith users' status and Wave->@Home domain switchover- the former issue was resolved a while back; I haven't heard anyone complain about not being able to get their old Zenith upgraded to a LANCity. The latter issue has been somewhat of a problem in BC. People were issued some strange e-mails regarding the switchover, as well as one or two e-mails that I would consider as rude. The attendees expressed that every Wave subscriber was contacted numerous times, well ahead of the potential switchover date, so that they would not be caught off guard when the actual time came. Also, many Wave subscribers apparently resisted the switchover and as a result, it was postponed by a while. I've been alerted that since the switchover, things have gone well. People experiencing problems were contacted and had their concerns addressed, albeit taking quite a bit of time in some cases, during peak hours. The switchover was definitely not without incident. I don't think anything like this will every happen again. If it does, please make it known to the RHUA immediately. Your concerns shall be relayed to management and you've got my word on that.

10. Rogers's plans for the new AT&T backbone for @Home- Vic Pollen and Rob Boucher said that this is all a matter of phasing in different regions, to accomodate their individual bandwidth needs. So the less bandwidth your region has available right now, the quicker you'll be connected to AT&T's backbone in the US. The attendees stressed that it's all a matter of time. As it stands right now, there are roughly 600 nodes and 132 POPs in Ontario that need to be phased in, so this can't happen overnight. Check out http://work.home.net/whitepapers/backbone.html for more information on the AT&T backbone. As I understand it, @Home's architecture will borrow heavily from what you see there.

11. Further rate caps and forced proxy usage?- people keep on spreading totally unfounded rumours about these two issues in the newsgroups, but you can rest assured that nothing of the sort will ever be implemented with our service. Downstream rates will continue to be capped at 3 Mbps, while upstream rates will stay at 400 Kbps. Upstream rates will NOT be capped any further, as some of you may believe. Again, I give you my personal assurances that there will never be any further rate caps to our existing service, without there first being a proportional reduction in our monthly fees. There are no plans for either. As for forced proxy usage, there is a software component of the newest @Home client release, which effectively "forces" proxy usage. This can easily be avoided, either by using an older client version, hacking the existing version, or not using the client at all. Let's face it- it's a Good Thing if the casual subscriber is using the proxy servers because they end up saving all the more bandwidth for the rest of us. The choice to use the proxy server is still there, only you have to possess a bit of technical know-how in order to be able to circumvent it. I've been notified that port blocking, a la Sympatico HSE, will never find its way into our service (well, other than the NetBIOS blocking that's already there). And speaking of Sympatico HSE, I asked if anything similar to the Redback PPPoE (point-to-point protocol over Ethernet) service will ever be instituted on Rogers @Home. I got a resounding answer of "NO", so there's nothing to worry about on this front.

12. MTU/RWIN tweaks for Windows Oses- this is something that I thought would be a good idea to see put into practice. Unfortunately, there are no plans for any such tweaks to be released. I voiced my discord, but as it stands, you're all on your own. So those of you who at one time tweaked your connections for dialup modems, please make sure to re-tweak your connections for broadband services. One dead giveaway that your connection isn't tweaked properly, is low throughput to all sites (<128 Kbps), even in the dead of night, coupled with no loss of block sync. I recommend picking up an MTU-related program at a site such as TUCOWS and trying out different settings. I promise to post some of my prior writings on cablemodem tweaking, on the RHUA website. Please stay tuned.

13. Mail/news server problems- this is an interesting topic. At one time, the RDC servers simply couldn't handle the amount of traffic pouring in because of software problems. This was addressed efficiently by David Low and others at @Home, shortly after my first meeting. Unfortunately, hardware-related problems started affecting the RDC servers around the beginning of April. It took seven weeks until the news server was back up to par and performing admirably again, but the other servers, notably the mail server, perform badly to this very day, during peak hours. For instance, I logged an incredible 31% packet loss to the news server, only earlier tonight! I've had to wait in excess of 30 seconds on some occasions, to retrieve only one or two e-mails. The same box that acts as the Ontario RDC mail server, also happens to serve DHCP requests. That means this piece of equipment is absolutely crucial to all of Ontario and must be repaired as soon as possible. The RDC outage last night did nothing to help performance in this area, I'm afraid. Luckily, the mail server is seemingly the only server that is affected by this unreliability at this time. The proxy servers seem to be performing many times better than they used to, since only about a couple of days ago. I expect the mail server to follow suit. Vic Pollen and Rob Boucher volunteered that the new HA servers will help reduce the traffic load on the current RDC servers.

14. Rogers @Work service deployment- I've had a number of people ask me when @Work will be available in Canada. The attendees told me that this service will be a totally separate offering from @Work in the US; it will even be called something completely different; July/August is the rollout timeframe; it shall be cablemodem-based, just like Rogers @Home; it'll access the internet via another network; and the retail cost is unknown at this time, but I think it's safe to say that the basic service will cost more than $39.95 a month.

15. Recent Brantford Wave gateway (uplink problems- this is an issue that crept up within the past few weeks and was resolved just before my latest meeting. The gateway was apparently misconfigured and was dropping anywhere from 50-88% of all ICMP packets. Average throughput was greatly reduced to below 128 Kbps and it affected anyone in Brantford who used that gateway. An incompatibility between the hardware used and a version of Cisco's IOS, was the root of the problem. Like I said, all is well right now, with respect to this issue.

16. Recent changes in the Rogers @Home EUA- this is the last issue that was discussed at my meeting. As you may or may not know, the Rogers @Home End User Agreement, which can be found at http://rogers.home.com/CustomerSupport/EUA.html, was altered at some point in May, I suspect the 19th. Of particular interest is section 7, subjection (j), which states that "you will not use the Equipment or the Services to, directly or indirectly: operate a server in connection with Rogers@Home or the Services including but not limited to mail, news, file, gopher, telnet, chat, web, or host configuration servers, multimedia streamers, or multi-user interactive forums." I made it clear that this was not acceptable and the attendees clarified their stance for me. They explained that the change was implemented to specifically catch the abusers of the service. Casual servers apparently are allowed; this is defined as a non-advertised, low bandwidth server, that is preferrably operated outside of peak hours. I've run servers on many occasions to transfer some files with a few friends and there's nothing wrong with that. Unfortunately, there are people out there who will not only run active servers 24 hours a day, 7 days a week (obviously using a lot of bandwidth up in the process), but they'll get advertising revenue for each login as well. I stressed at my last meeting that these people should not be using Rogers @Home at all, as it is a residential service. Rogers @Home isn't a home T1 and it shouldn't be treated as such. In the end, it's up to you how you actually use your server, but please try to keep it among friends and be mindful of how much bandwidth you use. Remember, you'll be warned if you use too much bandwidth within a given period. If you stay within the above definition of a casual server, then you have absolutely nothing to worry about.

In conclusion, I would like to express my thanks to all those in the RHUA who have taken a lot of their free time to forward the cause of our organization. Because of your effort, the RHUA has expanded, the website's traffic is growing rapidly month by month and our collective voice is getting louder and louder. I'm happy that an open channel of communication still exists between me and Rogers @Home senior management, and I sincerely hope that the above issues which are unresolved, are addressed quickly and efficiently.

Yours Truly,

Christopher Weisdorf

On Behalf Of:

The Rogers @Home User's Association

All trademarks belong to their respective owners.
Send your comments concerning this site to the Webmaster (webmaster@rbua.org).
All other questions should be directed to the Appropriate Regional Representative
Content on site ©1999-2005 Residential Broadband Users' Association.
Design and Implementation by Nexus Internet Services