The Rogers @Home Users’ Association

September 21st, 2000

Meeting Attendees:

  • John H. Tory, Q.C., President and CEO, Rogers Cable
  • Dermot O’Carroll, Senior Vice President, Network Engineering and Operations, Rogers Cable
  • Christopher Weisdorf, President and Technical Director, Rogers @Home Users’ Association

Meeting Agenda:

1. Major mail server problems

  • problems have existed for nearly 10 months now
  • literally many hundreds of outages have occurred in that span of time
  • problems exist on the mail servers in both the Ontario and BC regional data centers
  • @Home mail core in Redwood City, California often problematic, delaying mail for hours, days and sometimes even longer
  • mail storage services are often unavailable
  • SMTP services are sometimes down, so sending out e-mail is not possible
  • many duplicate e-mails arrive at times
  • mail is lost on the rare occasion, although this is definitely the exception and not the rule
  • the service is just plain slow

2. Ontario news server retention times and performance

  • problems existed for roughly 9 months before finally being corrected
  • binary retention times in Ontario were in the 15-20 hour range at the low point of the problem; retention times of other @Home MSOs’ news servers were sometimes no more than 5 or 6 hours at their low points
  • connections would, at times, be severely throttled and even forced into an idle state when downloading binaries, for no apparent reason
  • retention times have been in excess of 90 hours since the end of August when the problems were rectified

3. Ontario DHCP server unreliability

  • problems were very severe in June, July and the beginning of August
  • server reliability was roughly 20-30% almost around the clock, during its lowest point
  • reliability still in question during peak hours, although greatly improved since the June/July time period
  • one server ( for ~200 thousand subscribers in Ontario?

4. Peering problems between @Home and UUNET in the Eastern US

  • problems were finally solved on August 11th after a minimum 17 month wait
  • issue first brought up at initial Rogers@Home management meeting in March, 1999
  • issue again brought up at second Rogers@Home management meeting in June, 1999
  • issue continually brought up in e-mail exchanges both in anticipation of that second meeting and following it
  • the problem was that (@Home service originating) packets destined for any network serviced by UUNET, would be routed through some part of California, due to a total lack of interconnection in the Eastern US
  • terrible performance was exhibited on any direct route to UUNET, and/or on any transit route through UUNET in order to reach another destination network
  • Rogers@Home management’s handling of the problem and attitude towards it was insulting and untrustworthy to myself, the RHUA and the entire Rogers@Home subscriber base

5. Massive congestion on segment-to-RDC distribution circuits

  • the circuit leading from the St. Laurent segment (slnt1) to the Ontario RDC was overloading for about 3 weeks
  • one circuit that was deployed to slnt1 last Friday (the 15th) quelled some of the problems, but congestion still exists
  • another circuit is on order, but a timeline is unavailable
  • the circuit leading from the York Mills segment (ym1) to the RDC has been overloading for the past 3-4 weeks
  • ym1 is one of Rogers@Home’s most heavily populated segments, as is slnt1
  • all subscribers on those segments are affected by the distribution circuits overloading; over 20 thousand subscribers could be affected
  • problems are so severe that average RDC and external latencies are even worse than that yielded from a 28.8 Kbps dialup modem; packet loss is very high (>10%) in some subscribers’ cases; maximum upstream and downstream throughput are obviously greatly reduced because of this
  • local interconnects between Rogers@Home, and both MetroNet/RNS and PSINet/iStar are located at ym1, so any subscribers’ packets that touch those networks are going to incur a great deal of latency and unreliability

6. Servicewide mail services credit, due to those facts mentioned in the first issue

  • announced to both the entire Rogers@Home subscriber base and the media at the end of January, and also in February
  • still pending, but might never come
  • word on the street is that the credit will only total $10 per subscriber
  • contrary to the original announcements to both subscribers and the media, the credit will not be automatic; subscribers will have to call technical support in order to claim the credit
  • ~$2 million for nothing?

7. Numerous sporadic and serious outages suffered by many subscribers with Terayon TeraPro cablemodems

  • affects both jet black rectangular and dark blue “shark fin” models
  • online light remains lit, while IP connectivity is totally lost
  • connectivity can be lost: when a connection is idle for a certain period of time; when large files are being downloaded; when smaller files are being transferred back and forth; and during normal browsing activity
  • power cycling the TeraPro will sometimes restore the connection
  • problems start affecting brand new subscribers after almost exactly one week
  • reports suggest that the problem could be related to the TeraPro’s network inactivity “sleep mode”
  • no one at Rogers seems to know even the slightest bit about this extremely serious problem

8. @Home'’s IRC server is no longer operational

  • occurred near the beginning of the month and was in operation even before Rogers partnered with @Home
  • IRC is a very popular network activity, which consumes relatively little bandwidth
  • most EFnet IRC servers ban the entire @Home domain; less than 5 servers in North America currently allow @Home subscribers access and may turn to banning as their loads increase
  • @Home, as a service provider, has a strict obligation to provide an IRC server for its 2 million users

9. Congestion on Rogers@Home’s circuit to CANIX/TORIX

  • problem solved in August, but existed for over 6 months
  • network performance to domestic destinations greatly suffered because of this
  • all Rogers@Home subscribers’ packets flowing through this interconnect exhibited high latency and unreliability during this period of time

10. The crisis at technical support

  • major problems have been evident since perhaps the April/May period of this year
  • problems first peaked in May/June of 1999 and greatly improved in July, as did telephone hold times
  • deterioration began in late fall of 1999
  • problems were peaking seemingly only yesterday, with subscribers unable to reach technical support at all; only busy signals were possible during peak hours
  • tales of technical support “losing” trouble tickets and past documentation is commonplace among subscribers who regularly converse with technical support personnel
  • e-mails go completely unanswered for some subscribers
  • credits are promised to some subscribers, but are never actually delivered
  • referral credits are not delivered to some subscribers
  • credits are flat-out refused to the vast majority of subscribers who not only greatly deserve the credits, but can offer proof that they do
  • personnel are seemingly never, ever aware of any major outages or incidents that greatly affect the quality of subscribers’ connections; this especially goes for aforementioned issues 5 and 7, along with local fiber cuts
  • support personnel seem to have been systematically trained and conditioned to believe that if there’s a problem, it’s almost always the subscriber’s fault; “regardless of how knowledgeable the subscriber might appear, the problem does not exist with the service, but with the subscriber’s own computer”

Christopher Weisdorf

President and Technical Director,
Rogers @Home Users’ Association

All trademarks belong to their respective owners.
Send your comments concerning this site to the Webmaster (webmaster@rbua.org).
All other questions should be directed to the Appropriate Regional Representative
Content on site ©1999-2005 Residential Broadband Users' Association.
Design and Implementation by Nexus Internet Services