|
RHUA
The Rogers @Home Users’ Association
September 21st, 2000
Meeting Attendees:
- John H. Tory, Q.C., President and CEO, Rogers Cable
- Dermot O’Carroll, Senior Vice President, Network Engineering and Operations, Rogers Cable
- Christopher Weisdorf, President and Technical Director, Rogers @Home Users’ Association
Meeting Agenda:
1. Major mail server problems
- problems have existed for nearly 10 months now
- literally many hundreds of outages have occurred in that span of time
- problems exist on the mail servers in both the Ontario and BC regional data centers
- @Home mail core in Redwood City, California often problematic, delaying mail for hours, days and sometimes even longer
- mail storage services are often unavailable
- SMTP services are sometimes down, so sending out e-mail is not possible
- many duplicate e-mails arrive at times
- mail is lost on the rare occasion, although this is definitely the exception and not the rule
- the service is just plain slow
2. Ontario news server retention times and performance
- problems existed for roughly 9 months before finally being corrected
- binary retention times in Ontario were in the 15-20 hour range at the low point of the problem; retention times of other @Home MSOs’ news servers were sometimes no more than 5 or 6 hours at their low points
- connections would, at times, be severely throttled and even forced into an idle state when downloading binaries, for no apparent reason
- retention times have been in excess of 90 hours since the end of August when the problems were rectified
3. Ontario DHCP server unreliability
- problems were very severe in June, July and the beginning of August
- server reliability was roughly 20-30% almost around the clock, during its lowest point
- reliability still in question during peak hours, although greatly improved since the June/July time period
- one server (24.2.9.85) for ~200 thousand subscribers in Ontario?
4. Peering problems between @Home and UUNET in the Eastern US
- problems were finally solved on August 11th after a minimum 17 month wait
- issue first brought up at initial Rogers@Home management meeting in March, 1999
- issue again brought up at second Rogers@Home management meeting in June, 1999
- issue continually brought up in e-mail exchanges both in anticipation of that second meeting and following it
- the problem was that (@Home service originating) packets destined for any network serviced by UUNET, would be routed through some part of California, due to a total lack of interconnection in the Eastern US
- terrible performance was exhibited on any direct route to UUNET, and/or on any transit route through UUNET in order to reach another destination network
- Rogers@Home management’s handling of the problem and attitude towards it was insulting and untrustworthy to myself, the RHUA and the entire Rogers@Home subscriber base
5. Massive congestion on segment-to-RDC distribution circuits
- the circuit leading from the St. Laurent segment (slnt1) to the Ontario RDC was overloading for about 3 weeks
- one circuit that was deployed to slnt1 last Friday (the 15th) quelled some of the problems, but congestion still exists
- another circuit is on order, but a timeline is unavailable
- the circuit leading from the York Mills segment (ym1) to the RDC has been overloading for the past 3-4 weeks
- ym1 is one of Rogers@Home’s most heavily populated segments, as is slnt1
- all subscribers on those segments are affected by the distribution circuits overloading; over 20 thousand subscribers could be affected
- problems are so severe that average RDC and external latencies are even worse than that yielded from a 28.8 Kbps dialup modem; packet loss is very high (>10%) in some subscribers’ cases; maximum upstream and downstream throughput are obviously greatly reduced because of this
- local interconnects between Rogers@Home, and both MetroNet/RNS and PSINet/iStar are located at ym1, so any subscribers’ packets that touch those networks are going to incur a great deal of latency and unreliability
6. Servicewide mail services credit, due to those facts mentioned in the first issue
- announced to both the entire Rogers@Home subscriber base and the media at the end of January, and also in February
- still pending, but might never come
- word on the street is that the credit will only total $10 per subscriber
- contrary to the original announcements to both subscribers and the media, the credit will not be automatic; subscribers will have to call technical support in order to claim the credit
- ~$2 million for nothing?
7. Numerous sporadic and serious outages suffered by many subscribers with Terayon TeraPro cablemodems
- affects both jet black rectangular and dark blue “shark fin” models
- online light remains lit, while IP connectivity is totally lost
- connectivity can be lost: when a connection is idle for a certain period of time; when large files are being downloaded; when smaller files are being transferred back and forth; and during normal browsing activity
- power cycling the TeraPro will sometimes restore the connection
- problems start affecting brand new subscribers after almost exactly one week
- reports suggest that the problem could be related to the TeraPro’s network inactivity “sleep mode”
- no one at Rogers seems to know even the slightest bit about this extremely serious problem
8. @Home'’s IRC server is no longer operational
- occurred near the beginning of the month and was in operation even before Rogers partnered with @Home
- IRC is a very popular network activity, which consumes relatively little bandwidth
- most EFnet IRC servers ban the entire @Home domain; less than 5 servers in North America currently allow @Home subscribers access and may turn to banning as their loads increase
- @Home, as a service provider, has a strict obligation to provide an IRC server for its 2 million users
9. Congestion on Rogers@Home’s circuit to CANIX/TORIX
- problem solved in August, but existed for over 6 months
- network performance to domestic destinations greatly suffered because of this
- all Rogers@Home subscribers’ packets flowing through this interconnect exhibited high latency and unreliability during this period of time
10. The crisis at technical support
- major problems have been evident since perhaps the April/May period of this year
- problems first peaked in May/June of 1999 and greatly improved in July, as did telephone hold times
- deterioration began in late fall of 1999
- problems were peaking seemingly only yesterday, with subscribers unable to reach technical support at all; only busy signals were possible during peak hours
- tales of technical support “losing” trouble tickets and past documentation is commonplace among subscribers who regularly converse with technical support personnel
- e-mails go completely unanswered for some subscribers
- credits are promised to some subscribers, but are never actually delivered
- referral credits are not delivered to some subscribers
- credits are flat-out refused to the vast majority of subscribers who not only greatly deserve the credits, but can offer proof that they do
- personnel are seemingly never, ever aware of any major outages or incidents that greatly affect the quality of subscribers’ connections; this especially goes for aforementioned issues 5 and 7, along with local fiber cuts
- support personnel seem to have been systematically trained and conditioned to believe that if there’s a problem, it’s almost always the subscriber’s fault; “regardless of how knowledgeable the subscriber might appear, the problem does not exist with the service, but with the subscriber’s own computer”
Christopher Weisdorf
President and Technical Director,
Rogers @Home Users’ Association
|
