US WEST Communications, 1600 Seventh Avenue Suits 3204 Seattle, Washington, 98191 Phone: 206 345-2002 FAX: 206 346-5616 C Scott McClellan Vice President Washington January 31, 1997 Sharon L. Nelson Chairman Washington Utilities and Transportation Commission 1300 S. Evergreen Park Dr. SW Olympia, Washington 98504-7250 Re: January 24, 1997 Letter to Scott McClellan
Dear Chairman Nelson:
This is in response to your letter of January 24, 1997, in which you describe, and ask for resolution of, problems related to customers' ability to complete calls to the Seattle Main D54 central office switch. U S WEST does not agree with the conclusions of the staff investigation cited in your letter; however U S WEST has not yet had an opportunity to review the investigation in detail with the Commission Staff. A meeting has been scheduled to review detailed interoffice traffic data with staff on February 10, 1997.
I have to say I am disturbed by the way this matter has been handled. When the Commission became aware of this problem, it did not call me to ask about the source of the problem and to work on a solution. Instead, it fired off a letter to me, essentially accusing U S WEST of mismanagement and threatening substantial fines. I was given until today to respond to that letter.
However, on Wednesday, before I had an Opportunity to respond, the Commission's letter was leaked to the media, which resulted in long, negative, front-page stories in Washington's major newspaper. That approach to doing business is, to say the least, not conducive to rational, productive problem solving.
Following are the facts as U S WEST knows them at this point, including the magnitude of the problem and the likely cause. I also plan to address the suggestion that U S WEST has violated a Commission rule; I do not believe this to be the case. Finally, I will address policy issues raised by this situation.
On December 24, 1996, U S WEST sent a letter to the Commission that contained traffic data and recent engineering job history for specific interoffice routes. The letter explained procedures used by U S WEST to ensure appropriate interoffice trunking levels and included detailed confidential reports from our trunk servicing system. These reports are used by U S WEST to forecast future trunking requirements based on predicted call busy hour and blockage information. They generally provide for an engineering design that exceeds actual call volume requirements.
The milestone report attached to our December letter provided a summary of upgraded interoffice trunking activity due to increased call volumes based on actual traffic data.
Attached is an actual depiction of traffic usage, call attempts, and percent blockage data for the trunk group between the Seattle tandem 355T and Seattle Main DS4 switch for one week per month from September, 1996 to January, 1997. The report depicts dramatic increases in the usage and call attempts during this time. The number of calls attempted over this trunk group for the week of January 5, 1997 was over 650,000 compared to 340,117 in September. The number of calls nearly doubled in just four months. The report also indicates that usage steadily increased and is spread over many hours as opposed to being concentrated in a single busy hour. The busy hour for this trunk group historically was 10:00 to 11:00 AM. Now the trunk group has steady high usage from 10:00 AM to midnight, with the highest usage being from 3:00 to 4:00 PM. The average usage per trunk nearly doubled from November to January. Detailed traffic reports are too voluminous to include with this report; the specific reports will be shared with the Commission staff at the February 10, 1997 meeting.
U S WEST assigned a task force to investigate these dramatically increased call volumes and fundamentally new call characteristics. Part of this work focused on making a determination about what the specific changes were and why they were occurring. U S WEST determined that a great deal of traffic which ordinarily flows into Seattle Main on direct trunks was overflowing onto the tandem. U S WEST determined that a large number of calls were of much longer duration than expected for voice calls. Many of the calls had hold times of more than two hours and were made primarily to lines with modems. Thus, it is our preliminary determination that much of the changed traffic is data or Internet related.
Reinforcing this determination is the fact that during December, 1996, AOL began a very public offering of flat rated unlimited Internet access. The avalanche of traffic on the network increased by 60% in just two weeks.
Although U S WEST has taken extraordinary steps, including increasing the number of high-capacity trunk lines between its Seattle and Bellevue offices from 96 to 240, it hasn't been enough. Interoffice trunk blocking cannot be resolved by simply adding additional network capacity.
In an October 1996 letter, U S WEST advised the Commission that Internet calls generally last forty to sixty minutes while traditional calls last six minutes. If consumers can access an Internet provider and do not pay for that access on a usage basis, there is no incentive to terminate the call even if the subscriber is no longer using the service. Because consumers have had difficulty getting through to their Internet provider, they frequently will leave the connection up indefinitely.
U S WEST has identified approximately 15 Internet providers (IPs) served by the Seattle Main DS4 switch. We have also identified several ISP being served out of the alternative local exchange company (ALECs) offices. The traffic stimulated by these Internet providers is very different both in quantities and duration than traditional voice traffic for which the public network has been engineered. On average, local customers' usage is 3.5 hundred call seconds (CCS) or less than six minutes, during the busiest hour of the day. The ISP traffic is much higher, with many of the users stimulating 36 CCS per hour for hours at a time (the equivalent of being on the line an entire hour). In fact, connections have been identified that were established on Monday morning and remain up until Friday afternoon.
U S WEST's local network is designed and managed to long-standing traffic engineering standards. These standards are based on statistical models that predict the probability that calls will be delivered at prescribed blocking levels, given specific traffic characteristics. Local networks are designed with the objective of blocking less than one percent of calls during the busiest hour during the busiest period of the year. To meet these blocking objectives, a variety of trunking configurations are applied in local networks. In large metropolitan networks, like Seattle, traffic is completed over circuits directly between end offices and through tandem or transit offices that connect multiple local offices.
Trunks between a specific end office and a tandem are common trunks and shared by all traffic connected through the tandem. When callers from all over the Seattle area are trying to reach ISPs served in a single switch, and the ISP cannot answer all of the calls, the direct trunks between end offices begin to fill up, causing traffic to be rerouted through the tandem. If the calls are still not answered, the tandem quickly reaches capacity. When this occurs, the calling customer receives a no circuit announcement (all circuits busy) or tone.
Calls to Internet providers have introduced additional traffic management complexity. Many Internet providers have significantly more customers than access connections. When all of their access connections are in use, ISPs' customers continue to generate multiple, unsuccessful attempts. Because these attempts are generated by computers, unsuccessful call attempts may be generated until a connection is established, These multiple attempts create congestion in the local network, that block both ISP and traditional voice traffic.
Further worsening the congestion at the tandem is the fact that some ALECs are delivering all of their traffic to the tandem. The combination of the ALEC traffic and the IP, traffic stretches the engineered capacity of the tandem switch.
Some ALECs are connecting directly to the end office, which helps alleviate some of the pressure. U S WEST is working with these providers to re-configure their networks; U S WEST is also reassigning its operator services traffic to other switches.
Another factor in the blocking is the storm conditions that Washington recently experienced. The network is typically overloaded during such storms due to extraordinary "off-hook' conditions. Our analysis clearly indicates that many blockages that occurred in November and December were storm related.
As previously reviewed with this Commission, there are two processes by which we decide when to add trunks. Planned servicing is a monthly activity designed to bring trunk groups up to forecasted quantities based on historical data. Demand servicing occurs when, during continual monitoring of trunk groups, unforecasted demand appears and immediate action is required.
U S WEST has continued to add capacity to the tandem trunk group and to selected direct trunk groups. An additional 144 will be augmented to this trunk group by February 7, 1297. Direct trunk groups have been augmented among and between other offices in the Seattle area to decrease the total traffic presented to the tandem switch.
An additional 144 trunks will be added when an equipment addition is completed in March. This added capacity constitutes an increase of 127% over 17 months on this trunk group alone. This increase in trunks will have increased the call handling capacity of the trunk group by 136%, Prior period growth has been steady at approximately 12% per year or 1% a month.
U S WEST continually monitors the load balance in our switches to ensure all IP lines are not in the same part of the switch and the load is distributed over the whole switch. U S WEST also continually monitors trunking data and adds capacity where needed. However, because of the sudden growth, we have exhausted the capacity of trunk ports and additional equipment typically requires twenty-six weeks to engineer, install and certify for service.
U S WEST is also looking at several long term solutions which include:
Another solution, not within U S WEST's control, requires that the ISP have enough lines to complete the calls stimulated by their own advertising.
U S WEST has set up subscriber line usage studies to help determine how many lines they need. (This only covers the IPs on our own network and would not include those on the ALEC switches.) Network management controls have been invoked on U S WEST IP subscriber numbers resulting in the reduction of severe blocking during peak times to avoid impact on other callers.
The complaints that the Commission has received, and the customer concerns expressed by those complaints, are of major concern to U S WEST. U S WEST is also concerned that the appropriate legal and engineering standards be applied to determine whether a rule violation has indeed occurred. U S WEST has analyzed the provisions of WAC 480-120-515 and in our view the rule does not apply to the blocking problems which have generated the customer complaints.
The rule, by its express terms, sets forth standards for service quality measurements which are the minimum acceptable quality of service under normal operating conditions. The rule explicitly does not apply to service which is to be achieved during periods of emergency or under extraordinary or abnormal conditions of operation such as those resulting from disruptions of service caused by persons or entities other than the local exchange company.
The Commission, in its January 24 letter, has stated that blocking at the tandem may not exceed 0.5% and has suggested that this design standard is applicable to the blocking problems which have recently been experienced. However, the referenced section of the rule, WAC 480-120-515 (2)(b) sets forth a design standard of B.005 for intertoll and intertandem facilities. Intertoll and intertandem facilities are associated with toll traffic, and that design standard therefore applies only to toll traffic. The relevant standard for this situation is contained in WAC 480-120-515(2)(a), which applies to interoffice facilities and which sets forth a design standard of B.O1 level of service. The calls and the blocking which are the subject of this discussion are on interoffice trunk facilities, and the 1% standard is applicable.
Recognizing that the Commission may differ with U S WEST's interpretation, U S WEST submits that the rule is at best ambiguous. Furthermore, the rule was promulgated at a time when the Commission held local exchange service to be a monopoly. Thus, whatever the rule may have meant in 1993, it clearly did not apply to the situation a LEC would face when required to interconnect with and accept traffic from multiple other LECs at the tandem, such as is the case today. Further, ISP traffic was virtually non-existent at that time compared to today's levels.
Finally on this issue, even if the 0.5% were applicable to the local tandem, it is U S WEST's position that the standard would not apply to the blocking occurrences described in the Commission's letter. As noted above, some of the days where the standard was exceeded were blizzard or winter storm days, which would come under the force majeure or catastrophe exceptions in the rule.
The call blocking which is due to unlimited, flat-rated Internet promotions such as offered by AT&T or America Online (AOL) also falls squarely within the exceptions contemplated by the rule _the unlimited access to the Internet has resulted in extraordinary and abnormal conditions of operation. These conditions have never been faced before by any provider, and they fall far outside the scope of normal telephone usage. Call durations are, on average, more than ten times what the network is engineered for. While this new information will need to be taken into consideration for the future, it has not been known until now. A local exchange carrier is not required to achieve the design standards under these circumstances, which are clearly not normal operating conditions, The design standards also do not apply if the disruption of service is caused by persons or entities other than the local exchange company. These flat-rated Internet promotions are wholly outside the control of U S WEST and to the extent that the unlimited access has disrupted service, the cause is clearly persons or entities other than U S WEST.
The solution to this problem is to first acknowledge that the rate structure in Washington does not incent efficient use of the telecommunications network. It is also important to understand that the problem associated with interoffice trunk blocking cannot be resolved by simply adding additional network capacity. The main source of the problem is the improper rate structure that exists in Washington. As long as retail and wholesale consumers receive unlimited local service for a flat rate, there is no incentive to utilize their telecommunications service in an efficient manner and interoffice trunk blockage will continue. This is not meant to imply that U S WEST is advocating a blanket repeal of the current ban on measured service. There are likely several pricing structures that would enable people to use the network vigorously while not allowing the waste of this valuable resource.
In Washington Internet subscribers have no incentive to disconnect their connection to their Internet provider when they are no longer using the service; the effect on the local network has been staggering. With unlimited usage at a low flat rate, there is no reason for them to take down the call. In fact, because of the difficulty in getting on line, there is more reason for users to leave the connection up. U S WEST can continue to add as many trunks as possible and it still will not solve the problem. The network was not designed for calls that last five days. Subscribers who use the network in this interoffice not only tie up their serving central office switch, but they also tie up interoffice facilities and tandem switches indefinitely, as well as the ISP's serving central office switch.
Telecom regulations designed for the monopoly era, when "telecom service" meant nothing more than a simple voice call, no longer work in the new environment. Today, many companies provide local service and "telecom" encompasses a host of voice, data and video services, all delivered over the same lines as the old-fashioned telephone call. Unless this outdated, monopoly-era regulatory thinking is changed to work in this new world, Washington's statewide local telecom networks will receive too little investment, too much use and the result will be digital gridlock that will make the current situation seem minor.
The inefficient use of the network by Internet users is further complicated by the new area of multiple telecommunications providers. When IP's change their telecommunications provider, a massive shift occurs in the traffic patterns. Interoffice facilities, the local tandem switch and overflow trunk routes are no longer adequate, virtually overnight. U S WEST may correct the problem for an IP, but if the IP changes providers, they may completely change their traffic pattern and create new blocking problems. ALECs who do not coordinate this massive change with U S WEST create added burdens on the network.
Couple this phenomenon with the Commission's recent arbitration rulings that provide for all traffic to route through the local tandem and further network inefficiencies are incented. New investment is also not incented when U S WEST is required to make all of its network facilities and services available to new local competitors at prices that do not cover the cost of the new investment. Rather than threaten U S WEST with substantial fines and accusations of mis-management, the commission should attempt to better understand the current demand on the telecommunications infrastructure and the challenges now before all telecommunications companies.
The Commission needs to examine its pricing policies in light of these new issues and work with the industry to develop solutions, not to exacerbate the problem. More appropriate solutions could be obtained through industry forums facilitated by the Commission that address long term industry solutions. Re- examination of outdated regulatory policies would also be most productive. Areas that require industry attention include improved forecasting by all local service providers, improved coordination of significant traffic changes, improved compensation ar-rangements, implementation of standardized industry network controls, and new investment incentives.
It is simply inappropriate to consider this as a U S WEST problem when industry journals indicate this is a nationwide problem. Furthermore, it is uncertain how long this Internet traffic will remain on U S WEST's network.
Recent press releases and ad campaigns indicate that IPs and/or associated companies intend to build their own infrastructure and eventually bypass the "telephone network". (See the attached article.) If U S WEST continues to build to handle excessive traffic, our ratepayers will be left with the cost of stranded capacity when ISPs and ALECs build their own facilities. This is clearly not responsible public policy.
I look forward to future discussions with you and your staff to productively address this problem along with other industry participants. If you have any questions please call me at 2O6-345-2OO2.
Very truly yours
This is <http://www.panix.com/~oppedahl/lads/mcc.htm>.