Request for Coordination of Cells In Frames Specification
ATM Forum / 96-1104
Dr. Lawrence G. RobertsAugust, 1996
The Cells In Frames Alliance requests the ATM Forum to examine the attached specification of the Cells In Frames (CIF) Protocol with respect to its interoperability with the ATM Forumís latest specifications. If it is determined that the CIF specification will inter-operate correctly, then it is requested that the ATM Forum so notify the CIF Alliance.
This document has been prepared to assist the ATM Forum. It is offered as a basis for discussion and is not binding on the contributing organization, or on any other member organizations. The material in this document is subject to change in form and content after further study. The contributing organization reserves the right to add, amend, or withdraw material contained herein.
Cells in Frames (CIF) has been specified by the Cells In Frames Alliance in order to extend ATM protocol end-to-end to the desktop, even where it is uneconomic to install a new NIC card for the end-station on the desktop or even to open its case. All that is necessary with CIF is to downline load some additional software into the end-station and to connect it to a CIF Attachment device with ATM uplinks. The terminal will then operate functionally identically to an end-station connected directly to an ATM switch with an ATM NIC card. Running CIF, the end-station is in fact operating with ATM signaling and ATM ABR flow control over the legacy line protocol. ATM cells are packed into frames so as to permit the legacy protocol to operate as efficiently as it can. A CIF header before the cells includes the ATM cell header so that segmenting or reassembling ATM cells back and forth into CIF frames in the CIF Attachment device is rather simple. The benefit of using CIF for the final link to the desktop is that ATM can be extended far more economically to the desktop, a critical step if the benefit of ATM QOS and ABR flow control are to realized for the majority of users.
Figure 1. Typical CIF Installation using Ethernet
Motivation For CIF
Recent news stories have suggested that "ATM has Lost the Desktop". This story is based on the extremely low cost of switched Ethernet compared to ATM for the link to the desktop. Unfortunately, the story will turn out to be true unless far more economic access to the desktop is achieved for ATM. And, if the majority of the desktops connect with switched Ethernet, both the excellent and well defined QOS and ABR flow control of ATM will not be available end-to-end. Neither is useful unless it is end-to-end. Thus, unless RSVP is translated into ATM SIG 4.0 and TCP is depended upon instead of ABR flow control, there will be no need for ATM in the LAN backbone. But if TCP and RSVP are sufficient, then why translate at all? Under that assumption, switched Ethernet at 10 Mbps, 100 Mbps and 1Gbps would be quite sufficient for most LANís and might also be less expensive.
ATM at one time was the only way to achieve high speed switching due the advantage a fixed cell length had in building hardware switches. But today, this advantage has evaporated with the progress in ASICís and one can build a chip which switches Ethernet frames as easily as one to switch ATM cells. Thus, ATM cannot hold an advantage based on fast switching. Its only real advantage today is the Anchorage Accord protocol stack.
The truth is that the ATM Forumís SIG 4.0, PNNI 1.0 and TM 4.0 are such a powerful and important protocol suite that the IP community will require many years to duplicate it. The IETF has been working on RSVP to substitute for SIG 4.0 but they have no work planned to upgrade TCP to achieve the capability of ABR explicit rate flow control and they also do not have any capability planned which will be as effective as PNNI 1.0. Both capabilities are critical to future network operation and without them the Internet probably will have serious troubles in the near future. The ATM Forum protocol suite is complete, well documented and will be deployed well within the next year. Thus, if the ATM protocol can be used to connect all the way to the end-station, even over legacy facilities if that is more economic, then the LAN and the Internet stability and functionality will be greatly enhanced and ATMís role in the LAN backbone will be greatly increased. If we cannot find a way to reduce the cost of access using ATM protocol to the desktop to a level close to switched Ethernet very quickly, then ATMís potential utility and usage will be seriously undermined. CIF is a very simple way to permit the use of ATM protocol to the desktop economically in the near future. As such, CIF will be of major benefit to the advancement of ATM in preserving the use of ATM in the LAN backbone, and then with time, the desktop as well, as users need to upgrade and ATM interfaces get less expensive.
The need for End-to-End Quality Of Service
QOS is necessary to achieve delay or bandwidth guarantee requirements. Voice and video are clear cases where QOS must be signaled if one wants any reasonable quality. Less understood is the need for QOS on data calls to obtain a guaranteed responce time. But in all these cases, the QOS must be signaled completely from end-to-end. Also, all network elements like routers, bridges, and switches must implement multiple queues for packets with weighted fair queuing. Thus, to achieve QOS in a network, all network components must be upgraded or replaced to incorporate QOS signaling and weighted fair queuing with multiple queues. If one segment of the path is not upgraded, QOS can be totally lost. Ethernet switches, being typically hardware switching systems, will need to be totally replaced to introduce QOS. CIF attachment devices (CIF-ADís) are much the same as Ethernet switches which have already been upgraded to incorporate multiple queues and weighted fair queuing to support QOS. With these CIF-ADís, just like ATM switches, voice and video can be supported with quality.
The need for End-to-End Explicit Rate Flow Control
The second major addition which is supported in ATM is Explicit Rate Flow Control. Flow control is used to control the traffic sources so that they do not send too much data into the network at any moment. If a trunk in the network is overloading, all the sources using that link must be told to slow down. This is absolutely necessary for a data network because the self-similar characteristics of data traffic are such that simply under-loading the network will not work. The critical difference between flow control techniques is the time it takes to tell the source about congestion and to get it under control. Today, LANís typically use TCP to control the data flow. TCP was created 15 years ago when networks were much slower. It operates a binary rate flow control between the two end-stations, slowing down the data flow if the return path indicates that data was lost. If data is not being lost, it speeds up the flow. Thus, it oscillates with a period of 1-2 seconds on the Internet, losing data each cycle and under-using the network on the other part of the cycle. Its characteristic time is about one second, that is the time it takes TCP to stop the data flow due to increased congestion.
TM 4.0 has now completely specified a new generation of flow control, explicit rate flow control, which operates from 100-1000 time faster than binary rate, in-band flow control protocols like TCP. Explicit rate flow control operates with RM cells being sent around the network to convey the flow control information back to the source. A higher priority path can be used for the RM cells so that they do not need to be delayed by the data. When a switch sees a RM cell it marks it with the highest data rate it can currently support for the associated source. This way, when congestion occurs, the switch can quickly mark RM cells with a new rate which will eliminate the congestion and these cells can move at the speed of light back to the sources telling them exactly how fast to send data. Thus, compared to todayís TCP the delay comparison is:
- Priority for RM cells over data. - Typically in Internet today a factor of 10
- Explicit rate rather than oscillation - Typically oscillation adds another factor of 5
- Switch to source distance vs round trip - Anywhere from a factor of 2 to 10
Thus, explicit rate can stop the source 100-500 times faster than TCP. This means that the buffer storage at the switch can be reduced by the same factor, 100-500. This is shown graphically in Figure 2.
Figure 2 .TCP Flow Control Cycle vs Explicit Rate Control Cycle
Even though there tends to be statistical smoothing of the data transients at a switch for both techniques, the reduction of delay is directly reflected in reduced buffer size. This is critical because high speed hardware switches like ATM switches and 100 Mbps Ethernet switches cannot afford the same memory as the old software switches and routers had. A 100 Mbps router often has about 1 second of buffering at full input rate. A 10 Gbps ATM switch typically has about 20 ms of storage at full input rate and could not afford to increase it to 1 second since for the very fast SRAM memory needed in hardware switches, the cost would be unreasonably high. Thus, with memory of 50 times less a flow control delay gain of at least 50 is needed to make high speed hardware switches work. Explicit rate flow control has even more delay gain, which may help us fix the current brownouts and overloads in the Internet if it can be incorporated into the Internet.
As with QOS, explicit rate flow control is only useful if the RM cells flow end-to-end and all the congestion points arrange to mark the RM cells with the correct rate. If a current day router or bridge is in the path, the RM cells will be delayed behind data and no marking will be done. This will not have a major effect if the device is always significantly under-loaded, but if it ever becomes overloaded then TCP will need to be operated on top of the explicit rate flow control. Further, both end-stations must participate in the explicit rate flow control in order for it to work. If one or both do not support explicit rate, TCP will be required and much of the benefit of explicit rate will be lost.
If a VC has established explicit rate flow control from one end of the network to the other, end station to end station, then TCP can be controlled so as to only send data when a new block of data is needed. Updating TCP at the same time as the ATM signaling and CIF driver are downloaded is straightforward. Thus, TCP flow control would be used if ABR flow control was not available end-to-end, and would not be used otherwise, so as not to interfere with the explicit rate operation.
The need for QOS based Routing
One of the major problems today in the Internet is caused by the lack of bandwidth sensitive routing. TCP/IP has no current capability to route packets any way but the shortest operating route. The result of this is major outages during peak hours when one trunk is under provisioned. As peak hour approaches the trunk gets overloaded and slow. The router feeding it loses many packets and TCP slows all the users down and down. The retransmission load increases and increases. Soon no-one gets anything through. They try again, but the same route is selected. The paths routed over that trunk are inaccessible. With the fast growth of the Internet, this type of problem occurs frequently.
This problem would not exist with ATM because of PNNI 1.0. As soon as the data gets to the CIF-Attachment Device, it enjoys the benefits of ATM routing. This is another reason why using ATM protocol through CIF is far superior to hoping that TCP/IP will someday fix all its problems.
The CIF design is to operate software in the end-station which:
- Accepts the QOS request from Winsock2 and uses ATM signaling to request that QOS.
- Accepts data form either an ATM null stack, IP, IPX, etc.
- Puts that data into multiples queues based on their QOS
- Assembles up to 31 cells of data into a packet along with the CIF header (See Figure 3)
- Sends the most time critical data first to the driver using weighted fair queuing
- Receives packets from the driver, removes the CIF header, and sends them on
- Controls TCPís flow control activity based on if end-to-end ABR flow control is active
Figure 3. CIF Header Format
Figure 4 shows the components of software in a Windows end-station. The ATM signaling stack, the CIF Shim, and the modified TCP need to be downline loaded to the end-station. The end-station driver and the NIC card are not changed. The CIF Attachment Device in this case is an ATM-Ethernet Edge Switch with CIF capability. It must have multiple queues in hardware as well as RM cell marking for ABR flow control. The packet format is a standard Ethernet packet with an eight byte CIF header and then up to 31 ATM size (48 byte) cell bodies. The CIF header contains the ATM cell header (5 bytes) as well as 3 bytes of CIF control information.
Figure 4. Components in the CIF-AD and the CIF-ES
CIF ABR Flow Control
CIF Attachment devices may act as VS/VDís, reducing the number of RM cells flowing at the end-station by changing Nrm from 32 to something like 128 as shown if Figure 5. This insures that the interrupt and processing rate for RM cells in the end-station is about 3% of the packet rate, just as RM cells are 3% of the cell rate. The end-station Shim performs the RM cell processing and rate determination as specified in TM 4.0 for ABR source/destinations in software. Thus, the end-station is a proper ATM source/destination according to TM 4.0, but is all done in software.
CIF protocol can be used on both sides of an switch, in which case there is a need for the CIF switch to mark RM cells based on itís congestion or available throughput.
With the reduced RM cell rate, the processing overhead for the end-station is only about 3% higher than in an end-station without CIF. However, depending on the design of the driver, there could be some extra overhead in keeping the queues short and going to the Shim for each new packet so as to optimize QOS.
The scheduler in the end-station Shim uses the data rates from the RM cells to set the flow rate for each queue in the same manner set forth in TM 4.0 for source/destination processes. It is also valuable to pass the VC rate on the application so that it can determine how much data to send to the user where guaranteed responce time is desired.
Figure 5. CIF treatment of RM cells for ABR Flow Control
For half duplex Ethernet, the majority of the installed base (130 million end-stations), the maximum delay variation incurred due the use of the Ethernet is 2.4 ms, 1.2 ms if one has to wait for a full packet going out and then, since the line is half duplex, another 1.2 ms if the other end has a full packet ready to send. For voice, the most time critical application currently being considered, an extra 2 ms is no problem even if it occurs at both ends since the delay tolerance for voice is about 20 ms when going to an analog phone with echo without echo cancelers, and 100 ms if both ends are full duplex, on the network. In neither case is a 5 ms delay variation for the full round trip a problem. In fact, operating system delay variation will far exceed the 2.4 ms from Ethernet for most operating systems. Therefore, the actual operating performance of CIF over Ethernet will be virtually identical to that experienced with a direct 25 Mbps ATM connection. The only difference will be the maximum throughput rate, 10 or 100 Mbps and 1 Gbps for Ethernet vs 25 , 155 Mbps and 622 Mbps for ATM NICís.
For Token Ring, it appears to be possible to achieve good QOS performance with a multiple (N) end-stations on a shared 16 Mbps ring and one CIF-AD. Token Ring has dual priority which can be utilized to reduce the delay variation for voice. The maximum delay variation would be (N+1)*.75 ms on the low priority but with packet size constraints on the high priority it would far less. Thus the cost per port for upgrading a Token Ring installation to CIF could be extremely low depending on the number of stations that prove to work effectively.
CIF provides a way to economically extend ATM protocol end-to-end, from end-station to end-station. There is a major cost saving because a new Network Interface Card is not required in the end-station nor does the end-stationís case need to be opened. CIF allows ATM to be extended over any frame based legacy protocol like Ethernet and Token Ring. Once a CIF capable switch is installed , the end-station only needs to download some software to obtain ATM quality QOS and explicit rate flow control. It should then be able to obtain quality voice and video over its current Ethernet or Token Ring NIC card.
By lowering the cost of connecting end-stations with ATM protocol to about the same cost as other non-ATM options like switched Ethernet, CIF makes it possible to make available to the mass market the full power of ATM; itís QOS signaling, QOS routing, and Explicit Rate flow control, none of which are useful unless end-to-end. As a result, the availability of CIF should greatly expand the acceptance and installation of ATM.
The CIF Alliance, in meetings with over 30 organizations working together, has completed the attached specification of CIF which defines the details for using ATM over Ethernet and Token Ring. The Alliance desires to work with the ATM Forum to insure that the CIF specification is fully inter-operable with the ATM Forumís specifications. The working groups of the Forum which would be most impacted are the TM, Signaling, and ILMI groups. Of course if other groups feel they need to review the CIF specification, that would be fine. The Alliance wishes to follow the example of the WINSOCK 2 coordination and will quickly respond to questions. If any incompatibilities are found between CIF and ATM the Alliance will strive to quickly modify itís specification to fix the incompatibility. When the Forum is satisfied that the CIF specification is inter-operable with the Forums specifications, then the Alliance would like to receive a notification of such.
Time is critical due to the high speed of the marketplace. It is critical for both CIF and ATM that this coordination be completed as rapidly as possible. Therefore, it is requested that the schedule for this coordination process be as fast as possible. As part of speeding the coordination process, please include the CIF Alliance on Email which it is appropriate that they be included by adding the following address on such Email : email@example.com .
Copyright © 2001 Dr. Lawrence G. Roberts