RT-WiFi: Real-Time High Speed Communication Protocol

RT-WiFi: Real-Time High Speed Communication Protocol for Wireless Control Systems Yi-Hung Wei∗ 1 , Quan Leng∗ , Song Han∗ , Aloysius K. Mok∗ , Wenlong Zhang† , Masayoshi Tomizuka† ∗ Department

† Department

of Computer Science, The University of Texas at Austin {yhwei, qleng, shan, mok}@cs.utexas.edu

of Mechanical Engineering, University of California, Berkeley {wlzhang, tomizuka}@berkeley.edu

Abstract— Applying wireless technologies in control systems can significantly enhance the system mobility and reduce the deployment and maintenance cost. Existing wireless technologies, however either cannot provide real-time guarantee on packet delivery or are not fast enough to support high speed control systems, which typically require 1KHz or higher sampling rate. Nondeterministic packet transmission and insufficiently high sampling rate will severely hurt the control performance. To address this problem, in this paper, we present our design and implementation of a real-time high speed wireless communication protocol called RT-WiFi. RT-WiFi is a TDMA data link layer protocol based on 802.11 physical layer to provide deterministic timing guarantee on packet delivery and high sampling rate up to 6KHz. It incorporates configurable components for adjusting design trade-off including sampling rate, latency variance, reliability, and compatibility to existing Wi-Fi network, thus can serve as an ideal communication platform for supporting a wide range of high speed wireless control systems. We implemented RTWiFi on commercial off-the-shelf hardware and integrated it into a mobile gait rehabilitation system. Our extensive experiments demonstrate the effectiveness of RT-WiFi in providing deterministic packet delivery in both data link layer and application layer, which further eases the controller design and improve the control performance.

I. Introduction Wireless control systems (WCSs) have received significant attention these years because of their great advantages of enhanced mobility, easier deployment as well as reduced configuration and maintenance cost. WCSs have been widely applied in many application domains including industrial process control and automation [1], healthcare and biomedical systems [2],smart structures [3], and intelligent robotic systems [4], to name a few. A key step in building a wireless control system is to choose the most appropriate wireless communication protocol according to its desired control behavior. Control designers generally look at three most critical design factors when making the decision: sampling rate, reliability and real-time data delivery. These factors directly influence the quality of control, and the performance of the whole system. Many existing wireless technologies have been adopted in wireless control systems. However each of these protocols can only support a small number of specific control applications. Some of them are based on low data rate physical layers, like 802.15.4, and focus on the real-time data delivery 1 The

first two authors have equal contribution to this work.

Fig. 1: Vision of RT-WiFi protocol and reliability performance, thus are only suitable for lowspeed control applications; Others including Wi-Fi focus more on improving network throughput but pay less attention to the deterministic behavior of data delivery, so that cannot be adopted in control systems which have stringent timing requirements. The lack of a flexible and high speed real-time wireless communication platform makes it difficult to find an ideal wireless communication protocol for many wireless control applications and complexify the control designers’ job. Taking our recent work [4] of designing a semi-autonomous remote robotic system as an example, because no available wireless technology can support both high data rate sensing flow and real-time control flow at the same time, we have to use a combination of two wireless technologies, Wi-Fi and WirelessHART, which complicates the system design, prolongs the developing time and increases the maintenance cost. To address this problem, in this paper we introduce RTWiFi, a flexible real-time high speed communication protocol designed for supporting a wide range of wireless control systems. We keep the following goals in mind when we design the protocol: Real-time Data Delivery and High Sampling Rate: Control applications rely on bounded latency for estimating and controlling the state of target system. Our previous research [5] shows that a control system usually prefers a sampling

rate higher than 1kHz. Thus, RT-WiFi is based on 802.11 physical layer. To achieve deterministic timing behavior, we adopt TDMA mechanism in the RT-WiFi data link layer design for channel access and set stringent timing requirement on the transactions in each time slot. Flexible Data Link Layer Configuration: Different control applications may have different communication requirements on data delivery, it is hard to apply a rigid design to a wide range of applications. We envision to design RT-WiFi as a configurable platform, such that control designers have great flexibility to choose their own design parameters according to their target applications. These design tradeoff include sampling rate, energy efficiency, communication reliability, real-time data delivery, and co-existence to existing Wi-Fi networks. Transparent System Design: In order to make RT-WiFi easily available and simple to integrate with the existing applications, it should reuse current hardware and make minimum changes on softwares. Users should be able to use RT-WiFi on the commercial off-the-shelf (COTS) IEEE 802.11 network interface, and existing application should be able to run on top of RT-WiFi with no or only minimum changes. Additionally, the transparent system design allows us to use existing 802.11 security techniques and build mesh networks easily. Our contributions in this paper are three-fold: •

•

•

We designed the RT-WiFi protocol to provide a flexible real-time high speed wireless communication platform for a wide range of wireless control systems. According to their target applications, control designers can customize their design tradeoff among sampling rate, jitter level, reliability, and compatibility with regular Wi-Fi network. RT-WiFi supports up to 6kHz sampling rate in our selected COTS hardware, and to the best of our knowledge, this is among the highest sampling rates that can be supported by any existing wireless real-time communication protocols. We built a prototype for the RT-WiFi protocol on COTS hardware. We shared our first-hand experience in addressing the challenges we met during the RTWiFi implementation and discussed how to choose design parameters that helps users to customize RTWiFi depending on their target applications. We presented a comprehensive case study to utilize the RT-WiFi protocol to integrate heterogenous sensors and assistive robotic devices together to build a mobile gait rehabilitation system and evaluate the performance through extensive experiments.

The remainder of this paper is organized as follows: In Section II we give a review on existing wireless protocols applied in wireless control systems. We present the system design of RT-WiFi in Section III, and describe the implementation details in Section IV. We describe the use case of a mobile gait rehabilitation system in Section VI. We conducted extensive performance evaluation on RT-WiFi and summarize our experimental results in Section V. Section VII concludes the paper and discusses the future work.

II.

Related Works

There are some well-developed wireless protocols which can be used for wireless control. They include Bluetooth [6] and Zigbee [7] for home automation, WirelessHART [1] and ISA100.11a [8] for industrial process control, MBStar [9] for body area networks and so on. However these protocols are not perfect for high speed wireless control. Bluetooth [6] cannot guarantee real-time data delivery, except for voice packets. Zigbee [7] can provide real-time data delivery in beacon-enabled mode, but the data rate is limited to 250kbps. WirelessHART [1] is a Time division multiple access (TDMA)-based wireless protocol, which is specially designed for industrial process control. It can provide realtime communication, but the maximum sampling frequency is only 100Hz. ISA100.11a [8] is a another wireless protocol which is developped by International Society of Automation (ISA) to support industrial wireless control neeeds. Similar to the problem of WirelessHART, the data rate of ISA100.11a is low. MBStar is designed to provide higher sampling rate than WirelessHART for process control, but the highest rate it can reach is 400Hz, which is still far away from the preferred sampling rate for some applications. Zigbee, WirelessHART ,MBStar and [10], [11] cannot provide high enough sampling rate because their physical layer (IEEE 802.15.4 [12]) is designed for low rate wireless personal network. Comparatively Wi-Fi, based on IEEE 802.11 [13], is a wireless protocol designed for high speed wireless local area newtork and can have data rate up to 150Mbps (IEEE 802.11n). However, Wi-Fi does not provide any realtime guarantee on data delivery. Wi-Fi normally operates in distributed coordination function (DCF) mode, in which each station use distributed randomized algorithm to access the channel to avoid collision. Thus the delivery time of data is not deterministic. In Wi-Fi, there is another optional access method: Point coordination function (PCF) [13] In PCF, the access point (AP) operates as a point coordinator (PC) and determines which station has the right to transmit the data. Unlike DCF, PCF is a centralized channel access method. PC divides the interval between two beacon frames into two periods: contention free period and contention period. In contention free period, PC coordinates the data transmission and in contention period, stations use DCF to access the channel. Although PCF can be used to schedule packet transmission, it cannot rely on any timing information, and thus does not provide real-time data delivery guarantee. Also, once a station get the access to the channel, it may occupy the channel for a non-deterministic time. Thus, it is not sure when the subsequent packets will be sent. TDMA is a channel access method which relies on accurate timing information and may potentially provide real-time data delivery. Some TDMA protocol have been proposed [14], [15], [16], [17]based on IEEE 802.11. However, both of [14], [15] are focused on the time synchronization issue of multi-hop network. They are not targeted at providing flexible platform for control application, and they does not consider the coexistence with regular Wi-Fi network. On the other hand, the purpose of [16], [17] is to provide efficient long range wireless network in rural areas. It is because the original CSMA/CA

EĞƚǁŽƌŬ DĂŶĂŐĞƌ

ŽŶƚƌŽů ƉƉůŝĐĂƚŝŽŶ

ZdͲtŝ&ŝ ĐĐĞƐƐWŽŝŶƚ

ƉƉůŝĐĂƚŝŽŶ

ŽŶƚƌŽůƉƉůŝĐĂƚŝŽŶƐ

dƌĂŶƐƉŽƌƚ

hWͬdW

EĞƚǁŽƌŬ

/W ZdͲtŝ&ŝ

YƵĞƵĞƐ ZdͲtŝ&ŝ ^ƚĂƚŝŽŶϭ

ZdͲtŝ&ŝ ^ƚĂƚŝŽŶϮ

ZdͲtŝ&ŝ ^ƚĂƚŝŽŶϯ

dŝŵĞƌ

&ůĞdžŝďůĞŚĂŶŶĞů ĐĐĞƐƐŽŶƚƌŽůůĞƌ

DĞƐƐĂŐĞ ,ĂŶĚůŝŶŐ DŽĚƵůĞ

D ĐƚƵĂƚŽƌϭ

^ĞŶƐŽƌϭ

ĐƚƵĂƚŽƌϮ

^ĞŶƐŽƌϮ

ĐƚƵĂƚŽƌϯ

^ĞŶƐŽƌϯ

DĞƐƐĂŐĞ ,ĂŶĚůŝŶŐ DŽĚƵůĞ

Fig. 2: A example of RT-WiFi-based wireless control system

>ŝŶŬ^ĐŚĞĚƵůĚĞƌ ^ƵƉĞƌĨƌĂŵĞ dĂďůĞ

>ŝŶŬ dĂďůĞ

ĞǀŝĐĞ WƌŽĨŝůĞ

YƵĞƵĞ

mechanism in Wi-Fi network is inefficient when Wi-Fi is used for outdoor long-distance communication. The main objective of them is to increase the throughput of the network, but not to provide high sampling rate real-time data delivery. III.

RT-WiFi Protocol Design

In this section, we describe the design details of RT-WiFi protocol. RT-WiFi is a data link layer protocol based on 802.11 PHY and aims to provide real-time data delivery for a wide range of wireless control systems from low-speed industrial process control to high speed mechanical device control. The design goals of RT-WiFi are to provide both realtime high sampling rate and application-dependent flexible data link layer configuration. Real-time data delivery is crucial to control systems because control applications rely on precise timing for monitoring and controlling the state of a system. High sampling rate is required to achieve good quality of control and to support ever-increasing number of sensors and actuators in a control systems. For instance, a communication protocol is required to support an aggregate 2KHz sampling rate for a control system with one sensor and one actuator, if both of them demands 1KHz sampling rate. For supporting a wide range of control applications, the design of RT-WiFi shall consider the various requirements of different types of control applications. The design philosophy of RT-WiFi is to provide enough freedom for control designer to choose their preferred communication behavior, which best fits to their existing controller design. At the same time, the design of RT-WiFi minimizes the modification on the original Wi-Fi protocol, so that it can be transparent to both the upper layer software stack and underlying hardware, and provide the most compatibility and usability. Figure 2 shows the typical architecture of a control system that adopts RT-WiFi network. In this case, a system architect plans to deploy three sensors and three actuators in this control system. Depending on the physical proximity of each sensor/actuator, they are attached to different RT-WiFi stations. In a RT-WiFi network, a network manager and control programs are running on top of RT-WiFi access point (AP). The network manager is software in the application layer that is responsible for configuring the RT-WiFi network based on the communication requirements specified by a control designer. The network manager dynamically allocates the communication links and generates network configuration to configure the data link layer of RT-WiFi AP and RT-WiFi stations. After the configuration stage, RT-WiFi network periodically transmit the sender data to and receive control data from the control application. In the

WŚLJƐŝĐĂů

/ϴϬϮ͘ϭϭŽŵƉĂƚŝďůĞ,ĂƌĚǁĂƌĞ

Fig. 3: System architecture of RT-WiFi protocol following, we are focusing on the design and implementation of the RT-WiFi data link layer. Fig. 3 presents the overall architecture of RT-WiFi protocol. At the very bottom, RT-WiFi adopts the physical layer of IEEE 802.11 [13]. IEEE 802.11 is one of the most pervasively adopted wireless technologies and supports up to 150Mbps bandwidth which is sufficient for most wireless control systems. Control application users can easily deploy the RTWiFi data link layer on a commercial off-the-shelf IEEE 802.11 compatible hardware for supporting high speed and real-time data delivery. Sitting on top of the 802.11 PHY is the RT-WiFi data link layer which will be elaborated in the remaining of this section. The RT-WiFi data link layer provides a nice abstraction for the upper layers, thus standard UDP or TCP-based applications will be supported seamlessly. Various control protocols can be well supported by wrapping the their control payload in TCP/UDP packets. The core of the RT-WiFi protocol is the TDMA-based data link layer. Since there is no centralized channel access controller of a regular Wi-Fi network, it adopts carrier sense multiple access with collision avoidance (CSMA/CA) to avoid collision. This mechanism helps improve the network throughput but not feasible for supporting high speed real-time traffic with stringent timing requirements. As Fig. 4a shows, the transmission of the regular Wi-Fi packet with a hard deadline will be blocked by a nondeterministic of time because of carrier sense, and further delayed by random backoff mechanism. To address this problem, we adopts TDMA mechanism and a centralized channel and time management to access the channels according to strict timing. Fig. 4b presents the channel access pattern of a RT-WiFi network. Since only one node can access a certain channel in a given time slot, RT-WiFi provides collision free and deterministic communications. As shown in Fig. 3, there are three key components in the RT-WiFi data link layer: a timer for maintaining global synchronization among all RT-WiFi nodes and triggering timing events; a link scheduler for coordinating the access to shared media, and executing the dedicated event at scheduled time points; and a flexible channel access controller which dynamically configures the hardware parameters for executing the timing event according to the target application behavior. We shall explain the design of each of these components in

ƵƐLJ DĞĚŝƵŵ

/&^

ĞĨĞƌĐĐĞƐƐ

͙͘͘͘ ĂĐŬŽĨĨƐůŽƚƐ

Đϭ

/&^

ĂƚĂ

ĞĨĞƌĐĐĞƐƐ

ƚϭ

ĐϮ

^ůŽƚϬ

ĂƚĂ

W ƌŽĂĚĐĂƐƚ

ĂĐŬŽĨĨ ƐůŽƚƐ

ƚϮ Đϭ͕ĐϮ͗ĚĂƚĂƌĞĂĚLJƉŽŝŶƚ ƚϭ͕ƚϮ͗ĚĂƚĂƚƌĂŶƐŵŝƚƉŽŝŶƚ

(a) Regular Wi-Fi dŝŵĞ^ůŽƚ

ĂƚĂ

dŝŵĞ^ůŽƚ

ĂƚĂ

^ŚĂƌĞĚ

^ůŽƚϮ

^ůŽƚϯ

^ůŽƚϰ

^ůŽƚϱ

^ůŽƚϲ

^ůŽƚϳ

^dϭ

W

^dϮ

W

^dϯ

W

W

^dϭ

W

^dϮ

W

^dϯ

Fig. 5: An example of a superframe with 8 time slots. There are three key data structures maintained in the link scheduler:

dŝŵĞ^ůŽƚ

ĂƚĂ

(b) RT-WiFi

Fig. 4: Comparison of media access methods between regular Wi-Fi and RT-WiFi the following sections. A. Timer Design To achieve high sampling rate, RT-WiFi has very stringent timing requirements on each device. For instance, if the required aggregate sampling rate is 5kHz, the timer is required to deliver an accurate timer interrupt for every 200µs, and all tasks related to that sample (including data transmission, acknowledgement and possible in-slot retransmission) shall be completed within that short interval (called time slot). A time slot is a rigid channel access unit in the RT-WiFi network. In order to provide better timing predictability for the target applications, each RT-WiFi node can only access the channel in their pre-assigned time slots. Depending on the current sampling rate, the size of a time slot is required to be larger than the inverse of the sampling rate. Fig. 6a shows a generic RT-WiFi slot timing, and the slot size is determined by Equation 1. The guard interval is used to ensure the consecutive transmission does not interfere with each other due to synchronization inaccuracy. Data transmission is followed by the guard interval, and the transmission time depends on the transmission rate and packet size. When a station successfully receives the data, it should response an acknowledgement (ACK) message after short inter-frame space (SIFS). If sender does not receive ACK after the ACK wait period, then it assumes the packet is lost. T T imeS lot ≥ TGuardInterval + T Data + T S IFS + T ACK

^ůŽƚϭ

(1)

Our timer design is based on the Timing Synchronization Function (TSF) in IEEE 802.11. TSF is achieved by having all nodes keep a local 1MHz TSF timer, and synchronize to the master clock in RT-WiFi Access Point periodically through the beacon frames. The timer then triggers accurate time interrupts to ensure the correct operation of the data link layer. We shall present the implementation details of our timer module on WiFi chips in Section IV-B. B. Link Scheduler If accurate time synchronization with the master clock in the Access Point is achieved, every station in the RT-WiFi network forms a consensus to a global time. Based on this globally synchronized timing information, a link scheduler is designed to coordinate the channel access among the stations.

Link: A link describes the communication behavior within a time slot. It is specified by its source, destination and the associated link type. We define four link types in RT-WiFi: transmit, receive, broadcast and shared. A broadcast link is used to transmit a data frame to all the neighbor nodes in the network; a transmit link is used for dedicated data frame transmission to the given destination; a receive link is used for receiving a data frame from the given source; and a shared link is for multiple source nodes to compete for transmitting a data frame to the same destination. Notice that, RT-WiFi is designed to be compatible to existing Wi-Fi systems. A RT-WiFi node can support both realtime and non-real-time traffic simultaneously. Link scheduler shall differentiate the priority of different traffics and assign them to either dedicated or shared links. Superframe: Superframe is a sequence of consecutive time slots. It defines the device’s communication pattern with neighbors and the superframe will repeat itself infinitely. Fig. 5 illustrates an example of a superframe configured in a RTWiFi AP with 8 time slots (the time slot length is 250µs). In this RT-WiFi network 3 stations are present, and each station is configured with a pair of dedicated transmit and receive link to the AP in the superframe. Device Profile: RT-WiFi Access Point keeps the device profile for the centralized network manager to generate superframe and links for each RT-WiFi station. The device profile is gathered when a RT-WiFi station joins a RT-WiFi network, and the device information is forwarded to network manger via RT-WiFi AP. The network manger installed in the RT-WiFi AP performs access control, generates the schedule information by looking up the sampling rate requirement of the new joined device that is specified by control designer, and reply the superframe and link configuration for that station. The network manager design and schedule construction is out of the scope of this paper and will be briefly discussed in the future work. To execute the TDMA schedule specified by the superframe, link scheduler is invoked by every timer interrupt. The link type of current time slot is determined by looking up the superframe using the timing information. Depending on the link type, link scheduler either requests a frame for the destination node from the message handling module or redirect the receiving frame to the upper layer. Before put the transmit frame into the physical layer, link scheduler setup the transmission parameters by using the flexible using channel access controller, which shall be described in the following section. C. Flexible Data Link Layer Design To support a wide range of control applications, we apply a flexible design in RT-WiFi data link layer. RT-WiFi allows

the available computation resources and the specific control applications.

dŝŵĞ^ůŽƚ

ĂƚĂ 'ƵĂƌĚ /ŶƚĞƌǀĂů

^/&^

<

In-slot retry: The in-slot retransmission mechanism is shown in Fig. 6b. the retransmission is invoked immediately if the sender does not receive an ACK message from the receiver. The retransmission time of a in-slot retry should not exceed the length of a time slot.