RENEW has three tiers of computing resources: infrastructure, resource manager and development support. It is demonstrated with the figure below and will be explained in more detail in the following sections. The future experiment schemes will be discussed following the discussion of the three tiers of computing resources.
¶ Infrastructure
The infrastructure includes all HW and SW resources necessary for network functions at all layers.
¶ Programmable Radio Access
RENEW’s radio access layer consists of many-antenna base stations and clients. Its building block is a programmable SDR module, called IrisⓇ from Skylark WirelessⓇ - a Rice University spin-off startup company.
¶ Iris SDR module
An Iris module is a network-based SDR device which is supported with the SoapySDR API. An Iris module is equipped with a dual channel transmit and receive, high rate streaming, device synchronization with timestamps, and burst controls. Streaming rates are limited to gigabit Ethernet when using the RJ45 connection, but can exceed gigabit rate with additional hardware provided by Skylark.
The Iris module was developed for ArgosNet (Section 1.5.4). It is the basic building block of both base stations and clients. The heart of an Iris module is a Xilinx Zynq 7z030 SoC, which has a dual-core ARM Cortex A9 and medium-sized FPGA fabric. It shall be called Zynq7 processor throughout this booklet.
The Iris module includes a transceiver IC - LMS7002M from Lime MicrosystemsⓇ, which is a 2x2 MIMO SDR transceiver IC tuneable from 100 kHz to 3.8 GHz. It can sample a maximum of 56 MHz of contiguous bandwidth on each radio interface with a 12-bit ADC. It shall be called Lime7 transceiver throughout this booklet. The Lime7 transceiver communicates with the Zynq7 processor through high-speed I/O lines from the FPGA fabric for data and through the SPI bus for control.
An Iris module is 1.55” wide with two TX/RX antenna ports at the front-end. The form factor allows the half-wavelength separation at 3.8 GHz when both dual-polarization antennas are used.
An Iris module has an interchangeable RF front-end to support high RF performance at various frequency bands. There are three types of RF front-ends commercially available.
- Barebone front-end: It has no filtering or amplification. It supports the full frequency range from 50 MHz to 3.8 GHz. It feeds the Lime7 transceiver output to antennas with 0 dBm transmit power.
- UHF front-end: It supports 470 - 800 MHz UHF with up to 30 dBm transmit power.
- ISM front-end (BRS/BCRS): It supports 2.48 - 2.7 GHz and 3.55 - 3.7 GHz with up to 28 dBm transmit power.
- ISM front-end (BRS/BCRS): It supports 2.48 - 2.7 GHz and 3.55 - 3.7 GHz with up to 28 dBm transmit power.
¶ Base Station
The RENEW base station is built based on the Argos base station design with important updates. The architecture view and block diagram of the RENEW base station are shown in the following two figures.
Architectural view of RENEW base station
Architectural view of RENEW base station
RENEW base station PHY block diagram
RENEW base station PHY block diagram
The RENEW base station consists of daisy chains of Iris modules. Two specialized bus connectors on each side of an Iris module are used to carry the reference clock and the power supply as well as 4 GPIO lines and 13.2 Gbps serial link routed to the FPGA fabric. This design eliminates the need for cabling between Iris modules while locking all daisy-chained Iris modules in frequency and enabling phase-coherent MIMO operation. The Iris daisy-chain design is depicted in the figure below.
The hub board utilizes the Xilinx ZCU102 development board and combines up to 8 daisy chains coherently through a customized backplane, which provides high precision clocking power and bi-directional connectivity. Since each daisy chain can connect up to 10 Iris modules, each hub board can support up to 160 radios. It has SyncE/PTP support (NGFI). The hub aggregates and distributes streams of samples from and to all radios. The hub design can scale with multiple hub modules.
¶ Clients
Iris-based clients are deployed throughout the infrastructure. The client block diagram is shown in the figure below.
Client PHY block diagram
A client is capable of running a full stack Linux to drive the lower network layers implemented in FPGA fabrics. The traffic generator is on top of the Linux stack and behaves like a mobile app, which sends and receives packets through the Iris radio front-end. A researcher can control the traffic generator in one of two ways.
- Offline download a database to the client for the generator to use.
- Use the Ethernet or WiFi connectivity to the client to drive its traffic generator in real time, e.g. routing the traffic from a smartphone to the client.
¶ Radio Control
The radio control includes packet framing, TX power control, and packet detection.
¶ Framing
Packets are divided into frames and subframes. Framing is an important step in building the physical layer as well as logic related to radio control functions such as AGC and the reciprocity calibration.
¶ TX Power Control
The TX power control is built on top of the sounder framework. The client measures the power on the received beacon.
BS selects the TX power in dBm, maps the power to settings, and uses a lookup table to map the desired TX gain to the appropriate gain settings.
¶ Packet Detection
Currently, the packet detector has a simple implementation in FPGA. It continuously monitors RSSI and indicates a detection after RSSI has exceeded some threshold for a given number of samples.
The configurable packet detection parameters are a function enable flag, an RSSI threshold, a sample size threshold, and a new frame flag.
The new frame flag is used by both the packet detection core and the AGC core. This flag triggers a state reset.
The output of the packet detection is a packet detected flag. The AGC state machine uses it to transition from idle to states.
¶ AGC
The AGC system response time is 8 us. This is needed to satisfy the strict WiFi timing, i.e. after the gain is set, AGC should take no longer than 10 short training symbols to settle. Each short training symbol is 0.8 us long. The AGC finite state machine is implemented on FPGA. It has four states: idle, saturation detection or coarse adjustment, wait, and measuring power or fine adjustment.
The configurable AGC parameters are a function enable flag, a reset flag, IQ saturation threshold (RSSI saturation level), saturation size threshold, maximum saturation size threshold, packet detection RSSI target, AGC settle count threshold, AGC coarse adjustment step in dB, AGC fine adjustment step in dB, and AGC initial gain.
¶ RSSI Measurements
RENEW implements its RSSI core on FPGA, which consists of a moving average with a variable size window.
¶ MAC Layer
¶ Beacons
Beacons are pseudo-noise sequences in frames that are broadcast periodically in the downlink for clients to discover the base station call, measure the signal strength, and decide to associate and synchronize their operations.
¶ Beamsweeping
Beacons are transmitted from a base station and are used to associate clients to a base station. Clients have no prior knowledge of beacons and base stations. Therefore, the base station needs to sweep beacons with a fixed codebook over a period of frames, which is usually equal to the number of massive MIMO antennas. Depending on the location and the channel condition, a client can hear a subset of the beacons with varying received power. The client can decide which cell it wants to associate to after measuring the beam with the maximum RSSI.
¶ System Broadcast Information
Apart from the synchronization beacons, the base station also broadcasts some cell related information. The clients need to decode this information before the association.
¶ Initial Association
The client, once decided on a cell, will start the association process by sending an RACH signal (Zadoff-Chu sequence) during a subframe specified by the base station. The base station correlates against the received RACH sequences and decides the uplink time advance of the client in question. The base station also feeds back this information to the client in an RACH response message.
¶ Network Infrastructure
RENEW/POWDER has a high-speed, ultra-low latency network connecting multiple massive MIMO base stations relying on redundant 40 Gbps circuits, which are deployed over existing dark fiber resources. The network provides the capabilities of DNS, DHCP, SMTP, authentication, and logging services. Portions of these resources are available for experiments. The network implemented on the campus of Rice University is shown below.
Currently deployed base stations on the campus of Rice University.
¶ Distributed Programmable Computing Resources
The RENEW-POWDER platform has programmable computing resources distributed across the infrastructure that are adequate for real-time implementation of physical layer innovations, especially the software realization of radio-access and network functions. In addition, we provision abundant and heterogeneous processing capability at a data center in the deployment site.
¶ Functions for Cellular Core
On top of the flexible network infrastructure and data centers described above, the RENEW-POWDER platform will support a key set of cellular network functions for traffic between clients and the Internet. Instead of using expensive, non-reconfigurable specialized equipment as in today’s LTE networks, the RENEW-POWDER platform will realize these functions in software, leveraging SDN and NFV for greater flexibility, efficiency, and scalability.
¶ Resource Manager
The resource manager is a hybrid edge computer with only sub-ms latency connected to each base station, which is composed of FPGA and GPU/CPU-based processors. It has modest local compute and storage capabilities and the ability to access large amounts of cloud computing capacity. It is similar to a distributed operating system. It provides a virtual testbed abstraction that guarantees availability of specified resources to users, creating an illusion to the users that they are the sole users of these resources. Under the hood, the virtual testbed relies on a hierarchical namespace and a virtual filesystem to address and virtualize infrastructure resources.
It combines distributed system technologies and the special properties of wireless infrastructure. Similar to a distributed operating system, RENEW resource manager presents each experiment a virtual testbed and coordinates multiple virtual testbeds. By leveraging the event-driven and data-driven nature of the wireless network infrastructure, RENEW resource manager further allows experimenters to record behavior traces at various points in the testbed and across the network stack and to replay them to understand their designs under investigation.
RENEW provides an active open-source software repository with fully operational network stacks aiming to stay a generation or more ahead of wireless standards as well as supporting the current standards, e.g. LTE and WiFi, for baselining. The initial software repository supports the recent wireless advancements, such as massive MIMO, full-duplex, coordinated multi-point, diverse-spectrum PHY and access mechanism, software-defined networking (SDN), and network function virtualization (NFV). The repository will enable the community to research large system-level innovations against existing concepts throughout the network stacks.
On top of the physical infrastructure, the RENEW-POWDER platform implements a resource manager using the virtual testbed to manage and monitor its resources, to provide arbitrated access to them, and to enforce isolation between different users. A virtual testbed consists of a set of resources defined and guaranteed by the resource manager. Each experiment runs inside its dedicated virtual testbed and is isolated from other concurrently running experiments. The virtual testbed abstraction leverages many resources' innate support for concurrent users and capacity provisioning, e.g. servers, fibers, and network switches. For these resources, the resource manager admits users based on their capacity to ensure that concurrent running experiments do not interfere with each other. On the other hand, resources like antennas, FPGAs, and the wireless channels are difficult to share by concurrent users. For these resources, the resource manager will allow time-division multiple access by default and cannot allocate them to two concurrently running virtual testbeds. The virtual testbed abstraction is built on top of a hierarchical name space and virtual filesystem.
¶ Development Support
The development support is a small data center at the deployment site with very low-latency communication across multiple base stations. It provides interfaces and libraries for users from diverse research communities to develop and debug experiments that may access infrastructure resources at different layers of the infrastructure. The development support includes an open-source software repository of experiment implementations and building blocks to seed the research and to educate use cases for the research communities. The collaborative development of the research repository will both see and accelerate the use of the RENEW testbed by the broader research community.
Given RENEW’s complexity, we provide a core set of frameworks to facilitate experiment development.
¶ PHY and MAC experiment development
RENEW supports both software-based and hardware-based, i.e. FPGA-based, development flows, which allows I/Q samples to be processed by various computing resources from the FPGA in an Iris module to the server co-located with the base station and to the data center.
For real-time experiments that require the computational power of the FPGA inside the Iris module, RENEW supports the Xilinx FPGA development flow and provides scripts to automate much of it. Because RENEW abstracts each FPGA as a virtual file, to configure the FPGA is as easy as writing into the corresponding file.
For software-based development flow, RENEW abstracts Iris modules with SoapySDR, which is an open-source SDR abstraction framework. SoapySDR integrates multiple available SDR drivers on the market, e.g. UHD by Ettus Research. As a result, experiments implemented for the popular GNURadio framework can be readily run on RENEW. Additionally, RENEW supports development flows using multiple programming languages, including MATLAB, C++ and Python.
¶ Network experiment development
By supporting OpenAirInterface and OpenEPC, RENEW immediately supports their associated development frameworks. The orchestration layer on POWDER fully manages all of the resources, including compute and radio nodes. The experiments will be fully describable and repeatable using JSON configuration files.
¶ Open-access research repository
RENEW provides an open-source software repository spanning all layers of network stacks. While many of the simpler blocks are available as part of open-source development frameworks such as GNU radio, there are many advanced wireless algorithms that are not yet available. The available repositories are described in below sections.
¶ Physical Layer Blocks
Physical layer blocks are implemented in both hardware (Verilog/VHDL) and software (C/C++ and Python). These include centralized and decentralized algorithms for downlink precoding/beamforming and uplink equalization and detection. More specifically, these include maximum-ratio combining, zero-forcing, and minimum mean-square error methods for massive MIMO. In addition, channel coding and decoding, including turbo, LDPC, and Polar code families, will be provided. We will also release blocks for modulation waveforms, such as OFDM and filter bank multi-carrier waveforms for 5G research.
¶ Standard and Experimental Stacks
Standard and experimental stacks include IEEE 802.11g/n/ac FPGA implementation, and LTE Release 10 based on OpenAirInterface software framework with experimental modules such as massive MIMO with many client support. For massive MIMO research, RENEW provides a custom experimental stack developed in both hardware and software with support for channel sounding, calibration, control channels, beamforming, as well as massive MIMO full-duplex using the softNull technique, which scales gracefully into massive MIMO systems. For cloud RAN and CoMP research, RENEW provides C- and FPGA-based implementations of multi-cell time and frequency synchronization, automatic gain controls, and joint scheduling and beamforming.
¶ Low-latency cellular core realization
To demonstrate the flexibility of RENEW’s network and cellular core support, RENEW provides a low-latency cellular core realization that uses a dedicated container to host necessary network functions for each client, in contrast to today’s LTE EPC architecture where clients share instances of network functions. By locating the client container and required network functions in the edge computer close to the client, this design can significantly reduce cellular core latency. While in today’s LTE EPC, the client traffic has to be routed through shared network function instances that are often many hops removed from the client.
¶ Supported Research Experiments
The RENEW-POWDER platform will support core wireless research communities while also creating an environment that welcomes and supports new emerging and synergistic areas.
¶ Wireless architectures
With each generation, it has become increasingly hard to improve the spectral efficiency of wireless networks. It is not surprising to see the pattern of innovations in the last decade that have been challenging traditional paradigms, e.g. massive MIMO, fully-duplex, Polar codes, and CoMP. Each of these ideas was so radical at their introduction that pure theory was not considered acceptable and experimental validation were considered crucial. Extrapolating forward, we expect that the next round of ideas will further challenge conventional wisdom, and hence will need large-scale testbed instantiations.
The RENEW-POWDER’s computing and fiber networking architecture will achieve near-best-possible latencies across the whole network with massive MIMO, full-duplex, and diverse spectrum support. Thereby, it allows exploration of designs that can explore tradeoffs, e.g. between throughput, latency, antennas per base station, bandwidth, network core design, and geographical coverage.
¶ Distributed and network systems
The RENEW-POWDER platform will open the network and computing infrastructures, which are traditionally under tight control by carriers, with an expected impact similar to what SDN technology has had on the Internet and data centers. Existing cellular networks consist of two distinct subsystems before interfacing with the Internet: the RAN and the EPC network. While RAN testbeds like WARP, NI USRP, BigStation, and ArgosNet have allowed cutting-edge research about the latest wireless hop, researchers do not have access to a combined open-access RAN and EPC network. The RENEW-POWDER platform will provide the first such testbed for the end-to-end programmable wireless network. It will leverage general-purpose systems based on an open-source software stack and off-the-shelf connectivity, i.e. Ethernet over fiber.
¶ Mobile And Ubiquitous Computing
The high throughput, low latency supported by the RENEW-POWDER platform and the distributed computing resources in the testbed provide an opportunity to investigate technical challenges facing novel applications that serve mobile users. For instance, AR/VR applications are well-known for the demanding requirements in computational power and latency. As a result, there is significant research on off-loading computation to edge computers or cloudlets that rely on high-speed, low-latency links. Several emerging applications of AR/VR such as education, training of medical and military personnel, and entertainment will only be unlocked when such links are wireless. Our platform will provide the infrastructure to investigate novel methods in multiple frontiers, from programming models to data privacy and system security.
¶ 2.4.4 Security and privacy
Security in current and emerging mobile/wireless applications, such as autonomous vehicles, IoT-enabled cyber-manufacturing, etc., will be critical to the operational safety. While security has been explored at each layer, there has not been a systematic investigation into multi-layer network security. RENEW will provide a testbed to experimentally evaluate cross-layer security architectures, e.g. what security primitives should be built into the lower network layers, and across networked systems, from mobile devices to base stations to gateways and eventually to cloud services, in order to enable strong security guarantees.
¶ 2.4.5 Machine learning and artificial intelligence
Machine learning is expected to adapt the system dynamically to its most efficient state, and to ensure and monitor its security and safety. Inferring and detecting anomalies over networks is an active area of research using FPGA and GPU computing systems. Yet existing machine learning technologies are incapable of dealing with the unprecedented volume and rate of data required to protect future wireless communications. RENEW will provide an ideal testbed with real-time high-volume and high-speed wireless data. It will be invaluable for deploying and testing next generation machine learning algorithms for ultra-high speed data mining research.
The RENEW system architecture is made up of
- Hardware (Section 3)
- Firmware (Soapy SDR and API, Section 4)
- Software (C/C++, Python, MATLAB, Section 5)
From the PHY layer, move up to the higher layers of the network. All details of massive MIMO Base Station and its interaction with the core network.
Massive MIMO makes use of many antennas at the base station to do beamforming or beam management to multiple clients at the same time. RENEW builds on the existing FAROS base station - a commercial product from Skylark Wireless.
¶ 2.5 Framework Support
SoapySDR/Pothos (All major SDRs)
GNU Radio (UHD Driver)
OpenAirInterface
POWDER-RENEW
¶ Iris and Faros Hub Services
Iris modules and Faros base stations are designed as network devices. They both run full TCP/IP stack on their ARM cores with a Ubuntu linux system. The system modules such as clocking and networking are configured and controlled through services.
Each Iris module has a uniquely assigned MAC address and is set up as a DHCP client. This means that each Iris module is required to be assigned with an IP from a DHCP server which runs on either a host server or a network router. However, it is recommended to connect them directly to a host machine for the highest performance, i.e. higher sample streaming rate and lower latency.
¶ Iris and Faros Hub Drivers
Control and communication with Iris modules and Faros is enabled with the SoapySDR ecosystem which is an open-source generalized API and a runtime library. SoapySDR can be used to instantiate, configure and stream with most of the off-the-shelf SDR modules in a variety of environments. Many open-source applications rely on SoapySDR for hardware integration support. Each SDR device requires a hardware-specific device driver. Faros hub and Iris device drivers are both SoapySDR supported devices, and are called SoapyIris and SoapyFaros, respectively. They natively implement a SoapySDR API.
For other device drivers not natively implemented with a SoapySDR API, a shim layer software is needed to translate between SoapySDR calls and the hardware-specific driver calls. For example, the shim layer software for UHD devices such as NI USPR devices is called SoapyUHD. In this way, any SDR application using SoapySDR can readily run on a variety of devices. A demo in this YouTube video nicely demonstrates this.
SoapySDR also supports running the API call on a remote device. All control procedures such as setting frequency, sample rate, etc. need to run on the Iris or Faros linux system. However, the call usually originates from the application running on the host machine. The remote call support is provided by the SoapyRemote library, which also implements other features such as the network device discovery. Using services like SSDP and MDNS, Iris modules and Faros hub can be discovered from a host machine on the same network. More details on device discovery and other SoapyRemote features are on the SoapyRemote Wiki page.
The baseband data communication between the Iris module and the host machine is handled by the Iris device driver, called SoapyIris. It sets up streaming between an Iris module and the host machine through UDP sockets. The FPGA on each Iris module encapsulates baseband samples in UDP packets and subsequently in an Ethernet frame and forwards them to the network. To enable high performance and predictable latency, the data flow between the originating FPGA and the outgoing physical Ethernet port is on the FPGA fabric only. The data flow can happen on the same iris module, on another Iris in the same chain, or on the Faros hub. In other words, data does not pass through any CPU element until it reaches the host machine. The data streaming occurs with IPv6 headers. More details are on the SoapyIris Wiki page.
The architecture of the Iris module, Faros and their relationship with various Soapy ecosystem components are shown as below.
