October 2023 - people - netdevconf.info

0x17: Talk, Scripting the Linux Routing Table with Lua
by Jamal Hadi Salim 06 Oct '23

06 Oct '23

You can run Lua scripts inside the kernel using the Lunatik project! Lunatik has been previously used for scripting Netfilter and XDP subsystems, allowing users to create complex network filters using Lua. For instance, it has been used for filtering L7 traffic such as HTTP and HTTPS inside the kernel. In this talk, Lourival Vieira Neto, Marcel Moura, Ana Lúcia de Moura and Roberto Ierusalimschy present the evolution of this framework and illustrate a sample use case implementing a kernel-level adaptive routing service. This service allows the system to monitor the reliability of a network route and adjust its routing table dynamically in case of failure. For instance, a home router might use such service to fall back to the cellular network using a mobile phone as a gateway. Lourival et al also spice this service up by implementing a protocol that allows nodes to advertise themselves as alternative gateways and for the router to notify nodes when elected as the new gateway. Lourival et al will describe the kernel details and associate infrastructure interaction ( netdevice notification chain, kthread, socket, RCU and FIB, etc). cheers, jamal

1 0

0x17: Talk, eBPF Qdisc: a generic building block for traffic control
by Jamal Hadi Salim 06 Oct '23

06 Oct '23

Adding a new packet scheduling algorithm in the kernel requires implementing a Qdisc kernel module - which could be cumbersome. Writing a custom scheduling algorithm which may not make sense for the rest of the community is frowned upon. On top of Qdiscs, different classifiers and rules can be applied to packets as well, which further complicates the combination of choices. Amery Hung and Cong Wang introduce eBPF Qdisc, a fully programmable Qdisc with eBPF. Amery and Cong believe that the approach they are taking will provide developers not only a more flexible and safer way to implement packet schedulers, but also orchestrating packets in coordination with other networking components. In this talk, they first show that eBPF Qdisc is capable of realizing a sophisticated Qdisc by implementing fair queueing (fq) Qdisc. Then, they evaluate the performance of the fq against the existing fq in the Linux kernel. Finally, they will show how eBPF Qdisc can help developers implement Qdisc's with diverse functionalities through two use cases: 1) we build a more robust earliest departure time (EDT) rate limiter by coordinating with a fq Qdisc through eBPF maps and 2) show eBPF Qdisc as a flexible way to create network emulators. cheers, jamal

1 0

0x17 Talk, Leveraging Homa: Enhancing Datacenter RPC Transport Protocols
by Jamal Hadi Salim 06 Oct '23

06 Oct '23

Hyper-scale data centers, such as Bytedance, have a strong need for optimal RPC performance. Homa is purpose-built for RPCs making it ideal for such environments. In this talk, Xiaochun Lu and Zijian Zhang do a comparative study of gRPC's performance on Homa versus over TCP. They observe clear performance advantages in favor of Homa, especially for RPC messages under 50KB, in terms of both latency and throughput. Like John's talk[1], they optimize longer RPC messages and improve bidirectional messaging. Xiaochun and Zijian will further provide performance insights, describe their mods to the Homa kernel code, and guide participants through the tangible benefits of integrating Homa into their RPC transport framework. [1]https://netdevconf.info/0x17/sessions/talk/kernel-managed-user-buffers-in-homa.html cheers, jamal

1 0

0x17: Talk, Kernel-Managed User Buffers in Homa
by Jamal Hadi Salim 06 Oct '23

06 Oct '23

Reminder: One more day to go for early bird registration.. Homa was first introduced to us in John Ousterhout's exciting 0x16 keynote[1]. John argued in the keynote that it was time to replace TCP with Homa to avoid what he labelled as the "datacentre tax". For the uninformed: Homa is message-oriented like UDP or SCTP as opposed to TCP being stream-oriented. There is no notion of connection in Homa, only request/response RPC messaging. A single Homa socket can send/receive RPCs to many peers. One advantage that TCP has by virtue of using the streaming approach is it can pipeline very large messages (think 500KB across MTU sized packets on the wire) - whereas a message oriented approach will have to copy bytes to/from user space and DMA-able packet buffers in parrallel. In his experiments with an mlx5 based 25G NIC, Dr. Ousterhout found that TCP infact outperformed Homa with large messages. At 500KB message size TCP was able to achieve 17-19 Gbps throughput, Homa was only able to hit 10-11Gbps. How dare TCP beat Homa in any metric! So John set out to fix this situation ;->. When he was done with the preliminary work Homa was able to hit the same rate as TCP (17-19Gbps). In this talk, he will go over the approach he took in the kernel code to allow the kernel to manage user mmaped buffers, changes made to both recvmsg() and sendmsg() to make this happen. 1. https://netdevconf.info/0x16/sessions/keynote/keynote-ousterhout.html cheers, jamal

1 0

0x17: Talk, Zero Copy Receive using io_uring
by Jamal Hadi Salim 05 Oct '23

05 Oct '23

The war on memory bottlenecks[1][2] continues with this talk from David Wei and Pavel Begunkov. Socket recv() data is first copied/DMAed into the kernel memory and then again into user space memory - adding pressure to overall memory bandwidth and of course comes with a CPU cost. While there are other approaches to avoid the second copy on recv() such as DPDK, RDMA etc, David and Pavel argue that all of them have downsides - ranging from being proprietary, custom patched drivers, and difficult to debug and worse requiring a rewrite of the applications... Our esteemed speakers instead opt for continuing to use the network stack as is and io_uring as the user facing API. So what do we need to get this working? As in [2] that the NIC supports header splitting and RSS flow steering. On incoming data the headers traverse the TCP stack while data is DMA'ed not to the kernel memory but into user space... David and Pavel will discuss the overall approach they took and describe in some detail the kernel infra as well as what uapi would look like. They will further review kernel-existing zero copy approaches and how they plan to coexist with them. Last but not least they will dig into limitations and challenges of zero copy receive and how they overcome them to facilitate a real deployment. [1] https://netdevconf.info/0x17/sessions/talk/congestion-control-architecture-… [2] https://netdevconf.info/0x17/sessions/talk/device-memory-tcp.html cheers, jamal Reminder: 2 more days to go for early bird registration

1 0

0x17: Talk, Device Memory TCP
by Jamal Hadi Salim 05 Oct '23

05 Oct '23

My apologies for putting some marketing at the top of this email: Reminder, Early bird ticketing is open for 2 more days; the sooner you register the better it is for our planning... And now to tell you about one of my favorite talks this time.. ML/AI continues to be an inspiration for new Linux kernel infrastructure and a theme at netdev conf 0x17! There are a couple of patterns in ML traffic that inspire this work from Mina Almasry, Willem de Bruijn, Eric Dumazet and Kaiyuan Zhang: 1) There is _a lot of freaking data_ being transferred across the network - and it continues to increase dramatically (ex: to accommodate such large data transfers NICs with 800 Gps ports are starting to appear) and 2) The vast majority of this data is typically from a device(eg storage) to another device (eg GPU). For context, assume a scenario where data transfers from SSDs on machineA cross the network to GPUs on machineB; and on machine B: In the rx direction incoming data into the NIC goes to host memory, then from host memory to the GPU. In the tx direction outgoing data goes from GPU to host then out to the NIC. This data movement approach, with multiple round trips across the memory and PCIe busses, is no longer tenable at such large amounts of data quantity. Current host hardware is just not equipped to deal with this. While such bulk transfers amount to low packets/sec rates, you are simply not going to be able to achieve wire speed because of the PCIe and memory bandwith abuse from the multiple round trips (see talk [1] which delves into host limitations). Almasry et al introduce Device Memory TCP to solve these challenges. They implement socket API to enable a user to send directly from device memory (GPU in machineB above example or SSD in machineA) across the network - as well as be able to place incoming data directly in the device memory (GPU in machineB example). How does this magic happen? Well, we need some support from the NIC to split the header on rx such that the TCP headers go to the host and the rest of the data goes directly to the device. The reverse is true for tx direction. Packet headers get to go to stack and get processed in the standard (TCP) code path. Memory bandwidth use is cut dramatically and because NIC to device transfers now happen at the lowest level of the PCIe tree hierarchy PCIe bandwidth use is also cut down dramatically. With devmem TCP, Almasry et al with a setup of 4 machines of GPU+100Gbps NICs - and data sent and received directly from/to device memory - were able to reach ~96.6% line rate speeds. In the talk they will go over details of the kernel/uapi changes to achieve devmem TCP. cheers, jamal [1] https://netdevconf.info/0x17/sessions/talk/congestion-control-architecture-…

1 0

0x17: Talk, Lightweight Implementation of Per-packet Service Protection
by Jamal Hadi Salim 04 Oct '23

04 Oct '23

There are applications which MUST have reliable communication otherwise things fall apart! Think industrial automation, robotics, aerospace, automotive... Failure in timeliness of message delivery could be catastrophic. Typically when such a network is built, the operator makes sure that any congestion-related issues and packet losses are avoided. However, Murphy's law applies: the wiring may break, a radio link can be temporarily shielded by an obstacle, or worse the network nodes can be damaged... To deal with such losses, industry standardization has been introduced to provide per-packet service protection - these include Frame Replication and Elimination for Reliability(FRER) for Layer 2 Ethernet networks and the Packet Replication, Elimination, and OrderingFunctions(PREOF) forLayer3 IP/MPLS networks. Ferenc Fejes et al have implemented FRER in two different approaches: 1) in eBPF/XDP as a lightweight software implementation; and (2) in userspace using PF_PACKET. Ferenc et al will go over the architecture details of the two approaches and then evaluate XDP FRER via an experimental analysis and further compare the two FRER implementations. cheers, jamal Reminder: Three days to go before early-bird registration ends!

1 0

0x17: Talk, Synce4l: open-source implementation of Synchronous Ethernet
by Jamal Hadi Salim 04 Oct '23

04 Oct '23

Almost _a quarter_ of the world-wide consumer Internet traffic is pushed through mobile networks! Wireless mobile network traffic requires tight time synchronization with wired components of the 5G/6G network. PTP with its sub-microsecond synchronization just doesnt cut it anymore and Michał Michalik and Arkadiusz Kubalewski say we need sub-nanosecond synchronization! The only way we can achieve this is to use Synchronous Ethernet (SyncE) which allows transport of clock signals through the Ethernet physical layer. This prevents the clock skew and clock jitter between adjacent network nodes. In this talk, Michał and Arkadiusz will motivate us with the need of frequency synchronization and its role when coupled with PTP in 5G/6G networks. They will discuss the open-source project synce4l [0] and demonstrate, focusing mainly on the features, requirements (both in software and hardware), and configuration in a real network. Finally, the talk will describe how synce4l corresponds with the currently discussed new DPLL (Digital Phase Locked Loop) subsystem [1] in the Linux kernel and user-space networking community. [0] https://github.com/intel/synce4l [1] https://lore.kernel.org/netdev/20230824213132.827338-1-vadim.fedorenko@linu… cheers, jamal Reminder: Three days to go before early-bird registration ends!

1 0

0x17: Talk, Unstoppable Session Layer
by Jamal Hadi Salim 04 Oct '23

04 Oct '23

Back in circa 0x15[0], Alex Aring asked if you have an ECONNRESET on your socket, and even recovered, should we consider the communication reliable? Back then Alex made the argument that when an application encounters connection errors, the act of merely reconnecting reduces the likelihood of the application being considered reliable. He suggested a session layer to help in improving the reliability. To build up on that experience, Alex is back with what he calls "Unstoppable Session Layer" (USL) to allow you to easily switch your networking application to provide a session layer between application and transport layer. This talk will show how to easily put such a session layer on your networking application. It will be demonstrated by a basic TCP ping-pong application while tcpkill [1] tries to run crazy and interfere with the TCP application by sending TCP resets. The eventual goal is to port this implementation into the kernel so that that applications can benefit from it. [0] https://netdevconf.info/0x15/session.html?ECONNRESET-on-a-reliable-socket-a… [1] https://en.wikipedia.org/wiki/Tcpkill cheers, jamal Reminder: Three days to go before early-bird registration ends!

1 0

0x17: talk, Using eBPF to inject IPv6 Extension Headers
by Jamal Hadi Salim 04 Oct '23

04 Oct '23

IPv6 Extension Headers (EH) allows the IPv6 to be more flexible and extensible. Justin Iurman et al say, unfortunately, the EH does not survive easily on the global internet and it is often tricky to analyze these losses without real deployment; and for this reason they built an eBPF instrumented tool based that operators can use to test the survivability of specific IPv6 EHs. The tool can inject any of the extension headers (Hop-by-Hop Options Header, Destination Options Header, Routing Header, the Fragment Header, etc). The talk will go into the details of both the kernel and user space architecture and code of the tool. Justin will further demonstrate its potential usage through a use case deployed by a network service provider. cheers, jamal Reminder: Three days to go before early-bird registration ends!

1 0