- people - netdevconf.info

0x17: Talk, Netlink APIs to Expose/Configure Netdev Objects
by Jamal Hadi Salim 08 Oct '23

08 Oct '23

Work to reduce the amount of developer involvement in netlink coding has been widely successful[1]. To name a few existing consumers of this new approach: devlink, ethtool, DPLL, FOU, XDP, TLS handshake[2], etc. You essentially define your netlink messaging and attributes in YAML and a compiler outputs not only all the commodity code (parsing, validating, etc) that you would have had to write (and make mistakes in) but also documentation(as you all know: developers like to write documentation. Not)! In this talk, Amritha Nambiar and Sridhar Samudrala extend the netlink definition of netdev-genl to add support for additional netdev objects: queue and NAPI. Queues are exposed as netlink objects, enabling visibility to per queue attributes (example, NAPI instance servicing the queue, stats, memory model etc.). NAPI instances are used to expose attributes such as the NAPI id, NAPI device, interrupt number associated with the NAPI instance and PID of the NAPI thread. Amritha and Sridhar say their work opens a lot of opportunities for interesting use cases. Example, the new netlink communication could be used to create queues etc. for multiple usecases (RSS context handling, Rx zerocopy, XDP); adjusting the NAPI thread priorities and SMP affinity. Additionally, with this features in place, one could also configure NAPI pollers to queues from the userspace. This could be used to limit the number of NAPI instances by configuring a poller to be shared among queues that are on an interrupt vector, etc. In this talk, they will discuss the design and how it interacts with the driver subsystem; they will further discuss challenges faced and their solutions as well as future work. Come and engage them in the discussion! [1]https://docs.kernel.org/userspace-api/netlink/specs.html [2]https://netdevconf.info/0x17/sessions/bof/tls-handshake-for-in-kernel-consumers.html cheers, jamal

1 0

0x17: Talk, kernel offload with complete host kernel functionalities
by Jamal Hadi Salim 08 Oct '23

08 Oct '23

Offloading relieves the host resources to be used for application work. Traditional offloading has been focussed mostly with either plumbing features (TSO, checksum, etc) or limited L4 (TC offloads, etc). Offloading of TCP machinery -famously called TOE- for example has been frowned upon for many reasons[1]. Ryo Nakamura and Hajime Tazaki feel they could address these outstanding issues transparently. The application on the host has no idea that the rest of the stack is outsourced somewhere else... How, you ask? On the host side they intercept system calls made by the application and transparently route them to a syscall proxy sitting on an xPU. The syscall proxy, which is a lightweight user space program, interfaces on the northbound to the host and on its southbound to Linux Kernel Library(LKL)[2]. LKL is optimized to improve packet processing code paths. Our esteemed speakers have implemented this feature using the Nvidia Bluefield DPU. In this talk they will present the design of the offload and discuss results of various workload performance tests: 1) a typical TCP transmission flow with existing offloading feature (TSO/LRO, checksum), 2) avoiding data copying between host and NIC with sendfile syscalls, and 3) kTLS offload with sendfile syscall (but not optimized as no hardware crypto offloading) over existing implementations (nginx and openssl). Last, but not least, they will briefly discuss the opportunities for the future improvements, including further offload (crypto, larger segmentation size for TCP, our RDMA channel implementation, and a choice of passthrough method, etc). [1]https://wiki.linuxfoundation.org/networking/toe [2]https://lkl.github.io/ cheers, jamal

1 0

0x17: Talk, Nanotube: Turning XDP Code into Custom Hardware Pipelines for FPGA SmartNICs
by Jamal Hadi Salim 08 Oct '23

08 Oct '23

Many modern NICs have built-in packet processing capabilities that can effectively perform the type of use case that XDP is often used for. Stephan Diestelhorst, Neil Turton, Kieran Mansley, Kimon Karras, Rip Sohan , Thomas Calvert and Steven Pope say that being able to seamlessly offload this XDP code to the NIC – under the control of the Linux networking subsystem and tooling – would increase server efficiency and performance without affecting the user experience. Stephan Diestelhorst et al will discuss Nanotube: an open-source compiler that consumes standard XDP code and produces a tailor-made, application-specific dataflow pipeline suitable for execution on FPGA NIC platforms. They have demonstrated generated Katran on AMD/Xilinx FPGA-based NICs to offload with throughput at line rate (100 Gbps) on AMD NICs! In this talk they will describe (i) the structure of the compiler; (ii) performance of typical workloads; and (iii) how map accesses by host software are supported using specially annotated packets over the normal host data path. And last but not least they will discuss requirements to allow better integration and support by the Linux kernel, in particular the validator, map handling, and vendor-neutral support requirements. cheers, jamal

1 0

0x17: Talk, P4 Compatible Control Plane Driver for Linux Kernel
by Jamal Hadi Salim 07 Oct '23

07 Oct '23

Our friends at Intel and AMD will share their thoughts on how to integrate P4TC[1] into their drivers. In this talk Anjali Singhai, Neha Singh, James Choi, Namrata Limaye, Sridhar Samudrala, Mahesh Shirshyad and Vipin Jain will discuss enhancements to existing TC offload interfaces to accommodate multiple P4 pipeline and runtime control. They will elaborate on the association of the s/w and h/w datapaths, hardware P4 dataplane loading and initialization as well as runtime control (using the classical skip_sw/hw flagging). These details will include describing the kernel infrastructures and any extension to current TC offload. Anjali et all will discuss potential enable shimming of legacy kernel offloads such as tc-flower on to P4TC interfaces. Additionally, they will talk about interfaces to support legacy hooks (switchdev ops, FDB ops) and coexistence of these with P4TC compatible driver. They will make the infrastructure code at the conference. cheers, jamal [1] https://netdevconf.info/0x17/sessions/talk/integrating-ebpf-into-the-p4tc-d…

1 0

0x17: Talk, SO_TIMESTAMPING: powering fleetwide RPC monitoring
by Jamal Hadi Salim 07 Oct '23

07 Oct '23

Willem de Bruijn says timestamping, via SO_TIMESTAMPING, is key to debugging network stack latency. Instead of gut-feel finger pointing between network and kernel tribes we can just get down to the facts of where the latency really is. SO_TIMESTAMPING can isolate transmission, reception and even scheduling sources. Capturing connection state along with timestamps further enables root cause discovery, such as TCP receive window size. Capturing timestamps at more points, such as traffic shaping and NIC hardware, expands visibility to tough issues like incast. SO_TIMESTAMPING has seen iterative development to enable fleetwide RPC monitoring at Google. They presented details of this infra called "Fathom" at SIGCOMM 2023[1]. In this talk Willem will start us beyond where that SIGCOMM paper ends. He will take a deep dive on the Linux kernel infrastructure that makes fleetwide continuous latency analysis and attribution possible. API extensions include covering TCP bytestreams, capturing transport protocol state along with events (OPT_STAT), and supporting selective sampling (OPT_CMSG). The talk reviews the core SO_TIMESTAMPING API, discusses non-obvious extensions (MSG_EOR, SO_RCVLOWAT), summarizes gotchas from the field (OPT_ID_TCP), and explains how all this combines to enable robust continuous RPC monitoring. It touches on clock synchronization and precision. Finally, it compares this UAPI to dynamic tracing with uprobes, kprobes, tracepoints and BPF. Should be fun, cant wait to attend this talk! [1] search for "fathom" at https://conferences.sigcomm.org/sigcomm/2023/program.html cheers, jamal

1 0

Assessing the impact of Linux networking on CPU consumption
by Jamal Hadi Salim 07 Oct '23

07 Oct '23

You may be a guru at reading perf output and then with a handwave, swish, you can whip your bash script into a frenzy to create flame graphs... but you would still need to dig deeper and isolate where your bottlenecks in the network are. Davide Miola, Fulvio Risso and Federico Parola took a different approach by creating a tool using eBPF to instrument strategic functions in the network RX and TX entry-points. They crafted a novel tool that they call "Netto"[1] which gives you a focussed view of only the network infra's contribution. Netto can give you answers to coarse grained details of "how much is the rx softirq chewing vs RX syscall" all the way to more fine-grained rx softirq details like the driver cost, GRO, NF, etc. And then do all this in real time with low cpu overhead... In this talk, Davide et al will go into details of the design and functionality of Netto as well as describe the different kernel hook points. They will then illustrate how one would be able to use this tool to get insights into the network infra's overheads for a specific workload. [1]https://github.com/miolad/netto cheers, jamal

1 0

0x17: Talk, Scripting the Linux Routing Table with Lua
by Jamal Hadi Salim 07 Oct '23

07 Oct '23

You can run Lua scripts inside the kernel using the Lunatik project! Lunatik has been previously used for scripting Netfilter and XDP subsystems, allowing users to create complex network filters using Lua. For instance, it has been used for filtering L7 traffic such as HTTP and HTTPS inside the kernel. In this talk, Lourival Vieira Neto, Marcel Moura, Ana Lúcia de Moura and Roberto Ierusalimschy present the evolution of this framework and illustrate a sample use case implementing a kernel-level adaptive routing service. This service allows the system to monitor the reliability of a network route and adjust its routing table dynamically in case of failure. For instance, a home router might use such service to fall back to the cellular network using a mobile phone as a gateway. Lourival et al also spice this service up by implementing a protocol that allows nodes to advertise themselves as alternative gateways and for the router to notify nodes when elected as the new gateway. Lourival et al will describe the kernel details and associate infrastructure interaction ( netdevice notification chain, kthread, socket, RCU and FIB, etc). cheers, jamal

1 0

0x17: Talk, eBPF Qdisc: a generic building block for traffic control
by Jamal Hadi Salim 07 Oct '23

07 Oct '23

Adding a new packet scheduling algorithm in the kernel requires implementing a Qdisc kernel module - which could be cumbersome. Writing a custom scheduling algorithm which may not make sense for the rest of the community is frowned upon. On top of Qdiscs, different classifiers and rules can be applied to packets as well, which further complicates the combination of choices. Amery Hung and Cong Wang introduce eBPF Qdisc, a fully programmable Qdisc with eBPF. Amery and Cong believe that the approach they are taking will provide developers not only a more flexible and safer way to implement packet schedulers, but also orchestrating packets in coordination with other networking components. In this talk, they first show that eBPF Qdisc is capable of realizing a sophisticated Qdisc by implementing fair queueing (fq) Qdisc. Then, they evaluate the performance of the fq against the existing fq in the Linux kernel. Finally, they will show how eBPF Qdisc can help developers implement Qdisc's with diverse functionalities through two use cases: 1) we build a more robust earliest departure time (EDT) rate limiter by coordinating with a fq Qdisc through eBPF maps and 2) show eBPF Qdisc as a flexible way to create network emulators. cheers, jamal

1 0

0x17 Talk, Leveraging Homa: Enhancing Datacenter RPC Transport Protocols
by Jamal Hadi Salim 06 Oct '23

06 Oct '23

Hyper-scale data centers, such as Bytedance, have a strong need for optimal RPC performance. Homa is purpose-built for RPCs making it ideal for such environments. In this talk, Xiaochun Lu and Zijian Zhang do a comparative study of gRPC's performance on Homa versus over TCP. They observe clear performance advantages in favor of Homa, especially for RPC messages under 50KB, in terms of both latency and throughput. Like John's talk[1], they optimize longer RPC messages and improve bidirectional messaging. Xiaochun and Zijian will further provide performance insights, describe their mods to the Homa kernel code, and guide participants through the tangible benefits of integrating Homa into their RPC transport framework. [1]https://netdevconf.info/0x17/sessions/talk/kernel-managed-user-buffers-in-homa.html cheers, jamal

1 0

0x17: Talk, Kernel-Managed User Buffers in Homa
by Jamal Hadi Salim 06 Oct '23

06 Oct '23

Reminder: One more day to go for early bird registration.. Homa was first introduced to us in John Ousterhout's exciting 0x16 keynote[1]. John argued in the keynote that it was time to replace TCP with Homa to avoid what he labelled as the "datacentre tax". For the uninformed: Homa is message-oriented like UDP or SCTP as opposed to TCP being stream-oriented. There is no notion of connection in Homa, only request/response RPC messaging. A single Homa socket can send/receive RPCs to many peers. One advantage that TCP has by virtue of using the streaming approach is it can pipeline very large messages (think 500KB across MTU sized packets on the wire) - whereas a message oriented approach will have to copy bytes to/from user space and DMA-able packet buffers in parrallel. In his experiments with an mlx5 based 25G NIC, Dr. Ousterhout found that TCP infact outperformed Homa with large messages. At 500KB message size TCP was able to achieve 17-19 Gbps throughput, Homa was only able to hit 10-11Gbps. How dare TCP beat Homa in any metric! So John set out to fix this situation ;->. When he was done with the preliminary work Homa was able to hit the same rate as TCP (17-19Gbps). In this talk, he will go over the approach he took in the kernel code to allow the kernel to manage user mmaped buffers, changes made to both recvmsg() and sendmsg() to make this happen. 1. https://netdevconf.info/0x16/sessions/keynote/keynote-ousterhout.html cheers, jamal

1 0

2024

2023

2022

2021

2020

2019

2018

people