Vertical Perimeters: Virtualizable Architecture for High Performance Network Stack

The Networking stack in Solaris 10 uses a new architecture for doing network processing where the NIC is controlled by the network and the transport layers creating a per CPU vertical perimeter. The network stack is able to schedule the receive side packet processing by dynamically controlling the rate of packet arrival from individual receive rings on the NIC. The architecture minimizes context switches and allows processing of packets without losing CPU affinity or contending for any locks.

The network processing for a connection is performed from a vertical perimeter, which is implemented by a serialization queue and consists of one or more threads bound to a CPU for better locality. There is at most one queuing, and a packet once picked up for processing is processed all the way to the socket layer on the inbound case, and all the way to the NIC on the outbound case without needing to contend for additional locks or switch context.

The stack also provides the building blocks for network virtualization and resource control by creating virtual stacks around any service (HTTP, HTTPS, FTP, NFS, etc.), protocol (TCP, UDP, SCTP, etc.), or Virtual machines like Containers or Xen.

Each virtual stack can be assigned its own priority and bandwidth on a shared NIC without causing any performance degradation. The architecture dynamically manages priority and bandwidth resources, and can provide better defense against denial-of-service attacks directed at a particular service or virtual machine by isolating the impact just to that entity. The virtual stacks are separated by means of H/W classification engine such that traffic for one stack does not impact other virtual stacks.
Before joining Solaris Networking in 1997 Sunay did his graduate work at IIT Delhi and Stanford. At Sun Sunay demonstrated his technical leadership and innovative thinking by introducing the new FireEngine networking stack that became a key ingredient for the Solaris 10 success. Subsequently, Sunay extended the FireEngine architecture and introduced the Nemo GLDv3 device driver framework to simplify writing device drivers for Solaris. Nemo played a major role in driving Solaris 10 adoption by IHVs.

Most recently Sunay has been focused on CrossBow network virtualization to manage host networking resources and to meet the needs of Chip Multi-Threading and OS Virtualization. This sets Solaris apart from Linux.