NIX: A Case for a Manycore System for Cloud Computing

Embed Size (px)

Text of NIX: A Case for a Manycore System for Cloud Computing

  • NIX: A Case for a Manycore System for CloudComputingFrancisco J. Ballesteros, Noah Evans, Charles Forsyth, Gorka Guardiola, Jim McKie, Ron Minnich, and Enrique Soriano-Salvador

    Virtualized cloud computing provides lower costs, easier systemmanagement and lower energy consumption. However high performancetasks are ill-suited for traditional clouds since commodity operating systemsand hypervisors add cascading jitter to computation, slowing applications toa crawl. This paper presents an overview of NIX, an operating system formanycore central processing units (CPUs) which addresses the challenges ofrunning high performance applications on commodity cloud systems. NIXfeatures a heterogeneous CPU model and uses a shared address space. NIXhas been influenced by our work on Blue Gene and more traditional clusters.NIX partitions cores by function: timesharing cores (TCs); application cores(ACs); and kernel cores (KCs). One or more TCs run traditional applications.KCs are optional, running kernel functions on demand. ACs are also optional,devoted to running an application with no interrupts; not even clockinterrupts. Unlike traditional kernels, functions are not static: the number ofTCs, KCs, and ACs can change as needed. Also unlike traditional systems,applications can communicate by sending messages to the TC kernel, insteadof system call traps. These messages are active taking advantage of theshared-memory nature of manycore CPUs to pass pointers to data and codeto coordinate cores. 2012 Alcatel-Lucent.

    and persistence; for many tasks, the flexibility of this

    approach, coupled with savings in cost, management,

    and energy, are compelling.

    However, certain types of taskssuch as compute-

    intensive parallel computationsare not trivial to

    implement as cloud applications. For instance, high

    performance computing (HPC) tasks consist of highly

    IntroductionCloud computing uses virtualization on shared

    hardware to provide each user with the appearance of

    having a private system of their choosing running on

    dedicated hardware. Virtualization enables the dynamic

    provisioning of resources according to demand, and

    the possibilities of entire system backup, migration,

    Bell Labs Technical Journal 17(2), 4154 (2012) 2012 Alcatel-Lucent. Published by Wiley Periodicals, Inc. Published online in Wiley Online Library ( DOI: 10.1002/bltj.21543

  • 42 Bell Labs Technical Journal DOI: 10.1002/bltj

    interrelated subproblems where synchronization and

    work allocation occur at fixed intervals. Interrupts

    from the hypervisor and the operating system add

    noiserandom latencyto tasks, which then

    impacts all other tasks in the queue by making them

    wait for the tasks that have been slow to finish. This

    slowing effect may cascade and seriously disrupt the

    regular flow of a computation.

    HPC tasks are performed on customized hardware

    that provides bounded latency. Unlike clouds, HPC

    systems are typically not timeshared. Instead of the

    illusion of exclusivity, individual users are given fully

    private allocations for their tasks, with the allocation

    running as a single-user system. However, program-

    ming for single user systems is difficult; users prefer

    programming environments that mimic the time-

    sharing environment of their desktop. This desire

    leads to conflicting constraints between the need for

    a convenient and productive programming environ-

    ment and the goal of maximum performance. This

    conflict has led to an evolution of HPC operating

    systems towards providing the full capabilities of a

    commodity timesharing operating system. For example,

    the IBM Compute Node Kernel, a non-Linux* HPC

    kernel, has changed in the last ten years to become

    more and more like Linux. On the other hand, Linux

    systems for HPC are increasingly pared down to a

    minimal subset of capabilities in order to avoid time-

    sharing degradation at the cost of compatibility with

    the current Linux source tree. This convergent evo-

    lution has led to an environment where HPC kernels

    sacrifice performance for compatibility with com-

    modity systems, while commodity systems sacrifice

    compatibility for performance, leaving both issues

    fundamentally unresolved.

    We describe a solution to this tradeoff, bridging

    the gap between performance and expressivity, pro-

    viding bounded latency and maximum computing

    power on one hand and the rich programming envi-

    ronment of a commodity operating system (OS) on

    the other. Based on the reality of emerging manycore

    processors, we provide an environment in which

    users can be allocated at run-time on dedicated, non-

    preemptable cores, while at the same time, all the

    services of a full-fledged operating system are avail-

    able. This allows us to leverage bounded latency and

    exclusive uselessons learned from HPCand apply

    them to cloud computing.

    As an example of this approach we present NIX,

    a prototype operating system for future manycore

    central processing units (CPUs). This paper is an

    overview of the system and the approach.

    Influenced by our work in HPC, both on Blue

    Gene* and more traditional clusters, NIX features a

    heterogeneous CPU model and a change from the tra-

    ditional UNIX* memory model of separate virtual

    address spaces. NIX partitions cores by function: time-

    sharing cores (TCs); application cores (ACs); and kernel

    cores (KCs). There is always at least one TC, and it

    runs applications in the traditional model. KCs are

    cores created to run kernel functions on demand. ACs

    are entirely devoted to running an application, with no

    interrupts; not even clock interrupts. Unlike traditional

    HPC lightweight kernels, the number of TCs, KCs, and

    ACs can change with the needs of the application.

    Unlike traditional operating systems, applications can

    Panel 1. Abbreviations, Acronyms, and Terms

    ACApplication coreAPApplication processorAPICAdvanced programmable interrupt

    controllerBSPBootstrap processorCPUCentral processing unitFTQFixed time quantumHPCHigh performance computingIaaSInfrastructure as a service

    ICCInter-core-callI/OInput/outputIPCInterprocess communicationIPIInter processor interruptKCKernel coreOSOperating systemSMPSymmetric multiprocessingTCTimesharing coreTLBTranslation look aside buffer

  • DOI: 10.1002/bltj Bell Labs Technical Journal 43

    access OS services by sending a message to the TC

    kernel, rather than by a system call trap. Control of

    ACs is managed by means of inter-core calls. NIX

    takes advantage of the shared-memory nature of

    manycore CPUs, and passes pointers to both data and

    code to coordinate among cores.

    In the sections following, we provide a brief

    description of NIX and its capabilities. We then evaluate

    and benchmark NIX under standard HPC workloads.

    Finally, we summarize our work and discuss the appli-

    cability of the NIX design to traditional non-HPC


    NIXHigh performance applications are increasingly

    providing fast paths to avoid the higher latency of tra-

    ditional operating system interfaces, or eliminating

    the operating system entirely and running on bare

    hardware. Systems like Streamline [6] and Exokernel

    [7] either provide a minimal kernel or a derived fast

    path to bypass the operating system functions. This

    approach is particularly useful for database systems,

    fine-tuned servers, and HPC applications.

    Systems like the Multikernel [3] and, to some

    extent Helios [13], handle different cores (or groups of

    cores) as discrete systems, with their own operating

    system kernels. Sadly, this does not shield HPC appli-

    cations from the latency caused by interference from

    the system.

    We took a different path. As machines move

    towards hundreds of cores on a single chip [2], we

    believe that it is possible to avoid using the operating

    system entirely on many cores of the system. In NIX,

    applications are assigned to cores where they run

    without OS interference; that is, without an operating

    system kernel. In some sense, this is the opposite of

    the multikernel approach. Instead of using more ker-

    nels for more cores, we try to use no kernel for (some

    of) them.

    NIX is a new operating system based on this

    approach, evolving from a traditional operating sys-

    tem in a way that preserves binary compatibility but

    also enables a sharp break with the past practice of

    treating manycore systems as traditional symmetric

    multiprocessing (SMP) systems. NIX is strongly influ-

    enced by our experiences over the past five years in

    modifying and optimizing Plan 9 to run on HPC sys-

    tems such as IBMs Blue Gene, and applying our

    experience with large scale systems to general pur-

    pose computing.

    NIX is designed for heterogeneous manycore pro-

    cessors. Discussions with processor manufacturers

    revealed the following trends. First, the processor

    manufacturers would prefer that not all cores run an

    OS at all times. Second, there is a very real possibility

    that on some future manycore systems, not all cores

    will be able to support an OS. This is similar to the

    approach that IBM is taking with the CellBe [5], and

    the approach taken with the wire-speed processor

    [10] where primary processors interact with satellite

    special-purpose processors