Operating Systems - Lecture 1 - Introductionadrien.krahenbuhl.fr/.../M1-System/System-Lecture1-Introduction.pdf · Lecture 1 - Introduction AdrienKr¨ahenbuhl ... X SMP(SymmetricMultiProcessor)

Operating SystemsLecture 1 - Introduction

Adrien KrahenbuhlMaster of Computer Science

PUF - Hồ Chí Minh

2016/2017

Copyright and License

Copyright c© 2016-2017 Adrien Krahenbuhl.

X Thanks to Raymond Namyst to help me to build the course.X Thanks to David Goglin and Pierre-Antoine Champin for their

available slides.X This support is under license Creative Commons

CC-BY-NC-SA:

https://creativecommons.org/licenses/by-nc-sa/

https://creativecommons.org/licenses/by-nc-sa/

How to join me ?

Website http://adrien.krahenbuhl.fr/en/teaching/

Mail [email protected] Here!

http://adrien.krahenbuhl.fr/en/teaching/

mailto:[email protected]

Program

Lectures X 7×3h = 21hX with me, from 13/02 until 23/02

Tutorials X 13 × 3h = 39hX After lectures, supervised by Ms. Le Thanh Van

Evaluation X One exam after the lecturesX One project in pair

All documents will be available online on my website.

Before to start

X Who followed a course about “Computer architecture”?

X Who followed courses about “System programming”?X Who never used a Linux system?X Who already a linux kernel?

Before to start

X Who followed a course about “Computer architecture”?X Who followed courses about “System programming”?

X Who never used a Linux system?X Who already a linux kernel?

Before to start

X Who followed a course about “Computer architecture”?X Who followed courses about “System programming”?X Who never used a Linux system?

X Who already a linux kernel?

Before to start

X Who followed a course about “Computer architecture”?X Who followed courses about “System programming”?X Who never used a Linux system?X Who already compiled a linux kernel?

Before to start

X Who followed a course about “Computer architecture”?X Who followed courses about “System programming”?X Who never used a Linux system?X Who already modified a linux kernel?

Before to start

X Who followed a course about “Computer architecture”?X Who followed courses about “System programming”?X Who never used a Linux system?X Who already wrote a linux kernel?

What we will seeA Layered view:

System programming

Operating system

Computer architecture

A wide domain:X from software conceptsX to hardware implementationX using algorithms at each step.

It will be illustrated through the Linux kernel.

Course plan

Lecture 1 IntroductionLecture 2 Generic conceptsLecture 3 Concurrency and synchronizationLecture 4 Time managementLecture 5 Memory management

X 5.1 IntroductionX 5.2 PagingX 5.3 Kernel memory

Lecture 6 File systemLecture 7 Input/Output

Bibliography

X Modern Operating Systems, by Andrew S. Tanenbaum, ed.Pearson

X Operating Systems: Internals and Design Principles, byWilliam Stallings, ed. Pearson

X A lot of web references.

Introduction

A first OS: the BIOS

General concepts

Systems

Introduction A first OS: the BIOS General concepts Systems

What is an OS ?

A first definitionAn OS is an interface between softwares and hardware.

Objectives in 4A:Assist make things simple to users, programmers, softwares.

X code modularity & maintainabilityAbstract expose a virtual interface independent of hardware.

X code portabilityArbitrate manage access to resources

X system reliabilityAuthorize control access to resources

X system security

Operating Systems 1 - Introduction 9/46


What is an OS ?

A first definitionAn OS is an interface between softwares and hardware.

Objectives in 4A:Assist make things simple to users, programmers, softwares.

X code modularity & maintainabilityAbstract expose a virtual interface independent of hardware.

X code portabilityArbitrate manage access to resources

X system reliabilityAuthorize control access to resources

X system security



What is an OS ?

Operating system

Computer architecture

Proc. Proc. Proc. Proc. Proc. Proc. Proc.

{ { {User A User B User C

An OS needs to manage the resources of one computer with:Several processes

X even with few processorsX without resource conflictsX without starvation

Several usersX without starvationX with different privilege levels



What is NOT an OS ?

A compiler as gcc, ocamlc, . . .A windowing system as X.org, . . .

A command interpreter as bash, dash, . . .A command line tool as cp, mv, . . .

The root userA “men in black” process monitoring and controlling all the

computer from a back room



What CAN BE an OS ?

The kernelthe core executed in privileged mode.

Some libraries aroundas the standard libraries (libc in Linux) providing the systemcalls (GNU/Linux vs. Linux).

Some other tools, services or daemonsthat help the OS to be easier to use.



OS examples

X Linux (on a dozen architectures)X Windows (x86 and IA64)X MacOS X (x86 and PowerPC)X Solaris (x86 and Sparc)X FreeBSD, NetBSD, OpenBSDX GNU/HurdX PalmOS, Plan9, Symbian, VxWorks, . . .



Root vs. kernel

The root user is not the kernelX Root has a privileged access to the kernelX The kernel let the root user do more thingsX The kernel controls everything:

I it accesses to everything by system callsI it checks what the users want to do

X Root is privileged at software level by the kernelX The kernel is privileged at hardware level by the processor



Why is it interesting ?

X Know how an OS works allows to better exploit its computerX “Reinvent the wheel” is uselessX A lot of recurrent problems of three types:

technical hardware managementsoftware design of useful and efficient interfaces

theoretical hard algorithmic problems



Reminders on computer architecture

Processor

X executes instructions of programsX uses data in registersX transfers from/to memory by 1/2/4/8/16 bytes

Memory (Random Access Memory)X stores data and programsX volatile

Memory controller (Memory chipset)X manages memory accesses from processor or peripheral

devices



Reminders on computer architecture

Peripheral devices

X considered as input/output for storage, network, . . .Input/Output controller (I/O chipset)

X manages communications between peripheral devices,processor and memory

BIOS (Basic Interpreter Operating System)X Non-volatileX detects and configures peripheral devicesX minimal system to boot on the real OS



Numeric scales

Processor Frequency:

3Ghz

Instruction cycle:

1ns

Atomic instruction:

100ns

System call:

50ns - 1000ns

Memory

10Go, 50-100ns, 50Gb/s

Cache memory

10kb-10Mb, 1-10ns

Bus PCI

Gb/s - 100+ns

Interruption

5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales

Processor Frequency: 3GhzInstruction cycle:

1ns

Atomic instruction:

100ns

System call:

50ns - 1000ns

Memory

10Go, 50-100ns, 50Gb/s

Cache memory

10kb-10Mb, 1-10ns

Bus PCI

Gb/s - 100+ns

Interruption

5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales

Processor Frequency: 3GhzInstruction cycle: 1nsAtomic instruction:

100ns

System call:

50ns - 1000ns

Memory

10Go, 50-100ns, 50Gb/s

Cache memory

10kb-10Mb, 1-10ns

Bus PCI

Gb/s - 100+ns

Interruption

5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales

Processor Frequency: 3GhzInstruction cycle: 1nsAtomic instruction: 100nsSystem call:

50ns - 1000ns

Memory

10Go, 50-100ns, 50Gb/s

Cache memory

10kb-10Mb, 1-10ns

Bus PCI

Gb/s - 100+ns

Interruption

5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales

Processor Frequency: 3GhzInstruction cycle: 1nsAtomic instruction: 100nsSystem call: 50ns - 1000ns

Memory

10Go, 50-100ns, 50Gb/s

Cache memory

10kb-10Mb, 1-10ns

Bus PCI

Gb/s - 100+ns

Interruption

5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales


Memory 10Go, 50-100ns, 50Gb/sCache memory

10kb-10Mb, 1-10ns

Bus PCI

Gb/s - 100+ns

Interruption

5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales


Memory 10Go, 50-100ns, 50Gb/sCache memory 10kb-10Mb, 1-10ns

Bus PCI

Gb/s - 100+ns

Interruption

5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales



Bus PCI Gb/s - 100+nsInterruption

5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales



Bus PCI Gb/s - 100+nsInterruption 5µ

Disks Mechanic:

1+Tb, 10ms, 10-100Mb/s

SSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales




Disks Mechanic: 1+Tb, 10ms, 10-100Mb/sSSD, NVMe:

0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales




Disks Mechanic: 1+Tb, 10ms, 10-100Mb/sSSD, NVMe: 0.1-10Tb, 1-100µ, 0.1-1Gb

Network At home:

10-1000ms, 1-100Mb/s

HPC:

1µ, 10Go/s



Numeric scales





Network At home: 10-1000ms, 1-100Mb/sHPC:

1µ, 10Go/s



Numeric scales





Network At home: 10-1000ms, 1-100Mb/sHPC: 1µ, 10Go/s


Introduction


General concepts

Systems


What happens during the boot process?

The hardware boot:1. At the beginning, processors doesn’t know what to do... and

the memory is empty2. The first core is configured to execute the code stored at a

specific address:I The system is configured to have its boot code at this memory

addressI This address points to a special memory, a ROM (Read Only

Memory) non-volatile




Boot on the first OS:3. The processor executes the BIOS, UEFI or Open Firmware4. Low-level interface with hardware to detect components:

I Other processors, peripheral memories, . . .I Other resources as embedded memories, ports, . . .

5. Allows to do more or less things:I Graphical configuration for processor tunning, choice of boot

devices, . . .I Execute codes from several locations as the first hard disk, a

CD, a USB device, the network (PXE), . . .6. At the end, BIOS loads the OS code . . .

I often from the first hard disk (Master Boot Record)7. . . . and executes it.




6. After the BIOS, it is not an obligation to boot on a real OS.We can boot on a Boot loader (Lilo or GRUB), a minimalOS designed to choose:X On which real OS to boot

I on which partition of the hard diskI with which kernel file → imply to be able to read this file

X With which parameters




7. The Boot loader jump to the kernel program and the kernel:a detects hardware resources through the BIOSb builds basic software resources: code loading, page table,

device descriptors, . . .c starts other coresd starts some daemons (kernel threads for the kernel tasks in

background)



OS booting

X The task “kernel loading” turns into a real user process: InitI The kernel is functional, the OS starts.

X Init loads software services configured by administrator:I Peripheral manager: loading of driver modulesI User daemons: user services as background tasks, session

manager, . . .



Summary


Introduction


General concepts

Systems


OS goals

Make the system simpler to use

X abstraction of peripheral devicesImprove the system efficiency

X provide resources efficientlyFacilitate development

X keep the same way to provide the same servicesProtect programsSecure with regard to other users



Interface between human and computer

X Interface with a uniform access to peripheral devicesI hide technical details

X Program executionX Controlled accesses to data:

I Show the structured content of storing and network devicesI Provide functions of edition, development, communication, . . .

X Manage errors properly:I report hardware errors to programsI react in case of software error: “punish or correct”

X Provides usage and execution statisticsI the user can adapt the system configuration more easily (e.g.

see manpage of df )



Historical evolution

Serial Processing ~1950X Manual start of a new program after the previous one end

Simple batch system ~1960X Queue of programs

Multiprogrammed batch systems ~1970X Take advantage of hardware by alternating processes

Time sharing ~1980X Automatic alternating between processesX Interactivity



What is a process?

X A dedicated memory contextI Execution independent from other processes

X Data private for the OSI In order to manipulate the process

X Concurrent executions to maximize the use of hardwareresources

X An execution context (registers + stack)I Eventually several in case of multithreadingI Allows blocking I/O without block the entire process



Multitasking system and preemption

Multitasking systemExistence and execution of several and simultaneous tasks

Preemptive systemNon-constrained scheduling, depending of the current processX The system can force this scheduling



Memory management

X Process isolationI No collision between memories → clean data and instructionsI Allowed by virtual memory and hardware support

X Transparent allocation and managementI No constraint for the programmer

X Modular programmingI Reusable libraries

X Memory sharing with protectionX Storage in persistent memory



Data protection and security

X Authorization controlI Access to critical system dataI Access and modification of user data

X User authenticationX Survival to a technical problem ?



Resource scheduling and management

X Fair access to resourcesI Specifically for tasks of same type: how do that with

multithreaded processesX Make differences depending on task classes: interactive, I/O,

pure computing, . . .X Dynamic controlX Efficiency

I Hardware using and reactivityX Scheduling of processes and/or I/O

I In which order send reading/writing to the disk?X Deadline constraints

I Event must occur before . . .I Real time systems

X Hardware constraints (ex: elevator)



SMP and NUMA

X New scheduling constraintsX SMP (Symmetric MultiProcessor)

I Multiple processorsI Concurrent accessI Affinity for a processor: cache, TLB, ...I Affinity of threads of a same process

X NUMA (Non-Uniform Memory Access)I Affinity for memory areas


Introduction


General concepts

Systems

Structure


Structure of an OSX Kernel

I CoresI Drivers of peripheral devicesI Interface for the user

X Low-level libraries for userI System calls to discuss with the kernel → Essential ! provided

by the kernel.X Above: your applications.X Memory management, central part of the system

I ProcessesI File systemI Network

X File system and storageX SchedulingX Communication inter-processX Network



:



Monolithic kernel

Operating system

System

VFS, System call

IPC, File System

Scheduler, Virtual Memory

Device Drivers, Dispatcher, ...

Application

Hardware

Monolithic Kernelbased Operating System

kernelmode

usermode

X Set of procedures able to call each otherI Very few organization

X Often divided in:Main procedures system calls

Service procedures effective processingUtility procedures management of lists, hash tables, . . .



Monolithic kernel by layers

Operating system

System

kernelmode

usermode

Basic IPC, Virtual Memory, Scheduling

UNIXServer

DeviceDriver

ApplicationIPC

Hardware

"Hybrid kernel"based Operating System

Application

FileServer

X Try to organize the monolithic partsX Hard to define limits between layers:

I Memory management needs I/Os to fill pagesI I/Os need memory to allocate temporary buffers

X Passing through all layers restrict performances



Modular kernels

Operating system

System

kernelmode

usermode


UNIXServer

DeviceDriver

FileServer

ApplicationIPC

Hardware

Microkernelbased Operating System

Application

X Dynamic loading/unloading of optional code:I Device driversI Specific functionalities

X Reduction of the initial kernelX Allow dynamic evolutions (without reboot)



Monolithic or micro kernels?

Monolithic too big? X Not organized, even with layersX Hard to incorporate security: too many

interactionsX Hard to maintain and evolve

Micro kernel X Only the real necessitiesX Everything else in dedicated server

processes

Eternal dispute between Torvalds and Tanenbaum . . .



Micro kernels

Operating system

System

kernelmode

usermode


UNIXServer

DeviceDriver

FileServer

ApplicationIPC

Hardware


Application

DesignX A server for each task (processes, memory, . . . )

I Each server in a isolated memory spaceI Communication with applications by messages

X Simple and uniform interfacesX Extensible, flexible, reliable



Micro kernels

Operating system

System

kernelmode

usermode


UNIXServer

DeviceDriver

FileServer

ApplicationIPC

Hardware


Application

PerformancesX Use of messages slower than direct calls to kernel?X Too many critical interactions between OS components?X Hard to compare to monolithic kernels: no real OS

available (GNU/Hurd?)



Linux monolithic?

http://www.makelinux.net/kernel_map/


http://www.makelinux.net/kernel_map/

Examples


Windows

X Very modularI Set of Managers (Executive): I/O, Cache, PnP, Power,

Security, VM, Process, . . .I Abstraction layer for devicesI Device driversI Windowing and graphic design

X Hybrid kernel (pseudo micro kernel)I Almost all run in only one kernel space for performance reasons

X Local Procedure CallsI Communication by messages between applications and

managers: client/server model



Windows

X Object-oriented design (polymorphism)I Handles allow to manipulate everything by the same way

X SMP support, threads, interprocess communicationsX Scheduler of preemptive threads with priorities:

I Affinities with cache and NUMAI Dynamic adjustment of priorities according to passive waiting

(GUI) and CPU consumptionX Initially designed for office



Unix

X Created at the end of 60sX Designed for servers and networksX Portable because quickly wrote in CX “All is file”X Kernel + set of libraries and applications (libc)

I System calls through the System Call InterfaceX Monolithic kernel

I Common functionalitiesI Memory managerI Bloc devicesI Character devicesI SchedulerI Vnode/VFS interface for files



Unix

System V release 4 Academic and commercial (AT&T and Sun)Solaris (SunOS) Commercial distribution based on SVR4

BSD (Berkley Software distribution)X Widespread in the academic worldX Base of MacOS

Minix Clone with educational goal



Linux

X Appeared in 1991X Based on Minix: cheap for personal usageX Free (GPL-2)X Collaborative development on internetX Supports a lot of architectures and peripheral devicesX Designed for servers, workstations, office stations, embedded,

. . .X Dynamic loading of kernel modules

I Automatic loading depending on application needsI Hierarchy based on symbol dependencies

X Particular monolithic structureI Only some functions available for other modules

(EXPORT_SYMBOL)I No fixed interface



GNU Hurd

X The only real OS based on a micro kernel?I Unix on micro kernel MachI Limited device drivers

X IPC between components by the micro kernelI Server and application processesI The user can run its own servers (network, file system, . . . )

I without to be rootI without impact on the rest of the system

X Translator notionI Functionality managed by processes (file, folder, network

access, . . . )I Easy to browse a website, a ftp server, an archive, . . .


Documents

Operating Systems - Lecture 1 - Introductionadrien.krahenbuhl.fr/.../M1-System/System-Lecture1-Introduction.pdf · Lecture 1 - Introduction AdrienKr¨ahenbuhl ... X SMP(SymmetricMultiProcessor)