42
Refining Linux

Refining Linux

Embed Size (px)

Citation preview

Page 1: Refining Linux

Refining Linux

Page 2: Refining Linux

Jason Murray, D.CS

Cornwall-Lebanon SD

[email protected]

@jasonmurray72

slideshare.net/jasonmurray72

Page 3: Refining Linux

Agenda

Linux Intro

What is Optimization?

Performance

Server Optimization

Performance Monitoring

System Monitoring Tools

Benchmark Tools

Page 4: Refining Linux
Page 6: Refining Linux

Kernels

Linux – Monolithic Kernel Architecture: provides all

services

Windows – Micro Kernel Architecture: small core group of

services that executes other services and applications

Page 7: Refining Linux

Recompile the Kernel make mrproper - This target cleans up the build environment of any stale

files and dependencies that might have been left over from a previous kernel

build. All previous kernel configurations will be cleaned (deleted) from the

build environment.

make clean - This target does not do as thorough a job as the “mrproper”

target. It only deletes most generated files. It does not delete the kernel

configuration file (.config).

make menuconfig - This target invokes a text-based editor interface with

menus, option lists, and text-based dialog boxes for configuring the kernel.

make xconfig - This is an X Window System–based kernel configuration tool

that relies on the Qt graphical development libraries. These libraries are used

by KDE-based applications.

make gconfig - This target also invokes an X Window System–based kernel

configuration tool, but it relies on the GTK2 (GIMP) toolkit. This GTK2 toolkit

is heavily used in the GNOME desktop world.

make help - This target will show you all the other possible make targets and

also serves as a quick online help system.

Page 8: Refining Linux

What is Optimization?

Server Design

Stability

Availability

Performance

General Server Response

Hardware Response

Service Response

Page 9: Refining Linux

Performance

Hardware

Safe Clean Environment

Software

Enhancement in parameter improvements can compensate and

improve older hardware performance

Avoid a GUI

At least unload the GUI after using it

Page 10: Refining Linux

Server Optimization

Fine Tuning

Services

Hardware

Kernel

Network

Security

Database

Storage

Memory

Page 11: Refining Linux

Recommendations

Custom installs

Only load the essentials

Partitioning Planning

Know the purpose of your server

Page 12: Refining Linux

Performance Monitoring

CPU

Network

I/O

Memory

Page 13: Refining Linux

CPU Performance Metrics

Context Switch

Run Queue

CPU Utilization

Load Average

Page 14: Refining Linux

Context Switch

Context Switch - CPU moves from one thread to another

During context switch, the kernel stores current thread state in

memory

Simultaneously, the kernel retrieves previously stored thread state

from memory & loads it to the CPU

Context switching affords multitasking of CPU

High levels of context switching can cause performance degradation

Example: pidstat 2 5

Display five reports of CPU statistics for every active task in the system at

two second intervals.

Page 15: Refining Linux

Run Queue

Run Queue – total number of active processes in current

queue for CPU

Processes in sleep state of I/O wait state are not in the

run queue

High number of processes in the run queue can cause

performance degradation

Example: vmstat

The r column in the table represents run queue processes

Page 16: Refining Linux

CPU Utilization

CPU Utilization – percentage that the CPU is being used

Higher percentage of CPU utilization can cause

performance degradation

Example: top

Real time view of cpu utilization

Page 17: Refining Linux

Load Average

Load Average – average CPU load over a specified time period

Load average is displayed for the last 1 minute, 5 minutes, and 15

minutes

Easy monitoring for overall load

Load Average is calculated by combining the total number of

processes in the run queue and the total number of processes in the

uninterruptable task status

Example: uptime

“0.25 1.20 1.90” – System load coming down

.25 - Load average for last 1 minute

1.20 – Load average for last 5 minutes

1.90 – Load average for last 15 minutes

Page 18: Refining Linux

Disk I/O Optimization

4 different I/O schedulers

CFQ (default) – distributes all I/O bandwidth evenly among all

processes requesting I/O

NOOP – acts as a basic FIFO queue expecting the hardware controller to

handle the performance operations of the requests; attempts to use as

little CPU as possible

Anticipatory – designed to increase performance on systems that have

slow disks; reorders all disk I/O operations to optimize disk seeks

Deadline – places I/O requests in a priority queue so each is guaranteed

to run within a certain time; often used in real-time operating

Page 19: Refining Linux

Changing Schedulers

Which schedules do you have compiled

/sys/block/sda/queue # cat scheduler

anticipatory deadline [cfq]

Compile noop

/sys/block/sda/queue # modprobe noop-iosched

anticipatory deadline [cfq] noop

Change

/sys/block/sda/queue # echo anticipatory > scheduler

http://www.linuxhowtos.org/System/iosched.htm

Page 20: Refining Linux

Scheduler

Agility

Changing schedulers on the fly allows for benchmark

testing of your specific applications and hardware

Notes

The change will not be instant. All current processes in

run queue must complete first.

Once you find the optimal scheduler for your specific

situation, apply the change for subsequent reboots

Page 21: Refining Linux

Scheduler Suggestions

Use NOOP or Deadline on SSD drives

Benchmark each scheduler to determine your optimal

solution

CFQ is not always the best scheduler for your system

Example: Your system is running a raid

Page 22: Refining Linux

I/O Optimization

Consistent high I/O wait indicates an issue with the disk

subsystem

Monitor BI & BO (Block in & Block out)

Reads/second & writes/second can also identify trouble areas

Check TPS (transactions/second) – total of BI & BO

Example: vmstat

Page 23: Refining Linux

vmstat Procs

r: The number of processes

waiting for run time.

b: The number of processes in

uninterruptible sleep.

Memory

swpd: the amount of virtual

memory used.

free: the amount of idle

memory.

buff: the amount of memory

used as buffers.

cache: the amount of memory

used as cache.

inact: the amount of inactive

memory. (-a option)

active: the amount of active

memory. (-a option)

Swap

si: Amount of memory swapped

in from disk (/s).

so: Amount of memory swapped

to disk (/s).

IO

bi: Blocks received from a block

device (blocks/s).

bo: Blocks sent to a block

device (blocks/s).

System

in: The number of interrupts per

second, including the clock.

cs: The number of context

switches per second.

CPU

These are percentages of total

CPU time.

us: Time spent running non-

kernel code. (user time,

including nice time)

sy: Time spent running kernel

code. (system time)

id: Time spent idle. Prior to

Linux 2.5.41, this includes IO-

wait time.

wa: Time spent waiting for IO.

Prior to Linux 2.5.41, included

in idle.

st: Time stolen from a virtual

machine. Prior to Linux 2.6.11,

unknown.

Page 24: Refining Linux

Disk I/O OptimizationFile

System

Max File

Size

Max Partition

SizeJournaling Notes

Fat16 2 GB 2 GB No Legacy

Fat32 4 GB 8 TB No Legacy

NTFS 2 TB 256 TB Yes

(For Windows Compatibility) NTFS-3g is

installed by default in Ubuntu, allowing

Read/Write support

ext2 2 TB 32 TB No Legacy

ext3 2 TB 32 TB Yes Standard linux filesystem for many years. Best

choice for super-standard installation.

ext4 16 TB 1 EB Yes

Modern iteration of ext3. Best choice for new

installations where super-standard isn't

necessary.

reiserFS 8 TB 16 TB Yes No longer well-maintained.

JFS 4PB 32PBYes

(metadata) Created by IBM - Not well maintained.

XFS 8 EB 8 EBYes

(metadata)

Created by SGI. Best choice for a mix of

stability and advanced journaling.

Page 25: Refining Linux

Disk I/O Optimization

Disk Cache Policy

hdparm -a /dev/sda

Readahead value at least 1024

Write Policy

Writeback

Read Policy

Adaptive readahead

Penalties

Calculate the IOPS

Page 26: Refining Linux

Disk Optimization

Solutions

Bonnie++

Hdparm

Upgrades

BIOS

Firmware

Page 27: Refining Linux

Network

Monitoring

Total number of packets

Received

Sent

Dropped

Page 28: Refining Linux

TCP Optimization –

Large Production Servers

Port Availability

Echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range

Memory Allocation with Socket Buffers

Echo 262143 > /proc/sys/net/core/rmem_max

Echo 262143 > /proc/sys/net/core/rmem_default

TCP Stack Workload Reduction

Echo 0 >/proc/sys/net/ipv4/tcp_sack

Echo 0 > /proc/sys/net/ipv4/tcp_timestamps

Page 29: Refining Linux

Memory Optimization

Virtual Memory – swap space available on the disk & physical memory

(user space & kernel space)

Kernel’s file system cache takes advantage of the unused RAM

During a swap, Linux writes the least used memory pages from the

physical memory to the swap space on the disk

High levels of swapping will degrade performance, because the disk is

much slower

Vmstat command

Recommendation

Benchmark tests prove that ECC RAM is faster

Page 30: Refining Linux

Additional Helpful Commands

Netstat – lists network sockets

Ps – snapshot of current processes

Atop – interactive monitor for the load averages

Mtop – displays mysql threads

Iostat – reports cpu & I/O statistics

Xosview – displays CPU usage, load average, memory

usage, swap space usage, network usage, interrupts, and

serial port status

Page 31: Refining Linux

Kernel Optimization

The first thing to get loaded is the kernel. Keep in mind that no operating system

exists in memory at this point, and PCs (by their unfortunate design) have no easy

way to access all of their memory. Thus, the kernel must load completely into the

first megabyte of available random access memory (RAM). In order to accomplish

this, the kernel is compressed. The head of the file contains the code necessary to

bring the CPU into protected mode (thereby removing the memory restriction) and

decompress the remainder of the kernel.

Kernel Execution. With the kernel in memory, it can begin executing. It knows only

whatever functionality is built into it, which means any parts of the kernel

compiled as modules are useless at this point. At the very minimum, the kernel

must have enough code to set up its virtual memory subsystem and root file system

(usually, the ext3 file system). Once the kernel has started, a hardware probe

determines what device drivers should be initialized. From here, the kernel can

mount the root file system. (You could draw a parallel of this process to that of

Windows being able to recognize and access its C drive.) The kernel mounts the

root file system and starts a program called init.

Page 32: Refining Linux

Boot Loaders

MBR – starting on page 142 of Linux Administration

GRUB

Modify boot record & kernel initiation

Splash screen

Hidden menus

Title

Root

Initrd – preload modules in kernel

LILO

Page 33: Refining Linux

Database Optimization

MySQL – use MySQL tuning scripts

Optimize table

Page 34: Refining Linux

OpenLDAP Optimization

Important Decision – Which attributes will you build indexes on

Optimal Parameters – change before data entry

/etc/openldap.conf

Cachesize 10000

Dbcachesize 100000

Sizelimit 10000

Loglevel 0

dbcacheNoWsync

Index cn,uid

Index uidnumber

Index gid

Index gidnumber

Index mail

Page 35: Refining Linux

Apache Optimization

Good Benchmark Scores – many initial daemons

Example

MinSpareServers 20

MaxSpareServers 80

StartServers 32

If Apache is recompiled

MaxClients 256

MaxRequestsPerChild 10000

Notes – A high number of max servers and an honest

number of spare servers will work fine. For the most part

the StartServers help with the instant load that

benchmarks generate.

Page 36: Refining Linux

Apache Management

Default Directory - /var/www/html

Apache Config File - /etc/httpd/httpd.conf

/etc/httpd/conf.d/phpMyAdmin.conf

PHP Config File - /etc/php.ini

/etc/init.d/httpd start|stop|restart

Service httpd start|stop|restart

/var/log/httpd/error_log

/var/log/httpd/access_log

Page 37: Refining Linux

Poor Performance Website

Consider

Optimizers

Memcache

Adjusting Apache

Minimizing Apache modules

Alternative to apache - nginx

Page 38: Refining Linux

Benchmarking

Benchmarking is helping; HOWEVER,

Its impossible to duplicate real world situations

Understand the GOAL for benchmarking

Measuring the performance of ONE particular element very

accurately

Benchmarks are excellent tools if you understand what

they are doing and trying to accomplish

Page 39: Refining Linux

Benchmark Tools

Bonnie++ - file system benchmarking tool

Dbench – I/O workload generator to a file system or

networked CIFS or NFS server

Http_load – throughput tester for a web server

Dkftpbench – FTP download benchmark analyzer

Tiobench – multi-threaded I/O benchmark tool

Ttcp – network throughput measuring tool

Netperf – network performance tester

Page 40: Refining Linux

More Helpful Tools

Lsof – list open files

Tcpdump – network packet analyzer

Htop – process monitoring

Iotop – monitors disk I/O

IPTraf – real time IP LAN monitoring

Psacct – monitor user activity

Monit – process & services monitoring

Nethogs – monitor per process network bandwidth

Iftop – network bandwidth monitoring

Monitorix – system & network monitoring

Page 41: Refining Linux

Troubleshooting Methodology

Clearly understand the problem

Monitor & collect data

Eliminate and narrow issues

Make one change at a time & OBSERVE

Document everything!

Page 42: Refining Linux

References

Davor Guttierrez - @dguttierrez

Soyinka, W. (2008). Linux Administration: A Beginner's Guide.

McGraw-Hill, Inc..

http://www.tecmint.com/command-line-tools-to-monitor-linux-

performance/

http://www.thegeekstuff.com/2011/07/iostat-vmstat-mpstat-

examples/