Upload
jason-murray
View
101
Download
3
Tags:
Embed Size (px)
Citation preview
Refining Linux
Jason Murray, D.CS
Cornwall-Lebanon SD
@jasonmurray72
slideshare.net/jasonmurray72
Agenda
Linux Intro
What is Optimization?
Performance
Server Optimization
Performance Monitoring
System Monitoring Tools
Benchmark Tools
Kernels
Linux – Monolithic Kernel Architecture: provides all
services
Windows – Micro Kernel Architecture: small core group of
services that executes other services and applications
Recompile the Kernel make mrproper - This target cleans up the build environment of any stale
files and dependencies that might have been left over from a previous kernel
build. All previous kernel configurations will be cleaned (deleted) from the
build environment.
make clean - This target does not do as thorough a job as the “mrproper”
target. It only deletes most generated files. It does not delete the kernel
configuration file (.config).
make menuconfig - This target invokes a text-based editor interface with
menus, option lists, and text-based dialog boxes for configuring the kernel.
make xconfig - This is an X Window System–based kernel configuration tool
that relies on the Qt graphical development libraries. These libraries are used
by KDE-based applications.
make gconfig - This target also invokes an X Window System–based kernel
configuration tool, but it relies on the GTK2 (GIMP) toolkit. This GTK2 toolkit
is heavily used in the GNOME desktop world.
make help - This target will show you all the other possible make targets and
also serves as a quick online help system.
What is Optimization?
Server Design
Stability
Availability
Performance
General Server Response
Hardware Response
Service Response
Performance
Hardware
Safe Clean Environment
Software
Enhancement in parameter improvements can compensate and
improve older hardware performance
Avoid a GUI
At least unload the GUI after using it
Server Optimization
Fine Tuning
Services
Hardware
Kernel
Network
Security
Database
Storage
Memory
Recommendations
Custom installs
Only load the essentials
Partitioning Planning
Know the purpose of your server
Performance Monitoring
CPU
Network
I/O
Memory
CPU Performance Metrics
Context Switch
Run Queue
CPU Utilization
Load Average
Context Switch
Context Switch - CPU moves from one thread to another
During context switch, the kernel stores current thread state in
memory
Simultaneously, the kernel retrieves previously stored thread state
from memory & loads it to the CPU
Context switching affords multitasking of CPU
High levels of context switching can cause performance degradation
Example: pidstat 2 5
Display five reports of CPU statistics for every active task in the system at
two second intervals.
Run Queue
Run Queue – total number of active processes in current
queue for CPU
Processes in sleep state of I/O wait state are not in the
run queue
High number of processes in the run queue can cause
performance degradation
Example: vmstat
The r column in the table represents run queue processes
CPU Utilization
CPU Utilization – percentage that the CPU is being used
Higher percentage of CPU utilization can cause
performance degradation
Example: top
Real time view of cpu utilization
Load Average
Load Average – average CPU load over a specified time period
Load average is displayed for the last 1 minute, 5 minutes, and 15
minutes
Easy monitoring for overall load
Load Average is calculated by combining the total number of
processes in the run queue and the total number of processes in the
uninterruptable task status
Example: uptime
“0.25 1.20 1.90” – System load coming down
.25 - Load average for last 1 minute
1.20 – Load average for last 5 minutes
1.90 – Load average for last 15 minutes
Disk I/O Optimization
4 different I/O schedulers
CFQ (default) – distributes all I/O bandwidth evenly among all
processes requesting I/O
NOOP – acts as a basic FIFO queue expecting the hardware controller to
handle the performance operations of the requests; attempts to use as
little CPU as possible
Anticipatory – designed to increase performance on systems that have
slow disks; reorders all disk I/O operations to optimize disk seeks
Deadline – places I/O requests in a priority queue so each is guaranteed
to run within a certain time; often used in real-time operating
Changing Schedulers
Which schedules do you have compiled
/sys/block/sda/queue # cat scheduler
anticipatory deadline [cfq]
Compile noop
/sys/block/sda/queue # modprobe noop-iosched
anticipatory deadline [cfq] noop
Change
/sys/block/sda/queue # echo anticipatory > scheduler
http://www.linuxhowtos.org/System/iosched.htm
Scheduler
Agility
Changing schedulers on the fly allows for benchmark
testing of your specific applications and hardware
Notes
The change will not be instant. All current processes in
run queue must complete first.
Once you find the optimal scheduler for your specific
situation, apply the change for subsequent reboots
Scheduler Suggestions
Use NOOP or Deadline on SSD drives
Benchmark each scheduler to determine your optimal
solution
CFQ is not always the best scheduler for your system
Example: Your system is running a raid
I/O Optimization
Consistent high I/O wait indicates an issue with the disk
subsystem
Monitor BI & BO (Block in & Block out)
Reads/second & writes/second can also identify trouble areas
Check TPS (transactions/second) – total of BI & BO
Example: vmstat
vmstat Procs
r: The number of processes
waiting for run time.
b: The number of processes in
uninterruptible sleep.
Memory
swpd: the amount of virtual
memory used.
free: the amount of idle
memory.
buff: the amount of memory
used as buffers.
cache: the amount of memory
used as cache.
inact: the amount of inactive
memory. (-a option)
active: the amount of active
memory. (-a option)
Swap
si: Amount of memory swapped
in from disk (/s).
so: Amount of memory swapped
to disk (/s).
IO
bi: Blocks received from a block
device (blocks/s).
bo: Blocks sent to a block
device (blocks/s).
System
in: The number of interrupts per
second, including the clock.
cs: The number of context
switches per second.
CPU
These are percentages of total
CPU time.
us: Time spent running non-
kernel code. (user time,
including nice time)
sy: Time spent running kernel
code. (system time)
id: Time spent idle. Prior to
Linux 2.5.41, this includes IO-
wait time.
wa: Time spent waiting for IO.
Prior to Linux 2.5.41, included
in idle.
st: Time stolen from a virtual
machine. Prior to Linux 2.6.11,
unknown.
Disk I/O OptimizationFile
System
Max File
Size
Max Partition
SizeJournaling Notes
Fat16 2 GB 2 GB No Legacy
Fat32 4 GB 8 TB No Legacy
NTFS 2 TB 256 TB Yes
(For Windows Compatibility) NTFS-3g is
installed by default in Ubuntu, allowing
Read/Write support
ext2 2 TB 32 TB No Legacy
ext3 2 TB 32 TB Yes Standard linux filesystem for many years. Best
choice for super-standard installation.
ext4 16 TB 1 EB Yes
Modern iteration of ext3. Best choice for new
installations where super-standard isn't
necessary.
reiserFS 8 TB 16 TB Yes No longer well-maintained.
JFS 4PB 32PBYes
(metadata) Created by IBM - Not well maintained.
XFS 8 EB 8 EBYes
(metadata)
Created by SGI. Best choice for a mix of
stability and advanced journaling.
Disk I/O Optimization
Disk Cache Policy
hdparm -a /dev/sda
Readahead value at least 1024
Write Policy
Writeback
Read Policy
Adaptive readahead
Penalties
Calculate the IOPS
Disk Optimization
Solutions
Bonnie++
Hdparm
Upgrades
BIOS
Firmware
Network
Monitoring
Total number of packets
Received
Sent
Dropped
TCP Optimization –
Large Production Servers
Port Availability
Echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
Memory Allocation with Socket Buffers
Echo 262143 > /proc/sys/net/core/rmem_max
Echo 262143 > /proc/sys/net/core/rmem_default
TCP Stack Workload Reduction
Echo 0 >/proc/sys/net/ipv4/tcp_sack
Echo 0 > /proc/sys/net/ipv4/tcp_timestamps
Memory Optimization
Virtual Memory – swap space available on the disk & physical memory
(user space & kernel space)
Kernel’s file system cache takes advantage of the unused RAM
During a swap, Linux writes the least used memory pages from the
physical memory to the swap space on the disk
High levels of swapping will degrade performance, because the disk is
much slower
Vmstat command
Recommendation
Benchmark tests prove that ECC RAM is faster
Additional Helpful Commands
Netstat – lists network sockets
Ps – snapshot of current processes
Atop – interactive monitor for the load averages
Mtop – displays mysql threads
Iostat – reports cpu & I/O statistics
Xosview – displays CPU usage, load average, memory
usage, swap space usage, network usage, interrupts, and
serial port status
Kernel Optimization
The first thing to get loaded is the kernel. Keep in mind that no operating system
exists in memory at this point, and PCs (by their unfortunate design) have no easy
way to access all of their memory. Thus, the kernel must load completely into the
first megabyte of available random access memory (RAM). In order to accomplish
this, the kernel is compressed. The head of the file contains the code necessary to
bring the CPU into protected mode (thereby removing the memory restriction) and
decompress the remainder of the kernel.
Kernel Execution. With the kernel in memory, it can begin executing. It knows only
whatever functionality is built into it, which means any parts of the kernel
compiled as modules are useless at this point. At the very minimum, the kernel
must have enough code to set up its virtual memory subsystem and root file system
(usually, the ext3 file system). Once the kernel has started, a hardware probe
determines what device drivers should be initialized. From here, the kernel can
mount the root file system. (You could draw a parallel of this process to that of
Windows being able to recognize and access its C drive.) The kernel mounts the
root file system and starts a program called init.
Boot Loaders
MBR – starting on page 142 of Linux Administration
GRUB
Modify boot record & kernel initiation
Splash screen
Hidden menus
Title
Root
Initrd – preload modules in kernel
LILO
Database Optimization
MySQL – use MySQL tuning scripts
Optimize table
OpenLDAP Optimization
Important Decision – Which attributes will you build indexes on
Optimal Parameters – change before data entry
/etc/openldap.conf
Cachesize 10000
Dbcachesize 100000
Sizelimit 10000
Loglevel 0
dbcacheNoWsync
Index cn,uid
Index uidnumber
Index gid
Index gidnumber
Index mail
Apache Optimization
Good Benchmark Scores – many initial daemons
Example
MinSpareServers 20
MaxSpareServers 80
StartServers 32
If Apache is recompiled
MaxClients 256
MaxRequestsPerChild 10000
Notes – A high number of max servers and an honest
number of spare servers will work fine. For the most part
the StartServers help with the instant load that
benchmarks generate.
Apache Management
Default Directory - /var/www/html
Apache Config File - /etc/httpd/httpd.conf
/etc/httpd/conf.d/phpMyAdmin.conf
PHP Config File - /etc/php.ini
/etc/init.d/httpd start|stop|restart
Service httpd start|stop|restart
/var/log/httpd/error_log
/var/log/httpd/access_log
Poor Performance Website
Consider
Optimizers
Memcache
Adjusting Apache
Minimizing Apache modules
Alternative to apache - nginx
Benchmarking
Benchmarking is helping; HOWEVER,
Its impossible to duplicate real world situations
Understand the GOAL for benchmarking
Measuring the performance of ONE particular element very
accurately
Benchmarks are excellent tools if you understand what
they are doing and trying to accomplish
Benchmark Tools
Bonnie++ - file system benchmarking tool
Dbench – I/O workload generator to a file system or
networked CIFS or NFS server
Http_load – throughput tester for a web server
Dkftpbench – FTP download benchmark analyzer
Tiobench – multi-threaded I/O benchmark tool
Ttcp – network throughput measuring tool
Netperf – network performance tester
More Helpful Tools
Lsof – list open files
Tcpdump – network packet analyzer
Htop – process monitoring
Iotop – monitors disk I/O
IPTraf – real time IP LAN monitoring
Psacct – monitor user activity
Monit – process & services monitoring
Nethogs – monitor per process network bandwidth
Iftop – network bandwidth monitoring
Monitorix – system & network monitoring
Troubleshooting Methodology
Clearly understand the problem
Monitor & collect data
Eliminate and narrow issues
Make one change at a time & OBSERVE
Document everything!
References
Davor Guttierrez - @dguttierrez
Soyinka, W. (2008). Linux Administration: A Beginner's Guide.
McGraw-Hill, Inc..
http://www.tecmint.com/command-line-tools-to-monitor-linux-
performance/
http://www.thegeekstuff.com/2011/07/iostat-vmstat-mpstat-
examples/