Red Hat Enterprise Linux 7Performance Co-Pilot : Overview

Agenda:

➢ Objective of performance monitoring➢ How is this typically done ?➢ Introduction to Performance Co-Pilot➢ Component and Architecture

➢ Useful performance monitoring tools➢ Demo➢ Knowledge sources

Objective of performance monitoring

➢ Early detection of a ( potential ) problem.

➢ Rapidly drill down & pinpoint issue in specific program.

How is this typically done ?

➢ System log files ( rsyslog/syslog-ng/journald )

➢ Native performance monitoring tools ( top/iostat/vmstat/ps )

➢ Mixture of scripting languages ( bash/perl/python )

➢ Specific tools vary per platform

Why we need another Performance Monitoring Tool ?

➢ Ability to monitor performance statistics of a local or remote machines in a complex network.

➢ Ability to record and replay performance statistics.

➢ Ability to monitor multiple hosts performance statistics concurrently.

➢ Customization of clients application to collect specific metrics.

➢ Performance regression detection.

➢ Evaluation of effects of an operating system upgrade etc.

Introduction to Performance Co-Pilot

➢ PCP is a system level performance monitoring and performance management toolkit.

➢ PCP provides a range of services that may be used for collection, monitoring and analysis of system metrics.

➢ The Performance Co-Pilot includes many facilities for creating and replaying archive logs that capture performance information.

➢ Cross platform support : Linux, Mac OS and Windows.

➢ End-to-end : Hardware, Core OS, services and applications.

Introduction: Performance Co-Pilot

➢ Distributed architecture

Monitoring of local and remote nodes.

➢ Real-time or retrospective

Live system or archive.

➢ Pluggable

New agents system metrics within PCP.

Performance Co-Pilot - Components

Collectors

● Collect and export performance metrics● Performance Metric Domain Agents (PMDA)● Performance Metric Collection Daemon (PMCD)

Consumers

● Record, visualise, monitor and analyse performance data● Consume data either in realtime or replay historical data from

archive logs

Note: Hosts may operate as collectors, consumers or both

● Multiple consumers may connect with one or more collectors

Performance Co-Pilot - Collectors

Performance Metric Domain Agents (PMDA)

● Extracts & exports metric data from a system component● Communicates with pmcd on local system

Performance Metric Collection Daemon (PMCD)

● One pmcd process per host● Coordinates handling of fetch requests between consumer

applications and agents● Listens for connections from localhost & remote clients

● Authenticated & encrypted connection options

Performance Co-Pilot : Consumers● pmlogger

● Utility to capture and store metrics exported by PMCD● Concurrent logging of data from local and remote hosts● Archive playback by other consumer tools (eg pmchart, pmval)● Tools for log archival, log rotation etc● Operates on data in realtime only

● pminfo - display PCP metrics available on a host or in an archive

● pmchart - GUI utility providing graphical display of PCP data

● pmstat - vmstat-like utility

● pmatop - top-like utility

● pmie - inference engine and alerting utility

● pmval - display a metric

Performance Co-Pilot - Architecture

Webserver

DBMS

Network

Kernel

PMDAs

PMCD

pmchart

pmstat

pmlogger

Consumers

Archive ClientsConsumers

Performance Co-Pilot – Distributed Architecture

AgentsPMCD

PMDA'sPMCD

Host 2

PMDA'sPMCD

Host 3

PMDA'sPMCD

Host 1

ClientsConsumers

Clients

Host 4

Consumers

Installation:

➢ To install pcp and pcp-gui package, type the following command:

# yum install pcp pcp­gui

“pcp” package provides a framework and services to support system-level performance monitoring and performance management.

“pcp-gui” package provides a visualization tools for the Performance Co-Pilot toolkit.

Services:To start pmlogger and pcp services, type the following command:

# chkconfig pmlogger on

# service pmlogger start

Starting pmlogger ...

# service pcp start

Starting pmcd ...

Starting pmlogger ...

To check the status of pcp service, type the following command:

# service pcp status

Checking for pmcd: running

Checking for pmlogger: running

Note: Bug 1044682 - pcp should use systemd

Verification:pcp - Summary of PCP installation

# pcp

Performance Co­Pilot configuration on localhost.localdomain:

 platform: Linux localhost.localdomain 3.10.0­121.el7.x86_64 #1 SMP Tue Apr 8 10:48:19 EDT 2014 x86_64

 hardware: 2 cpus, 1 disk, 1 node, 1840MB RAM

 timezone: IST­5:30

     pmcd: Version 3.8.10­1, 6 agents

     pmda: pmcd proc xfs linux mmv jbd2

# pcp ­h `hostname`

Performance Co­Pilot configuration on localhost.localdomain:

 platform: Linux localhost.localdomain 3.10.0­121.el7.x86_64 #1 SMP Tue Apr 8 10:48:19 EDT 2014 x86_64

 hardware: 2 cpus, 1 disk, 1 node, 1840MB RAM

 timezone: IST­5:30

     pmcd: Version 3.8.10­1, 6 agents

     pmda: pmcd proc xfs linux mmv jbd2

Configuration files:

Main configuration file for pcp:

/etc/pcp.conf 

Main configuration file for pmcd:

/etc/pcp/pmcd/pmcd.conf

Default pmlogger config file:

/etc/pcp/pmlogger/config.default

PCP archive logging configuration/control:

/etc/pcp/pmlogger/control

Note: pmlogger is utility to capture and store metrics exported by PMCD

Log files:

Log file directories for pcp components:

/var/log/pcp/pmcd

/var/log/pcp/pmie

/var/log/pcp/pmlogger

/var/log/pcp/pmmgr

/var/log/pcp/pmproxy

/var/log/pcp/pmwebd

Directory Organization for Archive Log Files

Performance monitoring tools:

➢ For all PCP monitoring tools, metrics values may come from a real-time feed (i.e. from pmcd on some host), or from an archive log.

➢ Performance monitoring tools available in Performance Co-Pilot (PCP).

pmstat

pmatop

pmcollectl

pmval

pmchart

pminfo

Performance monitoring tools:

pmstat - vmstat-like utility, intended to monitor system performance at the highest level.

# pmstat

# pmstat ­t 1 ­T 3

# pmstat ­a /var/log/pcp/pmlogger/localhost.localdomain/20140607.20.20.0 

Performance monitoring tools:

pmatop - top-like utility

#pmatop

Record Mode:pmatop ­w rawfile [ interval [ samples ]]

Playback Mode:pmatop ­r [ rawfile ] [­g|­m] [­L linelen] [­h host]

Eg:# pmatop ­w atop.out 1 4# pmatop ­r atop.out

Performance monitoring tools:

pmcollectl - System-level performance monitoring utility that records or displays specific operating system data for one or more sets of subsystems.

Basic System­level performance monitoring:# pmcollectl

Subsystem Specific performance monitoring:# pmcollectl ­sm# pmcollectl ­sm –verbose# pmcollectl ­smcdn –verbose

Record Mode:# pmcollectl ­f perf­data ­c 3 

Playback Mode: # pmcollectl ­p perf­data  

Performance monitoring tools:

pminfo - Display PCP metrics available on a host or in an archive.

Full list of all available metrics: # pminfo ­F       

Fetch and print values for all or specific metrics:# pminfo ­f# pminfo ­f <metric># pminfo ­F ­a /var/log/pcp/pmlogger/localhost.localdomain/20140607.20.56.0 

Eg:# pminfo ­f proc.nprocs disk.dev.read filesys.free

Descriptions of all or specific metrics:# pminfo ­T ­d # pminfo ­T ­d <metric>

Eg:# pminfo ­T ­d mem.util.cached# pminfo ­T ­d proc.nprocs disk.dev.read filesys.free

Performance monitoring tools:

pmval - Dumps the current values for the named performance metrics. # pmval <metric>

Eg:# pmval proc.nprocs# pmval kernel.all.load# pmval  ­t 2sec ­s 4 kernel.percpu.cpu.idle

Performance monitoring tools:

pmchart - GUI utility providing graphical display of PCP data

Knowledge sources:

Performance Co-Pilot (PCP)

http://oss.sgi.com/projects/pcp/

PCP Manual:

http://www.performancecopilot.org/pcp.git/man/html/index.html

Blog:

Exploratory Performance Analysis with Performance Co-Pilot

http://developerblog.redhat.com/2013/11/19/exploratory-performance-pcp/

Knowledge sources:

Videos:

Using Performance Co-Pilot to monitor SNMP devices - Hamish Coleman

https://www.youtube.com/watch?v=2azBcj8QUdI

Exploratory Performance Analysis with Performance Co-Pilot

https://www.youtube.com/watch?v=zrAjevr8_Ds

Regression Analysis with Performance Co-Pilot

https://www.youtube.com/watch?v=Z9fSymDfuvQ

Questions ?