Embed Size (px)
Citation preview
TAKE CONTROL OF LOGS WITH ELASTICSEARCH
AGENDA
• Benefits of Collec;ng Log Data
• Why Use Elas;csearch (and the Elas;c Stack)
• Using the Elas;c Stack to Collect Logs
• Learning about your System
Why Collect Log Data?
WHAT LOGS TELL US
• Local error messages
• Isolated data
• Point-in-;me data
LOG SOURCES
• PIA_Access
• Servlet Logs (IDDA)
• APPSRV
• Anything you want
BENEFITS
• Search for log data in one loca;on
• Correlate events between servers
• See trends in log events
• Collect data for future decisions
• Capture overall system health
• Make preSy graphs!
Why Elasticsearch?
ELASTICSEARCH ADVANTAGES
• Control your own data
• You already (or will soon) use Elas;csearch
• Collect only the data you want
• Does not require PeopleSoW Performance Monitor
• Collect more than PeopleSoW data
• Easily build your own metrics
• Free and open source
ELASTICSEARCH VERSIONS
• PeopleTools uses Elas;csearch 2.3.2 • Kibana 4.5, Logstash 2.4
• Elas;c Stack 5.3 is latest
• Can I use my PeopleSoW Elas;c install for logs? • orcl_acl plugin breaks Kibana
• https://community.oracle.com/ideas/16330
• License Restrictions
Using the Elastic Stack
ELASTIC STACK ARCHITECTURE
Enrich Index SearchCollect
INSTALL ELASTIC STACK
Elas%csearch 1. Download Elasticsearch
2. Unzip Elasticsearch
3. Run bin\elasticsearch.bat
Kibana 1. Download Kibana
2. Unzip Kibana
3. Run bin\kibana.bat
INSTALL ELASTIC STACK
Logstash 1. Download Logstash
2. Unzip Logstash
3. Configure Filters
4. Run bin\logstash.bat
INSTALL ELASTIC STACK
Filebeat 1. Download Filebeat
2. Unzip Filebeat
3. Configure Files and Output
4. Run bin\filebeat.bat
Topbeat/Metricbeat 1. Download Topbeat
2. Unzip Topbeat
3. Configure Output
4. Run bin\topbeat.bat
INSTALL FILEBEAT
• Install Filebeat with Puppetpuppet module install pcfens-filebeat
• Configure Logstash output with Puppet class { 'filebeat': outputs => { 'logstash' => { 'hosts' => [ 'elastic.psadmin.io:5044', ], }, }, }
CONFIGURE FILEBEAT
filebeat: prospectors: - paths: - /psoft/logs/HRAPP003P/PIA/PIA_access.log fields: domain: hr92prd server_type: webApp region: PRD host: hrapp003p input_type: log document_type: access_log scan_frequency: 10s tail_files: true
CONFIGURE FILEBEAT
$pia_domain_list.each |$domain_name, $pia_domain_info| { filebeat::prospector {"${domain_name}-web": paths => [ "${pia_domain_info['ps_cfg_home_dir']}/webserv/${domain_name}/servers/PIA/logs/PIA_access.log", ], doc_type => 'access_log', input_type => 'log', ignore_older => '24h', fields_under_root => 'true', tail_files => 'true', fields => { domain => "${domain_name}", server_type => hiera('server_type'), region => hiera('region'), } } }
LOGSTASH CONFIGURATION
• Give structure to incoming data
• Collect, Enrich, Transport • input{} • filter{} • output{}
CONFIGURE LOGSTASH
input { beats { type => beats port => 5044 } } filter { } output { elasticsearch { hosts => [“elastic.psadmin.io:9200”] workers => 4 index => "logstash-%{+YYYY.MM.dd}" } }
LOGSTASH FILTERS
filter { grok { match => { "message" => “%{WL_IO_EXTENDED}"} } grok { match => { "request" => “%{PS_URI_REQUEST}"} } date { match => [ "timestamp", "MMM dd yyyy HH:mm:ss","MMM d yyyy HH:mm:ss", "ISO8601" ] } useragent { source => “useragent”, target => “agent”
} }
GROK EXPRESSIONS
/psc/hr92prd/EMPLOYEE/HRMS/c/ROLE_MANAGER.TL_MSS_EE_SRCH_PRD.GBL
PS_URI_REQUEST %{WORD:servlet}(/%{WORD:site_name})?(/%{WORD:portal}/%{WORD:node}/)?(%{WORD:content_type}/(%{PS_WEBLIB:iscript}|%{WORD:menu}\.%{PS_COMPONENT:component})?)?(\?%{GREEDYDATA:query_string})?
PS_COMPONENT %{WORD:componentName}\.%{WORD:market}
hSp://grokdebug.herokuapp.com
Explore Log Data!
OPERATIONS DASHBOARD
OPERATIONS DASHBOARD
OPERATIONS DASHBOARD
OPERATIONS DASHBOARD
TRAFFIC DASHBOARD
TRAFFIC DASHBOARD
SERVER DASHBOARD
SERVER DASHBOARD
SERVER DASHBOARD
SPONTANEOUS STRESS TEST
SHARE DATA
LOG CORRELATION
LOG CORRELATION
DATA RETENTION
• Elas;c Curator
• Define reten;on ;mes per index • curator --host servername --port 9200 delete indices --older-than 21 --time-unit days
--timestring %%Y.%%m.%%d
PleaseCompleteYour SessionEvaluationEvaluatethissessioninyourCOLLABORATEapp.Pullupthissessionandtap"SessionEvaluation" tocompletethesurvey.
SessionID: 100570
