Embed Size (px)
Citation preview
Every business is regulated…
• Labor regs minimum wage, paid sick leave, hours and breaks
• Money regs income tax withholding, accoun%ng prac%ces (SOX)
• Safety regs protec%ve equipment, training, repor%ng accidents
• Licensing regs business license, HAZMAT, serving liquor
“Highly Regulated”
Ac#ve Monitoring Level of Detail of regs
as it pertains to system administra#on
Ac#ve Monitoring Level of Detail of regs
Is a Policy in place? Are Procedures to implement that in place? Do employees receive Training on P&P? Can you Prove that P&P are followed?
• Separa%on of du%es • Data access • System access %meouts • Least privilege • Passwords
• “Passwords shall be at least eight characters in length, and shall include at least one uppercase character, one lowercase character, one numeral, and one special character.”
Ac#ve Monitoring Level of Detail of regs
Who Is The Boss?
FERC: Federal Energy Regulatory Commission
and its designee NERC: North American Electric Reliability Corp.
United States Constitution Art. 1, Sec. 8 “to regulate commerce among the several states”
to FERC to Congress
to NERC
Do this, or else. Or else what? $$ Fines, baby… fines.
Power Flow
Power Surge
• Used to be that NERC made sugges%ons only
• As electric power suppliers were deregulated, the need for predictable delivery increased
• In 2006, FERC designated NERC as the na%onal ‘Electric Reliability Organiza%on’
• NERC’s sugges%ons are now Standards.
How Can Companies Get On Track?
Obviously all these NERC P&Ps will massively increase produc%vity…. or not So how do we deal with the new strictures?
à We need a framework!
Anybody got one?
Coincidentally, on a Parallel Track… Aber the … excesses … of the dot-‐com era, the business side wanted to rein in IT
Information Technology Infrastructure Library (ITIL)
Two Tracks Align
• The FERC Reliability Standards, plus • The MBAs’ counteradack on Techies gave us CHANGE MANAGEMENT *
Change Management
The objective of change management ... is to ensure that standardized methods and procedures are used for efficient and prompt handling of all changes to control IT infrastructure, in order to minimize the number and impact of any related incidents upon service.
from the wikipedia ar#cle
Simplified Example of Change Flow 1. Sysadmin writes proposal for new sehng 2. Different sysadmin or manager agrees 3. Sysdmin becomes Change Owner, engages CM tool:
Describe business effects of doing/not doing Iden%fy systems/services/apps/users affected Design the procedure (including verifica%on and backout plan) Design and execute a test plan
4. Change Owner schedules %me for change 5. Every affected IT group assesses change impact 6. Every affected system/service/app/user reviews change and authorizes 7. Change Board considers all imminent changes, weighs risks and
conflicts, approves change for implementa%on 8. Change Owner executes procedure at scheduled %me 9. Change Owner completes change record
And that's just for the kiddie systems
The systems handling the power grid proper are a whole 'nuther animal.
Ques%ons? Correc%ons?
Why do we puppet the way we do?
Power
Accountability
History
Why do we puppet the way we do?
Power
Accountability
History
Why do we puppet the way we do?
Power
Accountability
History
puppet db
PGE – PEC (Puppet Enterprise Components)
database
webservice
puppet master
hdp
puppet console DB webservice prd tst
dev unk
All three are VMs, 2 cores/8GB ram, RHEL
PGE – Puppet Environments
Every node is in one and only one environment. The puppetmaster has three parallel directory structures: /etc/puppetlabs/puppet/environments/[dev|tst|prd]
The directories are all clones of a single git repo, and pull from that remote repo for manifest and module updates.
prd
tst
dev
unk
PGE -‐ Promo%on
dev – 80 or so systems deploy, then watch puppet reports to verify what’s changing... and that things don’t keep changing. tst – around 100 systems Collect ‘test results’ for inclusion in the Change
prd
tst
dev
change management bar
prd – around 120 systems Several new or revised modules are promoted as a single change – an ‘OS Release’
PGE -‐ Keeping Tabs With Custom Facts
• third-‐party sobware
• locate inconsistency
• feed our manifests and templates
PGE -‐ Custom Facts Defined # synergy_status.rb Facter.add("synergy_installed") do
setcode do File.executable?("/usr/bin/syninfo")
end
end
Facter.add("synergy_joined") do
confine :synergy_installed => true setcode do
domain = Facter::Util::Resolution.exec(‘syninfo --domain')
domain.eql?(“it.pgn.com")
end
end
Facter.add("synergy_status") do
setcode do
if Facter.value(:synergy_installed)
if Facter.value(:synergy_joined)
Facter::Util::Resolution.exec(‘syninfo --mode')
else
"Not_Joined" end
else
"Not_Installed"
end
end
end
PGE -‐ Custom Facts Realized
synergy_installed => true synergy_joined => true
synergy_status => connected
These are reportable/searchable via PuppetDB.
PGE -‐ Custom Facts Available #!/bin/bash
FACT=$1
VALUE=$2
curl -X GET -H "Accept: application/json" \ --cacert /home/marinus/puppetInventory/ca.pem \
--cert /home/marinus/puppetInventory/cert.pem \
--key /home/marinus/puppetInventory/private.pem \ 'https://puppetdb:8081/v2/facts/'${FACT} \
--data-urlencode 'query=["not", ["=", "value", "'${VALUE}'"]]'
Just show me systems where Synergy is not ‘connected’: /facts_without_value.sh synergy_status connected
PGE -‐ Really Simple Modules
• A few module-‐level variables probably set from facts or literals, not computed
• A File resource usually a .conf file
• A Service resource subscribed to the file resource
PGE -‐ Really Similar Modules
• If you’ve seen one, you’ve seen ‘em all
• Every file’s content comes from a template even if there’s no variability
• puppet-‐lint helps us enforce textual appearance
PGE – Common Module Layout class synergy { if $::synergy_installed != 'true' {
warning('This node does not have Synergy installed') } else {
$os = $::operatingsystem
$filegroup = $os ? {
/AIX/ => 'system',
/RedHat/ => 'root',
default => 'unk', }
File {
ensure => file,
mode => '0644',
owner => 'root',
group => $filegroup, }
file { '/etc/synergy/gid.ignore':
content => template ("synergy/gid.ignore.${os}.erb"),
}
file { '/etc/synergy/synergy.conf':
content => template ("synergy/synergy.conf.${os}.erb"),
} service { 'synergy':
ensure => running,
enable => true,
subscribe => File['/etc/synergy/synergy.conf'],
}
} }
coda
Puppet Enterprise gives us Power
lets us deal with our History eases Accountability
Marinus Damm [email protected]
Salem
Oregon City
Estacada
Hubbard
St. Paul
Sheridan
Willamina
Grand Ronde
Zigzag
Government Camp
Brightwood
Sandy
GreshamPortland
Fairview
Woodburn
HillsboroForest GroveCornelius
Banks North Plains
Scapoose
St. Helens
5
5
5
5
8484
84
99E
99E
99E
99E
213
217
213
213
214
214
214
219
213
219
219
210
211
221
221
211
211
211
212
224
224
47
47
47
43
18
22
22
18
8
8
10
10
99W
99W
99W
99W
405
205
205
Aurora
CanbyBarlow
Mt. Angel
Keizer
Gervais
Willsonville
Tualatin
Silverton
Marquam
Scotts Mills
Molalla
Mulino
Colton
NewbergDundee
Yamhill
Carlton
Gaston
Scholls
Lafayette
DaytonMcMinnville
Amity
Turner
Beaverton
King CityLake Oswego
Tigard
Rivergrove
Johnson City
West LinnCarver
Happy Valley
Boring
Eagle Creek
Milwaukie
Troutdale
Wood Village
26
26
26
26
30
30
30
26
26
26
PGE SERVICE TERRITORY
Washington
Columbia
Multnomah
Yamhill
Clackamas
Marion
Polk
HOOD RIVER CO
WASCO CO
HOOD RIVER CO
MULTNOMAH CO
MULTNOMAH COCLACKAMAS CO
WASHINGTON CO
YAMHILL CO
YAMHILL CO
YAMHILL CO
MARIO
N CO
CL
ACKAM
AS CO
MA RION
CO
POLK COUNTY
YAMHILL COUNTY
MARION COUNTY
CLACKAMAS COUNTY
WASHINGTON COUNTY MULTNOMAH COUNTY
Counties • About a million points of delivery • 1400 servers (Windows & UNIX) • Sixty people in IT Infrastructure
… and nice benefits
PGE Service Territory
