RHUG SPECIAL SATELLITE - people.redhat.compeople.redhat.com/mlessard/mtl/presentations/fev2017/domtar.pdf · [RHUG Special Satellite, Feb 16, 2017] 2013 – Linux VM number is growing

[RHUG Special Satellite, Feb 16, 2017]

Presented by: Domtar Shared Services

Christophe Paulus

James Dubuisson

Gueorgui Tcherecharov

RHUG SPECIAL SATELLITE RED HAT OFFICE MONTREAL, FEBRUARY 16, 2017


DOMTAR IN BRIEF

150+ Years Old: From Dominion Tar and Chemical Company, Ltd. To Domtar Corporation

From coal tar distillation to NCC (nanocrystalline cellulose)

Domtar today is 2 Divisions: Pulp and Paper Personal Care

Close to 10,000 employees US $5.3 Billion (83% P&P – 17% PC)

Stock Symbol UFS (NYSE;TSE)

More info at domtar.com

STRICTLY PRIVATE AND CONFIDENTIAL 2

http://www.domtar.com/


THE SYSTEMS ADMINISTRATION TEAM

IT Core Infrastructure Services is responsible for the following technology towers: ■ Data Centers

■ Servers

■ SAN

■ Virtualization

■ Backup

■ Operating System

■ Application Delivery

Shared Services – Systems Administration ■ Windows

■ Linux/Unix

■ F5 Big-IP


Today presenters are part of IT Core Infrastructure Services Shared Services – Systems Administration


FEW NUMBERS

1700+ Virtual Machines spread across N.A and E.U, the bulk of them in the 2 Montreal Data Centers.

3 Main OSs in multiple flavors: ■ Windows

■ Linux

■ Unix (hp-ux)

200 RHEL VM (Growing): ■ RHEL5 (25%)

■ RHEL6 (66%)

■ RHEL7 (9%)

50+ Linux Based appliances (Growing Fast!)



2013 – Linux VM number is growing Fast

The management Start to become painful, ■ There is a need for a patching tool

■ Plus we want a replacement tool for the current configuration engine.

■ Need to automate VM deployment.

Evaluation of different products led to the conclusion that Red Hat Satellite is the tool. The key words/phrase was:

Greater administration consistency, enhanced security (compliance), increased productivity.

With Satellite all the goal could be achieved.

December 2013 - Starting deployment of Satellite 5.6 with external DB (Oracle)

WHY WE SELECTED SATELLITE



2014 – Happy Sysadmin who deploy patches ■ Per scheduled

■ According to our standard

Then Satellite 6 is presented : ■ The architecture, functionality and the road map were appealing to us.

■ Plus we only have implemented one of the need (patching).

Decision is made to switch to Satellite 6

2015 Starting of the Migration

Less Happy Sysadmin for a while

Happiness gets back with 6.1 and Joy with 6.2

WHY WE UPGRADED TO SATELLITE 6



DOMTAR PATCHING STANDARD

The goal: Patching every quarter

■ Patches that doesn't require a reboot (eg most of the security patch) every quarter

■ Patches that requires a reboot (e.g: Bug Fix and Product Enhancement) at least once with a twice a year target.

Application Life Cycle:

SandBox, Development, Quality Assurance, Staging (pre-production), Production, Training


Four Time a year we draw a line in the sand that contains all the available patches at that specific time.


DOMTAR PATCHING STANDARD CONT’D

Each quarter the set of patches is pushed thru the application Life cycle.



DOMTAR PATCHING STANDARD CONT’D

A content view is a set of repositories to which we have subscribed.

The following Domtar composite content-view are associated with the environments:

Those composite “content views” are build from based content view to which we have added our own repos.


dt-jboss-rhel6-x86_64 Library, jboss-sx, jboss-dv, jboss-qa, jboss-st, jboss-pr, jboss-tr

dt-rhel58-x86_64 Library, SX, DV, QA, ST, PR, TR




dt-sap-rhel5-x86_64 Library, sap_sx, sap-qa, sap-st, sap-pr, sap-tr, sap-dv

dt-sap-rhel6-x86_64 Library, sap_sx, sap-qa, sap-st, sap-pr, sap-tr, sap-dv

out-of-sequence-rhel6 Library, out-of-sequence-rhel6_Upgrade

out-of-sequence-rhel7 Library, out-of-sequence-rhel7_Upgrade


RATIONALES FOR UPGRADE TO SATELLITE 6 ?

Centralized method for managing servers

Easy way to deploy patches in phases ( Dev, QA, Staging, Prod…etc )

Simple way to manage our RedHat Subscriptions

RBAC for different teams

Configuration Management is integrated

Automatic Server deployment

Nice dashboard

Many more features in roadmap



SATELLITE 5 TO 6.0 MIGRATION

Started March 2015

Fresh Install on RHEL6 Server

Import data from Satellite 5 to Satellite 6

Issues when importing content views to new server

Virt-who installed on standalone

Ended up only importing users and host collections

Issues when selecting multiple patches

Some servers were losing their subscription during VMotion



MIGRATION SATELLITE 6.0 TO 6.1

Upgraded started September 2015

Issue with the upgrade

Was fixed via Remote session with RedHat Support

Lots of bugs were fixed compared to previous version

Not losing subscriptions anymore

Issue with RBAC

Improved speed



MIGRATION SATELLITE 6.1 TO 6.2

Started August 2016

Fresh Install

As per RedHat recommendations, we installed RHEL7

No official tool to export Satellite 6.1 content to 6.2 if you install on new machine at time of migration. A script is now available to do this task.

Imported data using Satellite API to new server

Much improved User Interface

Now using at 6.2.7 ; Works like a charm



HOW DO WE PATCH OUR SERVERS

Scheduling is done by our team

Operation team then contact server owners to get patching approval

When server owner approves patching window, Ops team will then patch them

After patches are down, we verify if all patches were completed successfully on our custom dashboard

After 2 weeks testing, we promote the same patches to the next environment using Hammer



HOW WE PATCH OUR SERVERS - SCHEDULING


A small shell script generates our patching schedule in a text file


HOW WE PATCH OUR SERVERS - OWNERS

We add the Business owner info in the description field for each server

Business owner approves the patches for his/her servers

Ops team uses the info in the description field to create a CR before patching

Below image shows content of this info : Env | Server | Application, Owner ; CC(users)



HOW WE PATCH OUR SERVERS - VERIFICATION


Screenshot of our custom Dashboard


HOW WE PATCH OUR SERVERS CONT’D


Info we can get on patches that were not installed, or when server owner wants info on patches to install


NICE FEATURES – SCHEDULING



NICE FEATURES – SCHEDULING CONT’D



NICE FEATURES – SCHEDULING CONT’D



NICE FEATURES - JOB EXECUTION



NICE FEATURES - JOB EXECUTION CONT’D



NICE FEATURES - OPENSCAP


Upload new SCAP content


NICE FEATURES – OPENSCAP CONT’D





Assign compliance policy to a host(s)




Assign the compliance policy





HOW WE USE THE SATELLITE API


https://satellite/apidoc/v2.html




HOW WE USE THE SATELLITE API CONT’D






API SCRIPT


Example: Simple Perl API Script


API SCRIPT


Example 2 : The same information but from all hosts




Export and import comment data

Apply security errata

List errata by server – details and summary

List server name, host collection, activation keys, subscription

List unsubscribed servers

List servers that changed their subscription

Here are some actions that we do using the APIs


FUTURE USE OF SATELLITE AT DOMTAR

Integrate new features that Michael just talked about

Ansible to replace our current Configuration Management tool

Deploy OpenScap, now that we have completed the POC

Deployment of servers from RedHat Satellite

Integrate IDM to Satellite



RECOMMENDATIONS

Use the recommended settings for Redhat Satellite ■ 16 Gigs Ram

■ 4 CPUs

■ Monitor the server to see if these settings fit your needs ( ~ 300 Servers )

■ Allocate file System size for file systems ( /var/pulp/ , /var/pgsql/, etc.. )

■ Create Capsules Servers for remote locations

■ No need to install virt-who on separate machine. Can now be integrated on Satellite main server



RECOMMENDATIONS CONT’D


Snapshot of or disk utilization in Satellite


ADDITIONAL RESOURCES

RedHat Satellite Blog ( Rich Jerrido – Technical Product Manager ) ■ http://access.redhat.com/blogs/1169563


http://access.redhat.com/blogs/1169563


ADDITIONAL RESOURCES CONT’D

RedHat Satellite Documentation ■ https://access.redhat.com/documention/en/red-hat-satellite/


https://access.redhat.com/documention/en/red-hat-satellite/







Hammer cheat sheet ■ https://access.redhat.com/articles/2258471


https://access.redhat.com/articles/2258471



API Description ■ https://yoursatellite.yourdomain.com/apidoc/v2.html



ADDITIONAL RESOURCES (SCRIPTS)



QUESTIONS



THANK YOU


Documents

RHUG SPECIAL SATELLITE - people.redhat.compeople.redhat.com/mlessard/mtl/presentations/fev2017/domtar.pdf · [RHUG Special Satellite, Feb 16, 2017] 2013 – Linux VM number is growing