JBoss in the Trenches - Red HatJBoss In The Trenches Red Hat's EAP6 Migration 250 Applications...

JBoss in the Trenches

Andrew BlockSenior Consultant, Red Hat

Tim BielawaSenior Release Engineer, Red Hat

JBoss In The Trenches

Presenters

Andrew Block

•2nd year presenting JBoss in the Trenches

•JBoss master

Tim Bielawa

•Emacs Guru

•Team “point man” during the EAP6 migration

•Has an actual Tux mascot tattoo (watch for it)

Session Topics

•Beginning the Migration

•Configuration Management

•Best Practices

•Production Support

•Advance to The Cloud

Red Hat's EAP6 Migration

250 Applications

One Year

Legacy EAP 4.3/5.1 Clusters

Dense to Sparse Clustering 50 Red Hatters

Legacy Infrastructure

Client initiatesREST request

iRule determines ifclient is allowed

Request balancedto a member of the cluster.

An iRule is a TCLscript running onthe BIG-IP device.

It's like iptableson steroids.

New Infrastructure

Migration - Timeline

ApplicationAnalysis

InitialPorting

OperationsConsulting

DeploymentScripting

MonitoringTooling

Release,Cutover

January December

IN THE BEGINNINGSTARTING THE MIGRATION

Application Analysis

What applications do we need to migrate?

What are the Application dependencies?

•Databases

•Messaging

•Other Applications

How are applications currently built, deployed and managed and how should they be in the future?

Common Migration Tasks

•Updating libraries

•Adding/updating container specific deployment descriptors

•Updating JNDI references

•Determine enterprise wide logging framework

What if there was a tool which could aid in this process?

Good news, there is: Windup

Windup

●Application migration tool●Can be run on source code or compiled archives●Generates reports

● Provides an overall level of effort in “Story Points”● Displays XML and Java hints and classifications to aid in migration effort

●Extensible

•New hints and classifications can be defined

Windup – Story Points

Windup - Classifications

Windup - Hints

CONFIGURATION MANAGEMENT

JBoss Management

How do we manage the JBoss platform?

How can we manage our configurations efficiently?

What tools can we use?

How can we manage our application deployments?

Core Tools

Puppet●IT configuration management tool

libeap6 (custom RHIT Internal)●Puppet module for managing and deploying EAP6

jcliff (custom tool, open source)●Tool for applying configurations to EAP6

jcliff In Brief

Begin End

Puppet installs or updates jcliff config files

jcliff compares expected vs. actual (live) configuration.

Applies diffs if necessary via jboss cli.

Notify admin if JBoss needs reloading.

Run Puppet

Artifact Workflow

Builder packages JARs in RPMs

UploadTo Pulp

JBoss Configuration Management

The management of platform configurations are as important as the configurations themselves

Configurations should be stored and versioned in a source control management system such as Git

Similar to applications, configurations should be deployed to each environment in an automated fashion

Externalize environment specific configuration values

JBoss Configuration Management (cont.)

Red Hat IT stores puppet modules in Git. Each repository has branches for each server environment

Git hooks are in place to ensure configurations have been merged into all lower environments before being promoted

We call this the parenting hook

BEST PRACTICES

Deployment

Deployments should be handled in an automated scripted process

Have an established rollback strategy

Prepare for the worst (pre-deployment):

• Snapshot server configurations and backup file systems

Can be a component of incremental builds in a Continuous Integration Environment

High Availability

Critical component of any infrastructure

Common methods: Clustering and Load Balancing

Enables Rolling Updates during deployments

Red Hat's HA Strategy

Pre-Migration(Dense Clustering)

Post-Migration(Sparse Clustering)

Many apps,single cluster

Many clusters,fewer apps on each

JBoss Platform Management

Web Management Console

User friendly web interface for managing basic JBoss configurations and runtime settings

Only exposes a subset of management capability

Command Line Tool (CLI)

Tool for interacting with the JBoss platform providing access to the full set of management operations

JBoss CLI

Uses a Detyped Management Model to execute operations

GUI component one of the best “Easter Eggs” of JBoss

$ jboss-cli.sh -c --gui

Public CLI API can be used to build robust applications● Can be written using Java, Groovy, Javascript, etc

JBoss Management Access

Limit access as you get closer to production

Migration Effort● Admin console/management interface disabled in production

● Configurations can only be initiated through Puppet/libeap6

• Limit access by use of Role Based Access Control (RBAC)

Role Based Access Control

New feature of JBoss EAP 6.2

Provides separation of duties among users

7 standard roles with varying access

Fine tune access to resources using sensitivity, vault, and application level constraints

PRODUCTION SUPPORT

Day-to-Day Operations

Question: How do you manage the day to day operations of the JBoss platform and the applications deployed?

Answer: Monitor the right resources:● Application● System● Thresholds

Production Support

Have a plan in place to support application and overall server infrastructure. We call ours the Pager Playbook

•Description and example of the issue

•Areas to diagnose

•Steps to resolve

•Escalation personnel

Management Tools

jconsole

Splunk

Naigos

Taboot

Custom groovy scripts for monitoring

JBoss Operations Network

Metrics Via Munin

What happened

Splunk – Old/New Cluster Requests ChartsEAP 6.1

EAP 5.1

Stats Via Splunk

93.6% came from monitoring!

Patch Management

EAP 6.2 contains new patch management feature

• Application of individual and cumulative patches from Red Hat

• Simplified patching strategy from previous releases

Executing through the CLI tool● Ability patches● View applied patches● Rollback patches

ADVANCE TO THE CLOUD

Advancing Towards a Cloud Infrastructure

IaaS PaaSIaaS

●Offers technological and financial flexibility

●Additional considerations must be accounted for:

• Limitations

• Security●Leverage additional tools while

reusing the migration toolset●Endless possibilities!

WRAP UP

Bryan Madaras

A “THANKS!” of considerable

magnitude goes out to Bryan Madaras

for his efforts preparing this session

with us and for his machine-like focus

during IT’s migration.

Download this Slide Deck!

http://people.redhat.com/~tbielawa/summit2014/

Most slides have additional notes with links to any referenced material

Thank You!

Andrew Block

andrew.block@redhat.com

@sabre1041

http://blog.andyserver.com

Tim Bielawa

tbielawa@redhat.com

@tbielawa

https://blog.lnx.cx

Questions?

JBoss in the Trenches

Andrew BlockSenior Consultant, Red Hat

Tim BielawaSenior Release Engineer, Red Hat

Presenters

Andrew Block

•2nd year presenting JBoss in the Trenches

•JBoss master

Tim Bielawa

•Emacs Guru

•Team “point man” during the EAP6 migration

•Has an actual Tux mascot tattoo (watch for it)

SHARED

http://www.redhat.com/summit/speakers/session.html#tbielawa

http://www.redhat.com/summit/speakers/session.html#ablock

Tim:● Avid Blogger of technical things● Open Source contributor/maintainer for a decade● Writing a book● Co-creator of a scholarship at WVU to encourage

young adults to become involved in open source.

See also: “Thank You!” at the end of this presentation.

Thanks for joining us!

Over the next hour we're going to move through the story of how Red Hat IT migrated legacy 4.3/5.1 clusters to EAP6.1

Along the way we'll be identifying important implementation decisions, as well as best practices.

And now Andy's going to explain what that covers in greater detail.

Session Topics

•Beginning the Migration

•Configuration Management

•Best Practices

•Production Support

•Advance to The Cloud

Red Hat's EAP6 Migration

250 Applications

One Year

Legacy EAP 4.3/5.1 Clusters

Dense to Sparse Clustering 50 Red Hatters

TIM - CLICK THORUGH

Before we continue I want to give you an idea of the scope of the migration project, beginning with explaining why it even happened at all.

Why Migrated: We were running unsupported versions of JBoss in production.

How much migrated: Rough estimate of 250 apps.

What changed: Cluster with many apps turned into more clusters with fewer apps.

How many people: Nearly every team in IT was involved. 3-4 development towers, project management office, sysadmins, production control, networking, qa testing..

How much time: Took about a year.

Legacy Infrastructure

Client initiatesREST request

iRule determines ifclient is allowed

Request balancedto a member of the cluster.

An iRule is a TCLscript running onthe BIG-IP device.

It's like iptableson steroids.

Services call out to a known REST/SOAP endpoint.

Load balanced. Originally via mod_jk + httpd, now via F5 balancer pools.

services-security is an F5 iRule that operates as a firewall of sorts.

If you were allowed through the firewall then you were balanced to a member of the pool.

All services were hosted on the same cluster.

New Infrastructure

Similar to the original setup, however we have two routing points involved now.

Top is the ‘director’, basically an httpd ProxyPass. It routes traffic by inspecting the incoming request to determine the specific service endpoint requested and then forwards your request to the correct cluster.

Now services are partitioned into three separate clusters.

We kept the “Old SOA VIP” around during the migration as a catch-all. If a service wasn’t ported to EAP6 yet, then the request was directed back to the 5.1 cluster.

Migration - Timeline

ApplicationAnalysis

InitialPorting

OperationsConsulting

DeploymentScripting

MonitoringTooling

Release,Cutover

January December

CLICK THROUGH

IN THE BEGINNINGSTARTING THE MIGRATION

Application Analysis

What applications do we need to migrate?

What are the Application dependencies?

•Databases

•Messaging

•Other Applications

How are applications currently built, deployed and managed and how should they be in the future?

ANDYIntroduction slide to application analysis section

* What applications fall under the scope of the migration effort?

How are the applications currently being built?* Ant/Maven?

How are the applications currently being deployed?* Manually* Scripted process* Automated?

How are the applications managed?* Artifact repository?* Package manager?

Common Migration Tasks

•Updating libraries

•Adding/updating container specific deployment descriptors

•Updating JNDI references

•Determine enterprise wide logging framework

What if there was a tool which could aid in this process?

Good news, there is: Windup

ANDY - CLICK THROUGH

* Libraries include core JDK/JBoss libraries but also 3rd party libraries which may no longer be supported by the JDK or JBoss. * Deployment descriptors have evolved through JBoss versions. These must be updated to support the most recent container * jboss-classloading.xml -> jboss-deployment-structure.xml * JNDI - java:/TransactionManager -> java:/jboss/TransactionManager* Various logging frameworks exist. Being able to determine the particular logging framework to utilize up front will allow for a single strategy moving forward

Windup

●Application migration tool●Can be run on source code or compiled archives●Generates reports

● Provides an overall level of effort in “Story Points”● Displays XML and Java hints and classifications to aid in migration effort

●Extensible•New hints and classifications can be defined

ANDY -

* Story point is a unit of work. It can equate to 1 hour of work in a shop with senior resources, but can vary depending on the skill set* Hint can alert the developer that a modification may be required as part of a migration effort* Classifications are ways of identifying a type of file (eg: jboss-ejb3.xml = JBoss EJB descriptor)

Windup also features support for users to define their own set of hints and classifications. This can be useful when teams are migrating several applications and want t

Windup – Story Points

ANDY - When you run the windup tool, it generates a report to help you migrate your application. When viewing the report, you are presented with an overall diagram of what the migration entails. The story points are on the left as previously discussed. On the right is a depiction of the classes and number of references in which they appear in the classes within the application.

Windup - Classifications

ANDY - Classifications will appear as informative queues above certain files containing steps required for successful migration. In this example of a jboss.xml file, the classification describes the usage for this file and the requirement to update this file to a jboss-ejb3.xml specification.

Windup - Hints

ANDY -

Hints will appear in Java and XML classes as potential places which require migrations

CONFIGURATION MANAGEMENT

JBoss Management

How do we manage the JBoss platform?

How can we manage our configurations efficiently?

What tools can we use?

How can we manage our application deployments?

ANDY - CLICK THROUGH 4x

SEVERAL AREAS OF CONSIDERATION

For the migration effort, we need to begin to think how we would go about managing the JBoss platform * In previous versions of JBoss, you would utilize JMX/twiddle to view statistics on the container and applications

These are the critical questions which the migration team pondered and lead to the development of several core tools.

SEGUE: .... Which Tim will describe .

Core Tools

Puppet●IT configuration management tool

libeap6 (custom RHIT Internal)●Puppet module for managing and deploying EAP6

jcliff (custom tool, open source)●Tool for applying configurations to EAP6

Internet Resources:

• Puppet: http://puppetlabs.com/

• Libeap6 is currently an internal only module. There are no active plans to open it up at this time.

• Jcliff: https://github.com/bserdar/jcliff/

jcliff In Brief

Begin End

Puppet installs or updates jcliff config files

jcliff compares expected vs. actual (live) configuration.

Applies diffs if necessary via jboss cli.

Notify admin if JBoss needs reloading.

TIM - 1. Puppet run initiated2. Puppet determines is any configuration has been modified since last run via puppet managed config files in a format understood by jcliff, these files are managed by types in libeap6 so they can be controlled3. jcliff queries jboss using the cli and does comparison4. decision point if updated needs to happen5. jcliff has configurations files to match differences it finds and then configurations to create cli commands to rectify those differences using input from the found difference, then executes the cli commands to update the configuration. FYI they do have ordering of commands if anyone asks6. Puppet queries the state of jboss via cli to determine if it needs to be reloaded7. decision if it needs to notify anyone8. Notify via nagios (Tim check on this)

Run Puppet

Artifact Workflow

Builder packages JARs in RPMs

UploadTo Pulp

CLICK THROUGH - 3x

Pulp– Repository manager

JBoss Configuration Management

The management of platform configurations are as important as the configurations themselves

Configurations should be stored and versioned in a source control management system such as Git

Similar to applications, configurations should be deployed to each environment in an automated fashion

Externalize environment specific configuration values

ANDY - Configuration management is just important as the management of your applications.Jboss EAP 6 has consolidated the configurations from previous versions The key is to ensure each configuration is properly versioned to allow changes to be efficiently tracked and to allow for a rollback of previous configurations. The application of these configurations to the various servers, like the deployment archives themselves, should be handled in an automated fashion.

Externalize as many environment specific configuration variables into system properties or stored in properties files which can be loaded by the JBoss server at startup. These would also include sensitive resources such as passwords to provide another layer of abstraction (datasources, login modules). This provides a standard configuration file across multiple environments

JBoss Configuration Management (cont.)

Red Hat IT stores puppet modules in Git. Each repository has branches for each server environment

Git hooks are in place to ensure configurations have been merged into all lower environments before being promoted

We call this the parenting hook

Rollback strategy:

Revert back to the original jboss config (from installation time).

Revert the configurations stored in the puppet modules. Which are in git.

Reapply the previous configuration state via puppet+jcliff.

Up next: General Best practices

BEST PRACTICES

Deployment

Deployments should be handled in an automated scripted process

Have an established rollback strategy

Prepare for the worst (pre-deployment):

• Snapshot server configurations and backup file systems

Can be a component of incremental builds in a Continuous Integration Environment

ANDY - Numerous tools can be used to leverage the deployment process. From as simple as bash/batch files to an entire orchestration process such as the aforementioned automation with puppet:* Artifacts should never be manually deployed to servers

* Snapshots can be taken with the CLI tool. * :take-snapshot

* A Continuous Integration tool such as Jenkins can perform a deployment (especially in a lower environment and on feature branches) to shorten the development lifecycle

High Availability

Critical component of any infrastructure

Common methods: Clustering and Load Balancing

Enables Rolling Updates during deployments

ANDY - Critical piece in any infrastructure to ensure services are readily available

High availability centers around ensuring services achieve a prearranged level of operational performance and availability. Red Hat IT has several service levels it must meet from service to service.

As part of the migration effort, several JBoss components related to high availability were configured. The biggest change during the migration involved load balancing of the services (SOA) tier.

JBoss features support for both software and hardware based load balancers

JBoss traditionally leverages Apache httpd to serve as a front end load balancer to effectively route web traffic to a cluster of JBoss servers. In versions prior to EAP 6, mod_jk was the default and preferred module. In EAP 6, the mod_clustermodule became the default implementation, as it provides advantages over other connectors. * Dynamic configurations of HTTP proxies which allows EAP to adapt on the fly without additional configurations* Advanced server side load balancing calculations through the use of a separate communication channel (Mod-Cluster Management Protocol [MCMP]) not found in other connectors * Fine grained control of web applications by dynamically enabling and disabling contexts without resulting in 404* AJP protocol is optional. Support for http, https

Red Hat's HA Strategy

Pre-Migration(Dense Clustering)

Post-Migration(Sparse Clustering)

Many apps,single cluster

Many clusters,fewer apps on each

TIM - As part of the migration effort, several JBoss components related to high availability were configured. The biggest change during the migration involved load balancing of the services (SOA) tier.

In the dense scenario, there is no need for location-based pool routing. All contexts were served from the same cluster.

However, in the sparse scenario, not all contexts are available on every cluster. Therefore a director is necessary.

This director component lives between the requesting client and each cluster load balancer.

See New Infrastructure slide 7 for a more detailed diagram.

JBoss Platform Management

Web Management Console

User friendly web interface for managing basic JBoss configurations and runtime settings

Only exposes a subset of management capability

Command Line Tool (CLI)

Tool for interacting with the JBoss platform providing access to the full set of management operations

ANDY - Before discussing any specific implementations which might configure or manage the JBoss platform, lets discuss the most common methods of interacting with the JBoss platform

* Web Administration Console * The Web Administration Console is a GWT based user interface for interfacing with the JBoss platform. It fe

* CLI Tool* Introduced in EAP 6. Exposes the full set of

functionality of the management API

JBoss CLI

Uses a Detyped Management Model to execute operations

GUI component one of the best “Easter Eggs” of JBoss

$ jboss-cli.sh -c --gui

Public CLI API can be used to build robust applications● Can be written using Java, Groovy, Javascript, etc

ANDY - The API is described as "detyped" to reflect the fact that the API only exposes a small number of types besides JDK classes.

Red Hat IT created several application and tools for configuring, managing and monitoring servers. More on that later though.

SEGUE: <andy> CLI gives you great power, however with great power</andy>

CUT TO TIM : <tim> <click to spiderman /> comes great responsibility <click to access mgmt /></tim>

Comes great responsibility.

JBoss Management Access

Limit access as you get closer to production

Migration Effort● Admin console/management interface disabled in production

● Configurations can only be initiated through Puppet/libeap6

JBoss• Limit access by use of Role Based Access Control (RBAC)

As soon as code leaves DEV and enters the managed environments (qa, stage, prod) access is restricted to system operators.

Also, the admin console/mgmt interface is disabled.All application server configuration takes place strictly through puppet and jcliff.

Next Andy will tell you some more about RBAC.

Role Based Access Control

New feature of JBoss EAP 6.2

Provides separation of duties among users

7 standard roles with varying access

Fine tune access to resources using sensitivity, vault, and application level constraints

ANDY - RBAC Roles --- PLUS YOU CAN DEFINE YOUR OWN* Monitor* Operator* Maintainer* Administrator* SuperUser* Deployer* Auditor

Constraints* Sensitivity (attribute granulatiry) – Defines a set of resources which are considered sensitive. Can be a secret value (password), networking, JVM or System Property* Vault - Reading and writing to the security vault* Application - Set of resources that can be accessed by those in the deployer role. Can provide additional functionality to support the applications they serve

PRODUCTION SUPPORT

And now Tim will tell us about day-to-day operations.

Day-to-Day Operations

Question: How do you manage the day to day operations of the JBoss platform and the applications deployed?

Answer: Monitor the right resources:● Application● System● Thresholds

TIM - CLICK THROUGH x4

After applications are up and running, you need to make sure that it continues to function

Application – Any application specific resourcesSystem – JVM, CPU, SWAP etcThresholds – Determine the list of thresholds for both your applications, application servers and operation system

Production Support

Have a plan in place to support application and overall server infrastructure. We call ours the Pager Playbook

•Description and example of the issue

•Areas to diagnose

•Steps to resolve

•Escalation personnel

Management Tools

jconsole

Splunk

Naigos

Taboot

Custom groovy scripts for monitoring

JBoss Operations Network

* jconsole in the <EAP_HOME>/bin features a traditional jconsole but also includes the CLI GUI tool. 2 for the price of 1!* JBoss Operations Network * Monitoring * Deployment * Drift Management

Groovy scripts supply the information needed by nagios to determine health, and the metrics for munin to create charts from

Metrics Via Munin

What happened

Example of a munin chart showing SOAP access times per service (in miliseconds).

Something looks off though

Splunk – Old/New Cluster Requests ChartsEAP 6.1

EAP 5.1

This slide shows two custom splunk queries.

The top chart shows the number of requests to the new EAP6 Services cluster since the cut over.

The bottom chart shows the number of requests to the legacy Services cluster since the cut over.

Unfortunately, it seems that after the cutover the legacy cluster is still receiving requests. But from where? And for which contexts?

Stats Via Splunk

93.6% came from monitoring!

Still using splunk, we can “drill down” into this data and identify exactly which contexts are receiving the most requests on the deprecated cluster.

We went even further than that still and broke each context request down by requesting client IP address.

Patch Management

EAP 6.2 contains new patch management feature

• Application of individual and cumulative patches from Red Hat

• Simplified patching strategy from previous releases

Executing through the CLI tool● Ability patches● View applied patches● Rollback patches

ANDY -

Ensuring the JBoss server is kept up to date is essential and a component of system administration

No longer are you required to manually extract and apply patches to update the JBoss platform * patch apply –patch <path>* patch info * patch history* patch rollback

Because patch mgmt is CLI based it can easily be integrated into automated deployments and maintenance.

ADVANCE TO THE CLOUD

IaaS PaaSIaaS

ANDY -

When thinking about Jboss in a cloud environment, there are several areas and options to consider

●Offers technological and financial flexibility

●Additional considerations must be accounted for:

• Limitations

• Security●Leverage additional tools while

reusing the migration toolset●Endless possibilities!

ANDY -

Red Hat internally has been driving to migrate a number of its services to the cloud platform

Limitations – UDP traditionally not available

In AWS there are Security groups – Additional level of consideration and complexity that must be accounted for

Tools that you currently use to configure and manage JBoss can be reused in the cloud. Migration related tools such as libeap6 and jcliff can still be utilized

WRAP UP

Bryan Madaras

A “THANKS!” of considerable

magnitude goes out to Bryan Madaras

for his efforts preparing this session

with us and for his machine-like focus

during IT’s migration.

Download this Slide Deck!

Most slides have additional notes with links to any referenced material

Thank You!

Andrew Block

andrew.block@redhat.com

@sabre1041

http://blog.andyserver.com

Tim Bielawa

tbielawa@redhat.com

@tbielawa

https://blog.lnx.cx

Questions?

JBoss in the Trenches - Red HatJBoss In The Trenches Red Hat's EAP6 Migration 250 Applications...

Documents

Rewolucje w JBoss EAP6 oraz JON 3.0 jako strażnik infrastruktury.pdf

Trenches Lesson

Automation from the trenches

Symfony2 - from the trenches

the Trenches - AZBOCUG

To the Trenches

From The Trenches

Powerpoint trenches

EAP6 performance Tuning

V FROM TRENCHES

Training & Trenches FINAL

Trenches Around World

Life in the Trenches. What are trenches? Trenches are heavily guarded ditches dug into the ground. The trenches used during WWI were the depth of just

EU4UKR EaP6 Promotion Project

From The Trenches (Australia)

Lesson ppt trenches

Trenches Project

PIPE BEDDING IN TRENCHES

EAP6 Presentation - "The Savoy"

Life in the trenches