Training Webinar System Operations: the Java side

System Operations: the Java side(part 2)

System Operations: the Java side (part 2)

Recap

● OutSystems Platform Architecture○ Platform Services○ 1 Click Publish○ Standalone vs Farm Environments

● OutSystems Platform Licensing○ Activation Code○ Serial Number○ Intellectual Property Protection (IPP)

● Installation & Configuration○ Versioning○ Checklist○ Configuration Tool


Farm installation

● What is different in the farm installation?○ Multiple machines work together to serve or deploy the applications - multiple

Front-ends;○ Only one Deployment Controller – Deployment Controller Service is only

running on one machine;○ Start by configuring the Deployment Controller, then export the configuration file

to be imported in the Front-Ends

● What about adding a front-end to an existing farm?○ Applications are automatically deployed the moment Deployment Service is

started;○ Same installation procedure – same base server.hsconf


Farm installation

● What is different in the farm installation?

Controller address

Front End address


Farm installation

● Farm installation overview○ Similar procedure to standalone, having in mind the profile of each machine○ License, Service Center, System Components etc. need only be published to the

first Front-End machine

● Differences from Standalone installation○ In Configuration Tool:

■ Configure in Deployment Controller machine first■ Deployment Controller IP - not localhost anymore!■ Copy resulting server.hsconf file■ On every other machine, paste server.hsconf before running Configuration

Tool


Configuration

● Load Balancers○ Up to Layer 7 Load Balancers○ Most common balancing algorithms (round-robin, least connections, fastest

HTTP response, etc …)○ Both sticky or non-sticky sessions persistent due to the shared session data

model○ HTTP to HTTPS tunneling

● Compressed content (static and dynamic)

● Content caching (expiration headers)○ From 9.1 on, JBoss/WildFly stack only○ Cache suffix for static contents


Farm installation

● Network Requirements (connectivity)Publishing and Runtime

Source Destination Port Protocol Notes

Controller Oracle / MySQL 1521/3306 TCP Database Connection

Controller Front-End 12001 TCP OutSystems Deployment Service

Front-End Oracle / MySQL 1521/3306 TCP Database Connection

Front-End Controller 12000 TCP OutSystems Deployment Controller Service

Controller Front-End 2033 TCP RMI (needed for OutSystems Services)

Front-End Controller 2033 TCP RMI (needed for OutSystems Services)

Controller Front-End 5556 TCP Node Manager (WebLogic only)

Front-End Controller 7001 & 7002 TCP Admin Server (WebLogic only)

External Users DMZ Front-End 80 TCP Application's HTTP access

External Users DMZ Front-End 443 TCP Application's HTTPS access

Internal Users Internal Front-End 80 TCP Application's HTTP access

Internal Users Internal Front-End 443 TCP Application's HTTPS access


Farm installation

● Network Requirements (connectivity)Monitoring

Source Destination Port Protocol Notes

Front-End Controller 2033 & 12000 TCP RMI/OutSystems Deployment Controller Service

Front-End Controller 2033 & 12003 TCP RMI/OutSystems Log Service

Front-End Front-End 80 TCP Application Server

Front-End Front-End 2033 & 12001 TCP RMI/OutSystems Deployment Service

Front-End Front-End 2033 & 12002 TCP RMI/OutSystems Scheduler Service

Front-End Front-End 2033 & 12003 TCP RMI/OutSystems Log Service


Connectivity clarification

● Jboss/WildFly/WebLogic are normally configured to listen to ports 8080 and 8443○Instead of the default Web 80 and 443 ports○Reason: Security limitation - Linux restricts binding of ports under 1024 to

priviledged/root users

● IPTables (firewall) should be configured to forward Web traffic to the Application Server ports

○Port 80 to 8080 (HTTP)○Port 443 to 8443 (HTTPS)


Upgrading

● OutSystems Platform Server upgrade (changing M.m):

1. Backups!2. Pre-requirements check up and installation;3. Install the Platform Server software;4. Upgrade the metadata;5. Install Service Center6. Install System Components7. Upgrade the applications

● Development tools are upgraded as well


Upgrading

● Application upgrade○ In Development, do in-loco component upgrade

■ Validate documented breaking changes;■ Upgrade extension modules first;■ Upgrade all eSpaces;■ Assistance from developers!

○ In Production, use a previously upgraded solution!■ Faster and safer!■ Applications typically require changes after an upgrade;■ Operation can be done by operators■ Can be done with zero downtime!


Patching

● A patch is a light update:○ The release and/or revision changes (M.m.R.r);○ Development tools don’t change;○ No breaking changes;○ No additional software requirements.

● Steps:○ Install Platform Server;○ Update metadata and install Service Center;○ Update System Components○ Republish all applications.

● Should be applied in all staging environments before applying in Production!


Tuning

● Front-End Performance Tuning○ OutSystems Platform○ Application Server○ Applications


Tuning – OutSystems Platform

● The only tuning option is enabling/disabling Debug Mode (in eSpaces)○ If environment Running Mode is Production, OutSystems Platform enforces

Debug Mode OFF in eSpaces;○ If environment Running Mode is Development, you can choose the Debug Mode

for eSpaces■ Should be ON in Development environments;■ Should be OFF in Pre-Production environments, to be as close to the

Production Environment as possible.

● Advanced tuning is possible at connection string level:○ “If it ain’t broken, don’t fix it!”


Tuning – Application Server

● JBoss memory limits

● JBoss is highly configurable via .conf files○Though OutSystems recommendation is to stick to configuring the Xmx memory

only


Backups

● What should I backup?○ The OutSystems Database

○ The Configuration Tool export file

○ Any filesystem, Application Server customizations

Platform Internals


Platform Internals

● 1-Click Publish● 2-Stage Deployment● LifeTime● Timers and asynchronous processes● Logging mechanism


Platform Internals

● 1-Click Publishing Functional Process

Service Center Deployment Controller Deployment

1. Sends OML to Controller

1st Stage Compilation

2nd Stage Compilation

Pack Application

2. Compile eSpace

3. Broadcast Application Files

5. Deploys Application in App Server

4. Update DB

XML

Java ProjectOracle

JSP/JSFWAR

ZIP

ZIPOML

OML

JAR


Platform Internals

● 1-Click Publish:○ One single file (OML) contains the complete application definition (compressed

XML)○ Deployment across several front-ends farm (no manual intervention)○ Unified deployment procedure for both application code and database scripts

(no manual intervention)○ External dependencies are deployed with the application (no separate

installation procedures)○ Deployment actions are logged for auditing purposes


Platform Internals

● 1-Click Publishing Deployment○ Executed across all active Front-ends without manual intervention;

○ eSpaces are deployed to the running folder inside the OutSystems Platform Server directory – typically /opt/outsystems/platform/running;

○ Upon startup, each Deployment Service communicates with the Controller to ensure that it is up-to-date – and if not, begins deployment of eSpaces needing updates.


Platform Internals

● 1-Click Publishing Deployment○ Parallel File Distribution

■ Only modified files will be broadcasted in parallel to Front-Ends

○ Front-End Shared Server Library■ Compiled assemblies (JAR) are stored in a shared repository in the Front-End


Platform Internals

● 2-Stage Deployment

○ Allows to execute the long and heavy lifting operations in a preparation stage (1st stage - compilation and file deployment) without impacting application availability

○ Perform the application deployment in a narrowed time window (2nd stage – data model update and Application Server runtime directory updates) using hot deployment


Platform Internals

● 2-Stage Deployment


JSP/JSFWAR

Platform Internals

● 2-Stage Deploy - 1st Stage Functional Process

Service Center Deployment Controller Deployment

3. Broadcast Application FilesZIP

1. Sends OSP to Controller OSP

OSP



Pack Application

2. Compile eSpace

XML

Java ProjectOracle

ZIP

JAR


Platform Internals

● 2-Stage Deploy – 2nd Stage Functional Process

Service Center

3. Broadcast Application FilesZIP

1. Sends OSP to Controller OSP

OSP



Pack Application

2. Compile eSpace

XML

Java ProjectOracle

ZIP

JAR

Deployment

5. Deploys Application in App Server

4. Update DB

Deployment Controller

JSP/JSFWAR


Platform Internals

● LifeTime Architecture

○ It’s an OutSystems system application■ Web Application

○ Runs only on one environment

○ Deploys applications across environments

○ Requires that all environments are on the same version (M.m)


Platform Internals

● LifeTime on Production environment

Development Test Production

LifeTime User


Platform Internals

● LifeTime on it’s own environment

Development Test Production

LifeTime User


Platform Internals

● LifeTime Synchronization process

○ Keeps LifeTime application consistency and version information up to date

○ Occurs after every publication, and periodically

○ Uses Business Process Technology (BPT)


Platform Internals

● Scheduler service is responsible for○ Running Timers○ Executing BPT events and process activities○ Sending Emails

App n

.

...

App 2App 1

queue

Main database

Front-endScheduler Service

Timer thread

Process thread

Timer thread

Timer thread

Process thread

…

Process thread

Emailthread

Emailthread


Platform Internals

● Scheduler and timers○ The Scheduler checks timers to run on the database○ Evaluate the NEXT_RUN value against current date○ Locks the Timers data model and update the timer status to RUNNING○ This lock allows multiple OutSystems Schedulers to concurrently access the same

table, without running the same timer at the same time in different Front-Ends○ Executes timer with a local Web Service Request○ Upon completion, if there’s a schedule, redefines the NEXT RUN value


Platform Internals

● Scheduler and BPT processes○ The Scheduler checks events and process activities to execute on the database○ Evaluate the NEXT_RUN value against current date○ Locks the event or activity data model and update the event status ○ This lock allows multiple OutSystems Schedulers to concurrently access the same

table, without running the same event or activity at the same time in different Front-Ends○ Executes process activities associated with the event, using a local Web Service

Request


Platform Internals

● Scheduler and Emails○ The Scheduler checks the email queue on the database○ Evaluate the NEXT_RUN value against current date○ Locks the email data model and update the email status as RUNNING○ This lock allows multiple OutSystems Schedulers to concurrently access the same

table, without running the same email at the same time in different Front-Ends○ Sends the emails directly from the Scheduler Service process and updates it’s

status○ Retry mechanism


Platform Internals

● Logging○ OutSystems Platform database logs are designed to scale with thousands or

millions of logs;○ Each log entry is stored in a table from a set which is used in sequence

■ Log tables go from oslog_<TYPE>_0 through oslog_<TYPE>_9;■ For each type, a view is mapped to the current table in the rotation

(oslog_<TYPE>) and one to the week before (oslog_<TYPE>_Previous)○ Log writing is executed directly to the respective table, to improve performance;○ Log reading in Service Center uses the two views;

■ Current Week■ Previous Week

○ One sequence of log tables for each log type.


Platform Internals

● Logging○ Log rotation occurs weekly – every Friday at 11h45 PM;○ The log table used in each week uses a fixed algorithm:

■ Number of weeks since Jan 1st 2000 MOD 10;○ Every week the next table in each sequence is used, rolling over when necessary;○ The views for each log type are updated upon rotation;○ Logs from previous weeks are still in the database, though not accessible

through Service Center ■ Can be obtained by querying the tables directly;

○ Old log tables are cleaned after their retention period passes – preparing them for later use.


Platform Internals

● Logging

Log Database

Table 0Table 1Table 2

Table 8Table 9

.

.

.

Views

Log Service

Service Center OutSystems Applications

Log Read accessLog Write AccessCurrent DB View AssociationPrevious DB View Association

Message Queue

Services

Troubleshooting


Troubleshooting

● Basic troubleshooting techniques

● Advanced troubleshooting techniques


Troubleshooting

● Basic troubleshooting: Get the error

○ Check Environment Health

○ Analyze Service Center logs

○ Analyze OutSystems Services Logs

○ Analyze Application Server Logs


Troubleshooting

● Basic troubleshooting: Environment Health


Troubleshooting

● Basic troubleshooting: Service Center log


Troubleshooting

● OutSystems Log Files○ Each Platform Service has its own log (found in /opt/outsystems/platform/logs)

■ Includes OSTraces (if enabled)■ Logs rotate everyday (One file per day)

○ To configure OSTrace logging level■ Edit config file /etc/outsystems/os.<service_name>.service.properties■ Change the log4j.logger.outsystems setting to the intended level (FATAL,

ERROR, WARN, INFO or DEBUG)


Troubleshooting

● Application Server Logs (JBoss/WildFly)○ All found in $JBOSS_HOME/standalone/log/

■ e.g. /opt/wildfly-8.2.1.Final/standalone/log/

○General Log: <logpath>/server.log■Contains log4j logs from both the Application Server and the OutSystems

Platform

○Access Log: <logpath>/access_*.log

○General logs rotate with date suffixes (e.g. server.log.2015-12-25)


Troubleshooting

● Application Server Config (JBoss/WildFly)○ All found in $JBOSS_HOME/standalone/bin/

■ e.g. /opt/wildfly-8.2.1.Final/standalone/bin/

○App Server instance: <configpath>/standalone-outsystems.conf■Contains log4j logs from both the Application Server and the OutSystems Platform

○Message Queues instance: <configpath>/standalone-outsystems-mq.conf■Contains log4j logs from both the Application Server and the OutSystems Platform


Troubleshooting

● Application Server Logs (WebLogic)○ All found in

$WL_HOME/user_projects/domains/outsystems_domain/servers/<managedserver>/logs/■e.g.

/opt/Oracle/Middleware/wlserver/user_projects/domains/outsystems_domain/servers/ip-10-142-241-8/logs/

○ General Log: <logpath>/<managedserver>.log■ Contains log4j logs from both the Application Server and the OutSystems Platform

○ Access Log: <logpath>/access.log

○ Both these logs rotate with numeric suffixes (e.g. access.log00123)


Troubleshooting

● Basic Troubleshooting: Working the Error○ OutSystems Troubleshooting Guide1. Search “Troubleshooting the OutSystems Platform Server” according to error

description2. Validate possible “Cause” as described;3. Execute “Recovery Action”;

○ Search Internet Information Knowledge Bases■ Search for the error message and description■ Confirm that the scenario matches the error description■ Evaluate and apply recommended actions

NOTE: Always consider certified internet knowledge bases!


Troubleshooting

● Advanced Troubleshooting: Get the error○ System Performance Monitor○ System network tools

■ ifconfig■ ping■ netstat■ telnet

○ Network protocols sniffers (YATT, Wireshark, etc)


Troubleshooting

● Advanced Troubleshooting: Working the error

○ Search Internet Information Knowledge Bases■ Search for the error message and description■ Confirm that the scenario matches the error description■ Evaluate and apply recommended actions

NOTE: Always consider certified internet knowledge bases!


Troubleshooting

● Proactive Monitoring and prevention○ Proactive monitoring can help prevent problems by exposing them before they

reach a critical state○ A simple monitoring approach should consist of:

■ Check Platform Monitoring screen ■ Check Platform Server logs■ Check LifeTime Performance Analytics■ Check Infrastructure Monitor

○ A more in-depth monitoring approach should also include:■ Check the reports available in Service Center■ Check System logs messages for each node

We’ll be back in 5 min to answer your questions

Questions?Please place your questions in webex Q&A or chat

Next webinars

April 7 - "Detect performance bottlenecks (Performance CSI)" - with Paulo Garrudo

April 21 - "Building a Live Style Guide" - with Ruben Gonçalves

http://www.outsystems.com/profile/7005/paulo-cunha/

http://www.outsystems.com/profile/141/paulo-garrudo/

http://www.outsystems.com/profile/7005/paulo-cunha/

http://www.outsystems.com/profile/5754/ruben-goncalves/

Software

Training Webinar System Operations: the Java side