ACTICO Platform - Execution Server · Chapter 1. About this document This document describes the installation and operation of ACTICO Execution Server. 1.1. Audience This document

ACTICO Platform - Execution Server

Operations Guide

Version 9.0.0

www.actico.com

Operations Guide: Version 9.0.0

Operations Guide

Copyright © ACTICO GmbH iii

Table of Contents

1. About this document ........................................................................................................ 1

1.1. Audience ........................................................................................................................... 1

1.2. Content ............................................................................................................................. 1

1.3. Conventions ...................................................................................................................... 1

2. Introduction ........................................................................................................................ 2

2.1. Feature Overview .............................................................................................................. 2

3. System Overview, Links and Authentication ............................................................... 3

3.1. System Overview Diagram ................................................................................................. 3

3.2. Model Hub ........................................................................................................................ 3

3.3. Web Service Client ............................................................................................................ 3

4. Installation and Configuration ....................................................................................... 4

4.1. Unpack ............................................................................................................................. 4

4.2. Java Runtime .................................................................................................................... 4

4.3. License File ...................................................................................................................... 4

4.4. Configuration ................................................................................................................... 4

4.5. Connecting Engine to Model Hub ...................................................................................... 4

4.6. Connecting Model Hub to Engine ...................................................................................... 5

4.7. Server .............................................................................................................................. 54.7.1. Common Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54.7.2. SSL - Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54.7.3. SSL - Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64.7.4. Custom HTTP Response Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.7.4.1. Defining a custom header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74.7.4.2. Predefined Security Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.8. Script Environment .......................................................................................................... 84.8.1. Startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.8.2. Shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.8.3. Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.8.3.1. Special Configuration Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.8.4. Install as Windows Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.8.5. Install as Unix Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.9. Clustering ....................................................................................................................... 11

5. Operations and Maintenance ........................................................................................ 12

Operations Guide

Copyright © ACTICO GmbH iv

5.1. Temporary directories ..................................................................................................... 12

5.2. Backup and Restore ........................................................................................................ 125.2.1. File system folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.3. Logging ........................................................................................................................... 125.3.1. Log Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.3.1.1. Custom log4j2 configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.4. Monitoring ...................................................................................................................... 135.4.1. Endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.5. Deployments Cache ........................................................................................................ 14

Chapter 1. About this document

Copyright © ACTICO GmbH 1

Chapter 1. About this documentThis document describes the installation and operation of ACTICO Execution Server.

1.1. Audience

This document is intended for

• System Administrators

1.2. Content

This document considers the following topics

• Installation

• Configuration

• Operation

• Maintenance

1.3. Conventions

The following text conventions are used in this document:

Table 1.1. Conventions

Convention Meaning

boldface Used for elements, labels and terms from the userinterface.

monospace Used for filenames or URLs.

Chapter 2. Introduction


Chapter 2. IntroductionACTICO Execution Server is responsible for the execution of models via standard web service interfaces.

It embedds the ACTICO Engine, the core component for model execution.

2.1. Feature Overview

Main features are:

• High-performance model execution

• Standard web service interfaces (REST/SOAP, JSON/XML)

• Simultaneous hosting of multiple model versions

• Deployments and roll-backs without downtime

• Traceability of individual model executions

• Horizontal & vertical scalability

• Security

Chapter 3. System Overview, Links and Authentication


Chapter 3. System Overview, Links and Authentication

3.1. System Overview Diagram

The following diagram shows Execution Server and the involved systems:

3.2. Model Hub

The URL to connect Execution Server to Model Hub is: http://localhost:8080/. (URL might vary dependingon Model Hub configuration)

See Model Hub user guide how to create credentials for the Execution Server to authenticate against ModelHub.

3.3. Web Service Client

Any Web Service client can be used to execute models via REST or SOAP.

The Base URL to connect to Execution Server is http://localhost:9090/engine/v1/executions. (BaseURL might vary depending on Model Hub configuration)

No authentication is required.

http://localhost:8080/

http://localhost:9090/engine/v1/executions

Chapter 4. Installation and Configuration



4.1. Unpack

Unzip the file execution-server-application.zip.

The app folder contains:

• the binary file of the application

The config folder contains:

• the application-execution-server.properties file that is used to store configuration settings

• the log4j2.xml file that is used to configure the logging

The bin folder contains preconfigured start and stop scripts.

The data, logs and work folders are created when the application starts. They contain dynamic content.

All resources in the config folder are automatically on the classpath of the application. Placeadditional files, like JDBC driver JAR files, in the config/lib directory.

4.2. Java Runtime

Execution Server requires a Java runtime to be available in the java folder of the installation. If this folder isempty and the Execution Server installation is part of an ACTICO Platform installation, the Java runtime definedfor the ACTICO Platform is used. If the java folder of the installation is empty and no ACTICO Platform Javainstallation was found the environment variable JAVA_HOME is used.

Make sure the Java version matches with the System Requirements

4.3. License File

Copy your obtained license file to one of the following folders:

• <user_home_directory>/.actico/license

• <installation_directory>/config/license

In case you like to rename the license file, make sure the filename starts with license and ends with thesuffix .txt.

4.4. Configuration

The file config/application-execution-server.properties is used to configure the Execution Server.The specific configuration settings are described in the next chapters. All changes to this file necessitate arestart of Execution Server. Only then changes are picked up.

4.5. Connecting Engine to Model Hub

Execution Server needs a valid URL and credentials to connect and authenticate against Model Hub. Thesettings can be changed by the following properties. The API Key can be generated within the Model Hub UI(Navigate to Environments → <Your environment> → Settings)



actico.execution.server.model-hub.url=http://localhost:8080actico.execution.server.environment-identifier=PRODactico.execution.server.environment-api-key=<insert_generated_api_key_here>

If SSL/HTTPS is configured for Model Hub ensure that the JVM that is used by Execution Server,trusts the SSL server certificate of the Model Hub.

Use the CN or Subject Alternative Name contained in the SSL server certificate as hostname toconnect to Model Hub.

4.6. Connecting Model Hub to Engine

After the Execution Server connects to Model Hub, Model Hub retrieves information about the Execution Serverusing an embedded HTTP client.

This information includes the URL that can be used to connect to the Execution Server. By default ExecutionServer uses its IP address and detects the protocol (HTTP/HTTPS) by the configured SSL setting for theembedded server of the Execution Server. If server.ssl.enabled is set to true, HTTPS is used. To overwritethis default URL determination, configure the URL Model Hub uses to connect to Execution Server by thefollowing property:

# Public URL of this Execution Server used for connections by clients (e.g. Model Hub)actico.execution.server.public-url=http://localhost:9090

Use the CN or Subject Alternative Name contained in the SSL server certificate as hostname toconnect to Execution Server.

4.7. Server

Execution Server uses an embedded web server to host the web application and endpoints (e.g. REST). Thesettings are already preconfigured, but may need to be adjusted.

In order to configure the embedded web server to your requirements add settings starting withserver.tomcat to the config/application-execution-server.properties file. A complete list ofsettings can be found at Spring Boot Application Properties.

4.7.1. Common Settings

Common configuration settings are:

# Limits the size of http post requests to a maximum number of bytesserver.tomcat.max-http-post-size=104857600

Add these settings to config/application-execution-server.properties if necessary and configuretheir values.

Execution Server does not support custom context roots. See also chapter ??? setup.

4.7.2. SSL - Server

This section describes the configuration of SSL for the embedded server.

https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html



HTTP is enabled by default.

In order to use HTTPS instead of HTTP a keystore with a SSL certificate is required. For a test and productionenvironment a SSL certificate issued by an official authority is recommended. For a development or demoenvironment a self signed SSL certificate may be sufficient.

The following command line uses the Java keytool and creates a keystore with filename keystore.p12 with aPKCS12 SSL certificate having a validity of 1 year. Use the server IP address or hostname as CN (Common Name)of the certificate (i.e. localhost or mycompany.com) which will be used to connect to the server. Otherwisethe SSL hostname verification will fail.

keytool -genkey -alias execution-server -storetype PKCS12 -keyalg RSA -keysize 2048 -keystore keystore.p12 -validity 365

If it is no option to use the IP address or hostname as CN, it can be defined as a Subject Alternative Name. Thefollowing command line uses Java keytool and creates a keystore with filename keystore.p12 with a PKCS12SSL certificate having a validity of 1 year. Additionally the hostname is specified as a Subject Alternative Name.An IP address can also be specified this way.

keytool -genkey -alias execution-server -storetype PKCS12 -keyalg RSA -keysize 2048 -keystore keystore.p12 -validity 365 -ext san=dns:localhost

Steps:

• Create keystore with SSL certificate

• Place keystore file in the config folder

• Add the following SSL related properties to config/application-execution-server.propertiesand configure them accordingly

Configure SSL related properties:

# SSL Connector portserver.port=8443# Allow only HTTPS requestssecurity.require-ssl=true

# Whether to enable SSL supportserver.ssl.enabled=true# Alias that identifies the key in the key storeserver.ssl.key-alias=execution-server# Password used to access the key in the key storeserver.ssl.key-password=<password-used-during-key-creation># Path to the key store that holds the SSL certificate (typically a PKCS12 file)server.ssl.key-store=classpath:keystore.p12# Password used to access the key storeserver.ssl.key-store-password=<password-used-during-keystore-creation># Type of the key store (JKS/PKCS12)server.ssl.key-store-type=PKCS12

If the above configuration is used, the HTTP connector of the embedded server is disabled.

4.7.3. SSL - Client

This section describes the configuration of SSL for embedded HTTP clients.

The platform components are using embedded HTTP clients connecting to HTTP servers of other componentsof the ACTICO Platform. If SSL is configured for an HTTP server, the connecting embedded HTTP client must alsobe configured for SSL.

In order to configure the embedded HTTP client to connect to a server that uses SSL, the certificate of theserver must be trusted by the JVM the embedded client uses.



Steps:

• Get the SSL certificate of the server

• Import the SSL certificate into the truststore of the JVM the embedded client is executed in

The following command line uses the Java keytool to export a certificate from a keystore to a file namedserver.cert.

keytool -exportcert -alias execution-server -keystore keystore.p12 >> server.cert

The following command line uses the Java keytool to import a certificate named server.cert to the JVMtruststore.

keytool -import -alias execution-server -keystore <JAVA_HOME>/jre/lib/security/cacerts -file server.cert

Ensure that the SSL embedded client uses the hostname or IP address that was used as CN duringSSL certificate creation. Or alternatively ensure that the hostname or IP address of the SubjectAlternative Name is used that was specified during SSL certificate creation.

4.7.4. Custom HTTP Response Headers

By default, a set of predefined HTTP headers are sent with each server response. They have been designed toprovide the best security options without limiting the usability of Execution Server.

Through the configuration, some of these headers can be customized to a specific environment and securityneeds.

Customizing the predefined headers is not recommended, unless there is a specific need to doso. Keep in mind that customizing the response headers may affect the operation and security ofExecution Server.

4.7.4.1. Defining a custom header

A custom header can be defined with the following property syntax:

actico.server.http.response-headers[Custom-Header-Name]=Custom Header Value

The Custom-Header-Name is the name of the header and the Custom Header Value the value for thatheader. (The square brackets are part of the property definition). Multiple headers can be defined: just addanother configuration entry.

4.7.4.2. Predefined Security Headers

The following security headers are predefined and can be modified.

actico.server.http.response-headers[Content-Security-Policy]=default-src 'self'; style-src 'self' 'unsafe-inline'actico.server.http.response-headers[X-Permitted-Cross-Domain-Policies]=noneactico.server.http.response-headers[Expect-CT]=max-age=86400, enforceactico.server.http.response-headers[Referrer-Policy]=no-referrer

Default headers can be disabled by setting an empty value. For example, to disablethe Expect-CT header from being returned by the server, set it to an empty value:actico.server.http.response-headers[Expect-CT]=

The Content-Security-Policy header has a different default, when using an external OIDC Provider(actico.security.authentication.provider-type=EXTERNAL_OIDC). In that case, the protocol, hostand port of the external authentication provider will be added to the header value, e.g.:



actico.server.http.response-headers[Content-Security-Policy]=default-src 'self'; style-src 'self' 'unsafe-inline'; connect-src 'self' http://localhost:8091

The Content-Security-Policy header requires to have the unsafe-inline modifier for thestyle-src directive to support Angular-based applications, like Model Hub. Angular usesinline style elements. See GitHub Issue 37631. Despite the 'unsafe' keyword our default Content-Security-Policy header is significantly more secure than having no header set.

You can find descriptions and more details about the default security headers on Mozilla Developer Networkweb docs.

4.8. Script Environment

Execution Server comes with a preconfigured set of shell scripts that can be used to configure, start, stop,install and uninstall the server.

4.8.1. Startup

On a Windows system use the start-app.bat and stop-app.bat scripts for this purpose. On a Linux systemuse the start-app.sh and stop-app.sh scripts.

In a production environment it is recommended to install the application as a system service. On a Windowssystem use the install-service.bat file to install Execution Server as a service and use start-service.bat or any operating system mechanism to start the service.

4.8.2. Shutdown

Always properly shutdown Execution Server. This is important for files to be closed. If the application wasstarted with start-app.bat or start-app.sh it should be stopped with stop-app.bat or stop-app.sh.

If the Execution Server was installed as a Windows service, the stop-service.bat file or any operatingsystem mechanism to stop a service normally can be used.

Background Information:

• Execution Server is looking for a stop file actico.jvm.stop that is created in the work folder with aconfigured filename. If the file is detected, the application will initiate the shutdown process and willterminate. The stop flag file is configured using the --actico.stopfile command line argument.The --actico.stopfile.initsleep command line argument can be used in order to specify asleep time in seconds. The application waits for this duration until it looks for a stop flag file. Usethis argument to prevent a shutdown during the startup phase of the application. Finally the --actico.stopfile.jvmshutdown command line argument can be used to simply create the stop flag file,that signals an already running JVM to terminate.

• Use the --actico.stopfile and --actico.stopfile.initsleep command line arguments for theExecution Server itself.

• Use the --actico.stopfile and --actico.stopfile.jvmshutdown command line arguments toinitiate the termination of an already running Execution Server. Note that the stop flag file handling ispreconfigured in all provided scripts.

4.8.3. Configuration

In order to temporarily pass parameters to Execution Server just specify them after the start-app.bat orstart-app.sh script.

Example: start-app.bat myParameter

If a specific configuration of Execution Server is necessary or parameters should be permanently specified,create a file config.bat or config.sh in the config folder. Use this file in order to add or overwriteenvironment variables and parameters defined by the bin\windows\config.bat or bin/unix/config.shfile. It will be evaluated after the config.bat or config.sh file in the bin folder.

https://github.com/angular/angular/issues/37631

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers



A created custom config.bat or config.sh file is also used during the installation as Windows Service orUnix Service.

Use the following environment variables to add or overwrite settings:

• JAVA_HOME: Defines the JAVA_HOME directory. Note: A JDK or JRE in the java folder of the platform orproduct is automatically detected. Example for Windows: set "JAVA_HOME=C:\Programme\Java"

• JVM_XMS: Defines the heap size in MBytes. Example for Windows: set "JVM_XMS=128"

• JVM_XMX: Defines the maximum heap size in MBytes. Example for Windows: set "JVM_XMX=2048"

• JVM_XSS: Defines the stack size in kBytes. Example for Windows: set "JVM_XSS=4000"

• JVM_OPTIONS: Defines the JVM options. Note: For Windows use a semicolon to separate multiple settingsand use § to enclose paths which may contain spaces. For Linux use a space to separate multiple settingsand use \" to enclose paths which may contain spaces. Example for Windows: set "JVM_OPTIONS=%JVM_OPTIONS%;-Djavax.net.ssl.trustStore=myTrustStore"

• JVM_OPTIONS_APP: Defines additional JVM options if started as application. Note: For Windows use asemicolon to separate multiple settings and use § to enclose paths which may contain spaces. For Linux usea space to separate multiple settings and use \" to enclose paths which may contain spaces. Example forWindows: set "JVM_OPTIONS_APP=-ea"

• JVM_OPTIONS_SERVICE: Defines additional JVM options if started as service (only supported for Windows).Note: For Windows use a semicolon to separate multiple settings and use § to enclose paths which maycontain spaces. Example for Windows: set "JVM_OPTIONS_SERVICE=-verbose"

• CLASSPATH: Defines the classpath of the application. Note: For Windows use a semicolon to separatemultiple settings. For Linux use a colon to separate multiple settings. It is not recommended to overridethis environment variable as auto detection and auto configuration will be disabled. Just add new resourcesif they cannot be stored in the config or config/lib folder. Example for Windows: set "CLASSPATH=%CLASSPATH%;C:\libs\myLib.jar"

• START_PARAMS: Defines parameters for the started application. Note: For Windows use asemicolon to separate multiple settings and use ^ as escape character for " characters that definepaths. For Linux use a space to separate multiple settings and use \ as escape character for "characters that define paths. Example for Windows: set "START_PARAMS=%START_PARAMS%;--spring.profiles.active=production

• STOP_PARAMS: Defines parameters for the application that initiates the stop of a application. Note: ForWindows use a semicolon to separate multiple settings and use ^ as escape character for " charactersthat define paths. For Linux use a space to separate multiple settings and use \ as escape character for "characters that define paths.

Example additional config.bat file stored in the config folder:

@echo offrem Custom configuration file defining maximum heap size,rem a custom trust store and an additional Spring Boot profile.set "JVM_XMX=4096"rem Using already set JVM options and add a new oneset "JVM_OPTIONS_APP=%JVM_OPTIONS_APP%;-Djavax.net.ssl.trustStore=§%ACTICO_COMPONENT_HOME%\config\myTrustStore§"rem Using already set start parameters and add a new oneset "START_PARAMS=%START_PARAMS%;--spring.profiles.active=production"exit /B 0

Example additional config.sh file stored in the config folder:

#!/bin/bash# Environment configuration file defining maximum heap size,# a custom trust store and an additional Spring Boot profile.JVM_XMX="4096"# Using already set JVM options and add a new oneJVM_OPTIONS_APP="$JVM_OPTIONS_APP -Djavax.net.ssl.trustStore=\"$ACTICO_COMPONENT_HOME/config/myTrustStore\""# Using already set start parameters and add a new oneSTART_PARAMS="$START_PARAMS --spring.profiles.active=production"



4.8.3.1. Special Configuration Modes

The script environment supports additional configuration modes for the application. A configuration modeadds additional settings to environment variables that were preconfigured by the config.bat or config.shscript. To add a new configuration mode create a new file with the following filename schema: config-<newmode>.bat. Configure the settings inside the created file. The file will be executed automatically if youpass <newmode> as a command line argument for a script file (e.g. start-app <newmode>). Note that thismechanism works for start and stop scripts of the application.

Example additional config-debug.bat file, enabling debugging and stored in the config folder:

@echo offrem Environment configuration file for mode "debug".set "JVM_OPTIONS_APP=%JVM_OPTIONS_APP%;-Xdebug;-Xrunjdwp:server=y,transport=dt_socket,address=8778,suspend=n"exit /B 0

Example additional config-debug.sh file, enabling debugging and stored in the config folder:

#!/bin/bash# Environment configuration file for mode "debug".JVM_OPTIONS_APP="$JVM_OPTIONS_APP -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=8778,suspend=n"

4.8.4. Install as Windows Service

Execution Server also contains preconfigured scripts to install, start, stop and uninstall the applicationas a Windows service. Use the install-service.bat, start-service.bat, stop-service.bat anduninstall-service.bat scripts for this purposes. The Windows Service can also be started and stoppedusing any operating system mechanism.

4.8.5. Install as Unix Service

Execution Server can also be installed as a Unix service.

• Create a file /etc/systemd/system/actico-execution-server.service

• Paste the following content into that file

[Unit]Description=Execution Server

[Service]Type=simpleUser=rootEnvironment=JAVA_HOME=/usr/lib/jvm/jreExecStart=/actico/actico-execution-server/bin/unix/start-app.shExecStop=/actico/actico-execution-server/bin/unix/stop-app.sh

[Install]WantedBy=multi-user.target

• Adopt the example settings to your installation:

• User: The user id under which the application is started.

• Environment=JAVA_HOME: Points to the Java jre installation. If you are using a dedicated Javainstallation inside your Execution Server, you can remove this line. See also chapter Java Runtime.

• ExecStart: Points to the Unix start script of your Execution Server installation.

• ExecStop: Points to the Unix stop script of your Execution Server installation.

• Enable the new service (required only once):



• sudo systemctl enable actico-execution-server

From here, Execution Server will start automatically, when the system boots. You can use these commands tocontrol the service:

• Manually stop, start, restart and check the service:

• sudo systemctl stop actico-execution-server

• sudo systemctl start actico-execution-server

• sudo systemctl restart actico-execution-server

• sudo systemctl status actico-execution-server

• Show the service log:

• sudo journalctl -u actico-execution-server

4.9. Clustering

Multiple Execution Server instances can run per Environment.

Chapter 5. Operations and Maintenance



5.1. Temporary directories

The following temporary directories are used:

• the logs folder contains log files.

• the work folder contains the embedded web server’s temporary files.

5.2. Backup and Restore

A backup is optional as the Execution Server retrieves missing data from Model Hub. A backup makes sense forlog files and configuration.

5.2.1. File system folders

The data folder contains the Release Repository that is retrieved from Model Hub, and a deployments foldercontaining two caches for the Deployments and the active Versions of the Environment the Execution Serve isregistered for. Backup makes sense if Execution Server is run without connection to Model Hub. A backup alsomakes sense to execte the first model request as fast as possible.

The config folder can to be included in the backup as it may contain specific configuration files and specificconfiguration settings.

5.3. Logging

Execution Server uses Apache Log4j 2 by default and comes with a default log4j2 configuration available in theconfig directory.

The default configuration has been designed for production usage and provides the following settings:

• Log to console AND file

• Log errors to separate error log file

• Maximum size for log files: 10MB

• Maximum number of roll-over files: 20

• Files exceeding 10MB are zipped and placed in a archive directory next to the log file

• Maximum amount of zipped files to be kept: 20

These settings result in a maximum usage of about 70MB of disk-space consumed for log files.

5.3.1. Log Levels

Log levels can be configured in config/application-execution-server.properties (restart of serverrequired) OR in config/log4j2.xml

Examples configuring log levels in config/application-execution-server.properties:

# Log level configuration# Example to enable debug logging for a part of the applicationlogging.level.com.actico.repository=DEBUG

Examples configuring log levels in config/log4j2.xml:

https://logging.apache.org/log4j/2.x/



<loggers> <logger name="com.actico.repository" level="warn" /> ...  </loggers>

5.3.1.1. Custom log4j2 configuration

Custom log4j2 configurations should only be considered, when the default logging appenders are not sufficientor required to be changed.

If a custom log4j2 configuration is desired, for example to configure custom appenders, create your ownlogging file by copying the existing log4j2.xml and place it in the config directory. Afterwards activate theconfig in config/application-execution-server.properties:

The following config shows how to activate a custom log4j2-custom.xml for logging.

# Enable log4j2 custom configuration, if required. See operations guide for details.logging.config=${actico.component.home}/config/log4j2-custom.xml

Configuring your own log4j2 configuration can affect the standard behavior of the product andimpede maintenance and support.

5.4. Monitoring

To monitor the application, Spring Boot’s Actuator Web API is enabled. The actuator endpoints can be accessedat http://localhost:9090/actuator/<endpoint_name>.

Please check the Spring Boot Actuator Web API documentation for further information.

5.4.1. Endpoints

All actuator endpoints are enabled, here is a short list of some of them. The complete list can be seen at SpringBoot Actuator Endpoints documentation.

• info - Display application name and version

• health - Display health status of application (database, disk space)

• env - Display property environment configuration

• configprops - Display configuration settings

• threaddump - Display current thread dump

• metrics - Display application metrics

• logfile - Display the log file

• prometheus - Data source for the Prometheus monitoring solution

The endpoints require authentication. Either basic authentication, or OAuth2 token based authentication. Anauthenticated user must also have the configured authority (default: application.metrics). The authoritycan be configured with the actico.security.permission.actuator-endpoints-authority property.

See Chapter "REST Endpoints" how to authenticate using token based authentication.

The endpoints actuator/info and actuator/health do not require authentication orauthorization by default and therefore are suited best to be used for monitoring.

http://localhost:9090/actuator/<endpoint_name>

https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html

https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html

http://prometheus.io



5.5. Deployments Cache

In order to allow the execution of a Release by Stream in case the connection to Model Hub is offline or theExecution Server is restarted, the Deployments and active Versions are cached. These caches are contained inthe deployments folder located in the data directory.

The cache is updated every time the connection between Model Hub and Execution Server is established or theExecution server receives new Deployments or active Versions from Model Hub. If no cache exists on file systemthe directory is created automatically whenever new values are received.

Documents

ACTICO Platform - Execution Server · Chapter 1. About this document This document describes the installation and operation of ACTICO Execution Server. 1.1. Audience This document