Masaryk UniversityFaculty of Informatics

Clusterable Task Scheduler

Bachelor’s Thesis

Ján Michalov

Brno, Fall 2019

Masaryk UniversityFaculty of Informatics

Clusterable Task Scheduler

Bachelor’s Thesis

Ján Michalov

Brno, Fall 2019

This is where a copy of the official signed thesis assignment and a copy of theStatement of an Author is located in the printed version of the document.

Declaration

Hereby I declare that this paper is my original authorial work, whichI have worked out on my own. All sources, references, and literatureused or excerpted during elaboration of this work are properly citedand listed in complete reference to the due source.

Ján Michalov

Advisor: RNDr. Adam Rambousek, Ph.D.

i

Acknowledgements

I would like to sincerely thank my advisor RNDr. Adam Rambousek,Ph.D. for his guidance, patience and precious advice. I am also gratefulto my consultant Bc. Matej Lazar, who directed me and helped withthe design across countless hours of meetings. I wish to thank myfriends and family for their support during stressful days.

iii

Abstract

The purpose of this thesis is to create a microservice that would sched-ule tasks happening on other devices. The tasks can have differenttasks declared as dependencies, and the microservice must executethem in the correct order. Additionally, the microservice must be ableto be deployed in a clustered environment, which means ensuringdata consistency and preventing duplicate execution of a task. Thechosen platform for the microservice is Java.

iv

Keywords

microservice, dependency resolution, scheduling, Java, data consis-tency, cluster

v

Contents

Introduction 1

1 Theoretical background 31.1 Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Applicaton clustering . . . . . . . . . . . . . . . . . . . . . 31.3 Microservice architecture . . . . . . . . . . . . . . . . . . . 4

2 JBoss MSC 52.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 Service . . . . . . . . . . . . . . . . . . . . . . . . 52.1.2 ServiceController . . . . . . . . . . . . . . . . . . 62.1.3 ServiceRegistrationImpl . . . . . . . . . . . . . . 72.1.4 Dependency and Dependent . . . . . . . . . . . 72.1.5 ServiceRegistry . . . . . . . . . . . . . . . . . . . 82.1.6 ServiceTarget . . . . . . . . . . . . . . . . . . . . 82.1.7 ServiceContainer . . . . . . . . . . . . . . . . . . 9

2.2 Transitional period of a ServiceController . . . . . . . . . . 92.3 Concurrency and synchronization . . . . . . . . . . . . . . 112.4 Pros and cons . . . . . . . . . . . . . . . . . . . . . . . . . 122.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Infinispan 153.1 Client-server mode . . . . . . . . . . . . . . . . . . . . . . 15

3.1.1 Network protocols . . . . . . . . . . . . . . . . . 163.1.2 Server . . . . . . . . . . . . . . . . . . . . . . . . 163.1.3 Hot Rod Java client . . . . . . . . . . . . . . . . . 17

4 Application platform 214.1 JBoss EAP . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2 Thorntail . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.3 Quarkus . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.4 Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

5 Design 255.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.1.1 Must have . . . . . . . . . . . . . . . . . . . . . . 25

vii

5.1.2 Should have . . . . . . . . . . . . . . . . . . . . . 255.1.3 Could have . . . . . . . . . . . . . . . . . . . . . 265.1.4 Remote entity requirements . . . . . . . . . . . . 26

5.2 Differences and similarities to JBoss MSC . . . . . . . . . . 265.2.1 Naming changes . . . . . . . . . . . . . . . . . . 265.2.2 What stayed . . . . . . . . . . . . . . . . . . . . . 275.2.3 What changed . . . . . . . . . . . . . . . . . . . . 27

5.3 States, Transitions, Modes and Jobs . . . . . . . . . . . . . 285.3.1 Modes . . . . . . . . . . . . . . . . . . . . . . . . 295.3.2 Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . 295.3.3 StageGroups . . . . . . . . . . . . . . . . . . . . . 295.3.4 States . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.4 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

6 Implementation 336.1 Context dependency injection . . . . . . . . . . . . . . . . . 33

6.1.1 Maven module problem . . . . . . . . . . . . . . 336.2 Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6.2.1 Partial updates . . . . . . . . . . . . . . . . . . . 346.2.2 Prevention of duplicate execution . . . . . . . . 34

6.3 Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.4 REST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.5 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . 36

6.5.1 Prerequisites . . . . . . . . . . . . . . . . . . . . . 366.5.2 Setting up an Infinispan server . . . . . . . . . . 376.5.3 Compilation and execution . . . . . . . . . . . . 38

7 Testing 397.1 Local integration testing . . . . . . . . . . . . . . . . . . . 397.2 Clustered testing . . . . . . . . . . . . . . . . . . . . . . . 40

8 Conclusion 438.1 Future improvements . . . . . . . . . . . . . . . . . . . . . 43

Bibliography 45

A Attached files 47

viii

List of Figures

2.1 State-diagram of JBoss MSC. Source: [6] 65.1 State-machine diagram of a Task 285.2 The diagram of package dependencies in the

scheduler 31

ix

Introduction

This thesis was created as an effort from company Red Hat, to im-prove the scalability of a product called Project Newcastle. Nowadays,scalability is a common problem across products. There are two waysto scale a product. Vertically with an addition of power in the formof memory and CPU cores or horizontally with clustering. However,some products suffer from a sophisticated monolithic design. Thisproblem also persists in Project Newcastle. One of the techniques thatsolve this dilemma is microservice architecture. Microservice architec-ture aims to dissect these monoliths into smaller parts, each with theirfunction, that communicates with each other. These parts are simple,therefore easier to maintain, and should be designed in a way to scalein a cluster.

One of the functions of Project Newcastle is to schedule tasks,which are executed remotely. Additionally, these tasks have defineddependencies and therefore have to be scheduled in the correct order.The goal of this thesis is to create an open-source remote schedulerwith an ability to scale in the cluster and microservice architecture inmind.

The thesis is made of seven chapters excluding conclusion. The firstchapter focuses on the theoretical aspect of the thesis and introducesthe reader to complexities of scheduling, clustering and microservicearchitecture. The following chapter regards to an analysis of JBossMSC library. The library implements a scheduling solution with a dif-ferent use-case but flexible implementation, which concepts are usedin the design. Chapter three concentrates on a Red Hat developeddatastore solution Infinispan, which intent is to be used for enable-ment of clustering for a variety of applications. Next chapter is brieflyintroducing available Red Hat application platforms, their strong andweak aspects, andwhich is themost suitable for the scheduler. Chapterfive is the design of the application. The chapter defines and explainsthe requirements, points out the major distinctions against JBoss MSCand defines the states of a task and other essential models. Chaptersix delves into the problematics of the implementation part of the the-sis. This chapter points out some flaws of used libraries/frameworks,describes how is data consistency guaranteed and concludes with

1

a guide for compiling from source and subsequent execution of thescheduler. The last chapter before the conclusion is focused on testing.The testing includes local integration tests and clustered tests, whichthe chapter describes in detail.

2

1 Theoretical background

1.1 Scheduling

In a scheduling problem, there is a set of tasks and a set of constraints.These constraints state that executing a specific task could depend onother tasks being completed beforehand. These sets can be mappedinto a directed graph, with the tasks as the nodes and the direct pre-requisite constraints as the edges. If this graph has a cycle, then itimplies that there exists a task, which is transitively dependent onitself. Therefore, this task has to complete before it can start, whichdoesn’t make sense. Hence, the graph can not have cycles and canbe expressed as a directed acyclic graph (DAG). To schedule a setof tasks an ordering is needed. The order has to respect dependencyconstraints. For DAG, this ordering is called topological sorting. Everyfinite DAG has a topological sort. However, there can be more thanone possible topological sort.[1]

A topological sort can be found by iteratively marking a task thathas either no dependencies or all of its dependencies are marked. Theordering of marks suggests a topological sorting. This algorithm canproduce a different order if it has more than one task available to mark.

For parallel task scheduling, the algorithmmentioned above can bemodified. Instead of marking one task each iteration, it marks all tasksready for marking. All tasks marked in one iteration are independentof each other and can execute concurrently. A further modificationcould be to only allow a certain number of marks in one iteration,which simulates a situation where resources are limited.

1.2 Applicaton clustering

Application clustering typically refers to a method of grouping mul-tiple computer servers into an entity that behaves as a single system.A server in a cluster is referred to as a node. Typically, each noderuns the same copy of an application that is usually deployed on anapplication server which provides clustering features (Wildfly). For

3

1. Theoretical background

instance, Wildfly1 application servers can discover each other on anetwork and replicate the state of a deployed application [2]. Benefitsof clustering include [3]:

1. LoadBalancing (Scalability): Processing requests are distributedacross the cluster nodes. The main objective of load balancing isto limit nodes getting overloaded and possibly shutting down.Adding more nodes to a cluster increases the cluster’s wholecomputing capabilities.

2. Fail-over (High Availability): Clusters enable services to last forlonger periods. Singular servers have a single point of failure. Aserver can fail unexpectedly due to unforeseen causes such asinfrastructure issues, networking problems or software crash-ing. On the other hand, a cluster is more resilient. If one nodecrashes, there are still other nodes that can handle incomingrequests.

A direct method of developing a clusterable application is withoutkeeping a state. A stateless application does not retain data for lateruse. However, the state can be stored in a database instead. Eachstateless application can connect to the database where they keep allinformation. Stateless applications are easily scalable.[4]

1.3 Microservice architecture

Microservice is architectural style motivated by a service-orientedarchitecture (SOA) that appeared due to a need for flexible and conve-niently scalable applications as opposed to monolithic style, which ischallenging to use in distributed systems. Microservices handle gradu-ally increasing complexity of large systems by decomposing them intoa set of independent services. These services are loosely-coupled, andeach should provide specific functionality. Moreover, microservicescan be developed independently, which simplifies creating featuresand code maintenance.[5]

1. https://wildfly.org/

4

2 JBoss MSC

JBoss MSC1 (JBoss Modular Service Container) is a “lightweighthighly concurrent open-source dependency injection container”2 de-veloped by Red Hat and written in Java. It is internally used by JBossEnterprise Application Platform3 (JBoss EAP) which is Red Hat sup-ported version of WildFly4. JBoss EAP consists of a hundreds of com-ponents that are dependant on each other and have to start in thecorrect order. JBoss MSC solves this complexity by the ability to de-fine these relations while also providing features such as concurrentexecution, failure handling or retry mechanisms. This chapter is basedon an analysis of the source code and documentation[6] of the JBossMSC. It addresses internal architecture, design and positives alongwith negatives of the implementation against requirements of thisthesis.

2.1 Architecture

This section analyzes specific classes and interfaces, which serve asignificant role in the library’s logic. The analysis is based on 1.3.2.Fi-nal5 version JBoss MSC, which is less complicated. Version 1.4.0.Final6and later deprecate most of the library and provide new API to addsupport for Services, to return multiple values.

2.1.1 Service

Service is the central abstraction of the MSC. Definition of a Serviceis something that can be started and stopped. To create a Service, auser has to create a class implementing the Service interface and definethe start, stop and getValue methods. The getValue method serves as

1. https://github.com/jboss-msc/jboss-msc2. https://jboss-msc.github.io/jboss-msc/manual/3. https://www.redhat.com/en/technologies/jboss-middleware/application-platform/4. https://wildfly.org/5. https://github.com/jboss-msc/jboss-msc/tree/1.3/6. https://github.com/jboss-msc/jboss-msc/tree/1.4/

5

2. JBoss MSC

Figure 2.1: State-diagram of JBoss MSC. Source: [6]

a way for a Service to provide a value. Because, for a Service to start,it may need more information or specific value that another Serviceproduces. For example, HTTP server Service may need a value from aService providing host and port. Therefore, Services create a schedulingproblem defined by a directed acyclic graph, where the node is aService, and vertex signify a dependency.

2.1.2 ServiceController

ServiceController is an interface used to manipulate metadata requiredfor scheduling operations of the Service. Its implementation (Service-ControllerImpl) is both owner and manipulator of the metadata. EachServiceController has a unique identifier, and an instance of Serviceassociated. Essential metadata:

6

2. JBoss MSC

• ServiceName name: unique identifier for ServiceController, thatcan be imagined as a string in a format similar to Java packagenames (e.g. “org.jboss.msc.identifier”)

• Enumeration State and Substate: Represents a state of a Service-Controller in a scheduling process. Substate is a fine-grainedrepresentation of State. For a visualization, see Figure 2.1.

• Dependency[] dependencies: An array of dependencies thatController uses for sending messages to dependencies.

• ServiceRegistrationImpl registration: Registration used for retriev-ing dependents.

• Additional integers that keep track of information about depen-dencies/dependents. Example being, runningDependents andstoppingDependencies.

Purpose of a ServiceController is to handle transitions between states,handle communication to dependents and dependencies, providefeatures for removing Services from ServiceContainer.

2.1.3 ServiceRegistrationImpl

ServiceRegistrationImpl is a class that encapsulates ServiceControllerand implements further functionality. Namely adding dependentsat runtime and read/write locking support through Lockable class.Notable properties:

• ServiceControllerImpl instance: An instance of ServiceController-Impl that registration owns.

• Set<Dependent> dependents: A set of dependents on this reg-istration.

2.1.4 Dependency and Dependent

Dependency and Dependent interfaces serve as a medium for commu-nicating between ServiceController and its dependencies/dependents.Suppose ServiceController needs to alert its dependents that it stopped.Firstly, the Controller gets a collection ofDependents (available through

7

2. JBoss MSC

a property of ServiceRegistrationImpl). Secondly, for each dependent,it invokes dependencyDown()method. Due to ServiceControllerImpl im-plementing the Dependent interface, dependencyDown()method is in-voked on the ServiceController of the dependent. The method changesthe metadata (in this particular case, increasing stoppingDependentsinteger by one), which may cause a transition between states. More-over, ServiceRegistrationImpl implements Dependency interface, but theimplementation is mostly delegated to ServiceControllerImpl.

2.1.5 ServiceRegistry

ServiceRegistry is an interface that acts as a registry for Services andtheir Controllers. This interface exposes an API for retrieving installedServiceControllers. To retrieve a specific ServiceController, a user needsto know the ServiceName that uniquely identifies it. There are twomethods to retrieve a ServiceController. They differ in handling caseswhere the ServiceController is missing. One returns null, and otherraises a ServiceNotFoundException. The implementation of ServiceReg-istry uses ConcurrentHashMap as a data store, where the identifyingkey is ServiceName and value ServiceRegistrationImpl.

2.1.6 ServiceTarget

ServiceTarget is a target for Service installations. To install a Service, theuser has to invoke addService(ServiceName name) method which re-turns a ServiceBuilder. ServiceBuilder, with the usage of the builder pat-tern7, is used to define all information that ServiceController requiresto function in the container. ServiceBuilder API:

• setInitialMode(Mode mode): ServiceController’s starting mode.

• addDependency(ServiceName dependency): Declares a dependencyto created Service.

• setInstance(Service service): Sets the instance of a Service to exe-cute.

7. For additional information: https://www.journaldev.com/1425/builder-design-pattern-in-java

8

2. JBoss MSC

• install(): Invoked at the end to finalize the declaration of Serviceand its relations.

Firstly, installing ServiceBuilder causes Target to create new uniqueServiceRegistrationImpl and ServiceControllerImpl. Secondly, it informsexisting dependencies about new dependents, initializes initialModeand State for the Controller and checks for circular dependencies. Inthe end, it adds new ServiceRegistrationImpl to a registry.

2.1.7 ServiceContainer

ServiceContainer extends ServiceTarget and ServiceRegistry. Therefore,the implementation of ServiceContainer operates both as a registry andas a target for Service installations. Moreover, the interface providesa method for shutting down the ServiceContainer. To shut down, theServiceContainer changes Mode for each registered ServiceController toMode.REMOVE. Furthermore, the shutdown can be initiated by re-ceiving a SIGINT signal to the JVM process running the container. Toachieve this, ServiceContainer registers a shutdown hook to Java Run-time8 , which executes on JVM termination. Furthermore, ServiceCon-tainer provides an ExecutorService9 that is needed by ServiceControllerduring transition periods.

2.2 Transitional period of a ServiceController

ServiceController’s decision to transition is solely based on its ownmetadata. Hence, the metadata has to be modified before a transitioncan start. Transitions between States can initiate in two ways:

1. Different ServiceController/Registration invokes amethod throughDependent interface with the reason to inform the invoked Con-troller that something occurred. Based on this information, theController adjusts the metadata and checks if the change causesa transition.

8. https://docs.oracle.com/javase/8/docs/api/java/lang/Runtime.html#addShutdownHook-java.lang.Thread-9. https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html

9

2. JBoss MSC

2. Changing theMode through the setMode()method.All possible transitions are pre-defined and reflected in an enumera-tion Transition. Therefore, an invalid transition cannot arise. If a tran-sition does occur, ServiceController enters a transition period. Duringthis period, no other transition can occur until the period completes.Each transition has a pre-defined list of ControllerTasks which are runafter a Transition has been chosen. A ControllerTask is a class that actsas a unit of execution. They implement Runnable10 interface and areexecuted asynchronously in the ExecutorService provided by Service-Container. A notable mention is that ControllerTasks have direct accessto data of a ServiceControllerImpl they are instantiated in, due to beinga nested inner class11 of ServiceControllerImpl. Therefore, each Con-trollerTask is bound to some ServiceControllerImpl. There are numeroustypes of ControllerTasks:

• StartTask: Executes a Service through invocation ofService.start()• StopTask: Stops the execution of a Service through invocation of

Service.stop()

• DependentsControllerTask: Abstract class which implements away to send message to all dependents through Dependentsinterface.

• DependenciesControllerTask: Abstract class which implements away to send message to all dependencies through Dependencyinterface.

• DependencyStartedTask: Implements DependentsControllerTasksand informs all dependents that their dependency has startedthrough Dependent.dependencyUp() method.

• ...A transitional period concludes with all bound ControllerTasks com-pleted. In conclusion, the whole workflow can be divided into severalparts:

10. https://docs.oracle.com/javase/8/docs/api/java/lang/Runnable.html11. For additional information: https://docs.oracle.com/javase/tutorial/java/javaOO/nested.html

10

2. JBoss MSC

1. Invoke setMode or Dependency/Dependant interface method.2. Alter metadata.3. Check no bound ControllerTasks are running.4. Get the current State.5. Check for transitions from the current State according to altered

metadata.6. If affirmative, get ControllerTasks associated with the transition.

If negative, return.7. Set new State.8. Enter the transition period.9. Get an ExecutorService from ServiceContainer.10. Execute ControllerTasks.11. Exit the transition period.

2.3 Concurrency and synchronization

JBoss MSC is a highly concurrent library, where asynchronous Con-trollerTasks are the leading cause. Hence we can assume that the datawill be accessed from more than one thread.

Example being, two Serviceswith common dependent have startedat the same time. Both enter a transition period to theUP state andwantto inform the common dependent about it. The ControllerTasks (De-pendencyStartedTask) invoke dependencyUp() on the ServiceControllerof the common dependent. In the method, the stoppingDependenciesattribute is read (2) at the same time. Therefore it has the same valuefor both dependencies. Both reduce the variable’s value by one (1)and save. The variable now has an inconsistent value (1), due to de-pendencies not having serialized access to the variable.

If one of the dependencies waited until the other one saved, theinconsistency would not occur. One of the resolutions is definingborders in the code, which can be accessed by one thread at the sametime. Java has an answer in the form of synchronized blocks. Thesynchronized block construct takes a parameter of any instantiatedJava Object. When a thread enters a synchronized block, it acquires alock based on the passed parameter. Until it exits the block, no otherthread can enter the synchronized block with the same parameter.Therefore for synchronization to take effect, both threads have to enter

11

2. JBoss MSC

the same parameter. In this case, the instance of ServiceControllerImplis the solution.

JBoss MSC uses synchronized blocks in every ControllerTask andimplementations ofDependency/Dependentmethods. To avoid enteringmethodswithout synchronization, the library usesThread.holdsLock()12method, which takes a parameter of an Object instance. If the parame-ter was also used as a parameter in a synchronized block, it returnstrue.

2.4 Pros and cons

Purpose of the analysis of JBoss MSC is to investigate whether it isdeploy-able in a cluster and, ifmodified, tomeet the conditions definedin the requirements of this thesis.

Positives• Well-thought-out architecture with every component clearly

defined.• Highly flexible design that allows full modification of States,

Transitions and ControllerTasks to accommodate every demand.• Adding functionality is a matter of creating new ControllerTask

and adding it to desired Transitions.

Negatives• Using synchronized blocks hinders the ability to run in a cluster,

due to them being bound to JVM, which clusters do not share.• In-memory datastore with ConcurrentMap.• Very complicated code base resulting from supporting multiple

features, where most of them are not necessary for the thesis(retry mechanism, value injections). Consequently, modifyingthe library would add unnecessary complexity.

• Services run locally, whereas the intention is running remotely.• Missing feature for submitting multiple Services at once.

12. https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html#holdsLock-java.lang.Object-

12

2. JBoss MSC

2.5 Conclusion

Directly using the codebase for a clustered solution is not desirable dueto in-memory datastore and JVM-bound locking. With the support forall functionality, modifying the codebase would prove complicatedbecause each modification would require a more significant effort toimplement. Therefore, the decision is writing new codebase from thebeginning. The codebase will be written with an architecture close toJBoss MSC, but adjusted for remote scheduling and native clustering.

13

3 Infinispan

Infinispan is an open-source in-memory, distributedNoSQL key/valuedatastore solution. Its primary focus is to be used in conjunction withclusters, where Infinispan offers high-speed access with replicationand distribution of data. Infinispan is designed to work in a highlyconcurrent environment, while also providing functionality such astransactions, clustered locks, querying or distributed processing.

Apart from that, Infinispan implements and integrates with a myr-iad of known frameworks likeCDI,Hibernate, Apache Lucene, ApacheSpark, Quarkus and Wildfly. The supported version of Infinispan isRed Hat Data Grid1 or RHDG. Infinispan provides two modes:

1. Embedded: In the embedded version, Infinispan co-exists withthe application in the same JVM. To enable clustering and hori-zontal scaling of the application, Infinispan automatically dis-covers neighbouring instances of the application. Therefore,creating a cluster is a matter of creating multiple nodes andverifying they can interact.

2. Client-server: In the client/server version, Infinispan is sepa-rated from the application and runs as a remote server. Theapplication uses Infinispan client to connect to the server usinga network protocol.

This chapter focuses on the client-server version of Infinispan as it wasthe one chosen in the implementation. Reason being, limited supportof embedded mode in Quarkus application server2. The informationused in this chapter was inspired by Infinispan user guide[7].

3.1 Client-server mode

In client-server mode, applications access the data stored in a remoteInfinispan server through a network protocol. The Infinispan server

1. https://www.redhat.com/en/technologies/jboss-middleware/data-grid2. More in chapter Application servers

15

3. Infinispan

itself is capable of horizontal scaling in a cluster and can support hun-dreds of nodes. Therefore, the client-server mode achieves elasticitydue to the independent scaling of server and client.

3.1.1 Network protocols

Network protocols used by Infinispan are language-neutral. Conse-quently, clients can use other programming languages like C++, C#,Perl, Javascript, Ruby and Python3. Infinispan supports three distinctnetwork protocols for interacting with a server:

• Hot Rod protocol: It is a binary protocol directly developed byInfinispan developers. It is a recommended protocol to use ifrunning Java.

• REST endpoint: Infinispan server exposes RESTful HTTP inter-face to access caches. It is recommended for an environmentwhere HTTP port is the only one allowed.

• Memcached protocol: It is a text-based protocol with an appli-cation for users, that desire failover capabilities of their Mem-cached server, which is not natively clusterable.

Hot Rod protocol is the protocol with the most significant supportand functionality with features such as topology awareness, partialtransaction support, bulk operations, listeners and server queries.

3.1.2 Server

Infinispan server is a standalone server that provides caching abilitythrough a variety of network protocols. The server exposes additionalservices for management, persistence, logging, security and transac-tions.

Moreover, the server can be extensively configured. The documen-tation for configuring the server can be found in the server guide

3. https://infinispan.org/hotrod-clients/

16

3. Infinispan

on Infinispan website4, and all available configuration options areaccessible here5.

3.1.3 Hot Rod Java client

This section describes the configuration of the Hot Rod client, howto access and manipulate a cache, data marshallers, transactions andtheir limitations and querying through Query DSL. Additionally, HotRod protocol supports authentication, data encryption and versioninteroperability (different client and server versions).

ConfigurationBefore accessing a cache, Hot Rod client requires configuration. Theclient can be configured either programmatically or through a con-figuration file. A user has to specify at least the host and port of theInfinispan server. Furthermore, the user can define marshallers, trans-actionmodes andmanagers, balancing strategies, a maximum numberof near cache entries andmuchmore6. The configuration is then passedinto RemoteCacheManager7, which can retrieve RemoteCaches.

Remote cacheAll caches in Infinispan are named. Therefore, to retrieve a cache, theuser has to know its name. Remote caches implement ConcurrentMapinterface, which is the primary API for interacting with Remote cache.However, each invocation causes communication with a server. There-fore, to improve performance, the client can be configured to use nearcaching. Near cache is a local cache that stores recently used data. Ifanother client modifies the data, the local cache is invalidated, andthe next access causes communication with the server.

4. https://infinispan.org/docs/9.4.x/server_guide/server_guide.html#infinispan_subsystem_configuration5. https://docs.jboss.org/infinispan/9.4/configdocs/infinispan-config-9.4.html6. https://docs.jboss.org/infinispan/9.4/apidocs/org/infinispan/client/hotrod/configuration/ConfigurationBuilder.html7. https://docs.jboss.org/infinispan/9.4/apidocs/org/infinispan/client/hotrod/RemoteCacheManager.html

17

3. Infinispan

Versioned APIRemote cache provides additional API which is similar to the primaryAPI. The difference is that methods retrieve or take extra parameter inthe form of version. For each key/value pair, a version is an integer thatuniquely identifies each modification. Versioned API can prevent datainconsistencies. If a client saves a value, but he has specified an olderversion, the save will fail, and the client has to react appropriately.

MarshallingMarshalling is a process of mapping Java Objects to a format thatcan transfer over a wire. Infinispan client supports some marshallinglibraries natively such as JBoss Marshalling8 or Infinispan Protostreammarshaller9 that is based on Protobuf serialization format developedby Google10.

TransactionsHot Rod clients are able to participate in Java Transaction API (JTA)transactions11. However, the support is limited:

• Isolation level has to be REPEATABLE_READ, which providesa guarantee that data will not change for the duration of thetransaction once it has been read for the first time.

• Locking must be PESSIMISTIC. Locks are obtained on a writeoperation, while in OPTIMISTIC setting the lock is received atthe end of a transaction, where transaction manager checks fordata consistency.

• Transactionmode can be eitherNON_XAorNON_DURABLE_XA.NON_XA transactions are local and specific to one resourceor database. On the other hand, NON_DURABLE_XA transac-tions are global and can span across multiple resources.

8. https://jbossmarshalling.jboss.org9. https://github.com/infinispan/protostream10. https://developers.google.com/protocol-buffers11. https://javaee.github.io/javaee-spec/javadocs/javax/transaction/TransactionManager.html

18

3. Infinispan

QueryingIn client/server mode, Infinispan supports querying through theirquery language called Infinispan Query DSL. Example:

import org.infinispan.query.dsl.*;

// get the DSL query factory from the cache:QueryFactory qf = org.infinispan.query.Search.getQueryFactory(cache);

// create a query for all the books with a title containing \say{engine}:org.infinispan.query.dsl.Query query = qf.from(Book.class)

.having("title").like("%engine%")

.build();

// get the results:List<Book> list = query.list();

To enable indexing of entities, the server has to know the structureof data. Therefore, Infinispan encourages users to utilise Protostream-Marshaller, which forces users to create entity schemas in the form of“.proto” files with an option to specify indexed fields. These schemasare sent to the server on client initialization. Moreover, Infinispan pro-vides an automatic way to generate schemas through an annotationprocessor.

19

4 Application platform

The application server is a framework that provides an environmentfor creating enterprise web applications. Java application servers arebased on the Java Enterprise Edition(Java EE) standard, with the latestversion being Java EE 8. For an application server to be Java EE com-pliant, it has to provide implementations for numerous specifications.For instance, the most notable being:

• Java PersistenceAPI (JPA): interactswith the database, providesobject-relational mapping with annotations

• Java Transaction API (JTA): adds functionality for transactionsthat can span multiple resources

• Java API for RESTful Web Service (JAX-RS): provides the func-tionality to create web services corresponding to RESTful archi-tectural style

• Context Dependency Injection (CDI): framework enabling thecode to follow SOLID principle1

This chapter analyses all application servers provided by Red Hat oneby one and ends with a decision which one to utilise for implementa-tion.

4.1 JBoss EAP

JBoss Enterprise Application Platform is Red Hat supported applica-tion server of Wildfly that provides modular, cloud-ready architectureand robust tools for management and administration. Additionally,JBoss EAP Java EE 8 certified and offers tools for high-availabilityclustering, distributed caching, messaging and transactions. Moreover,JBoss EAP provides a command-line interface for run-time modifica-tion of configuration[8]. It is a mature and thoroughly tested productwith a history spanning at least 17 years2.

1. https://itnext.io/solid-principles-explanation-and-examples-715b975dcad42. Archive - https://jbossas.jboss.org/downloads

21

4. Application platform

4.2 Thorntail

Thorntail3 is an application server primarily tailored for developmentof microservices. The core of Thorntail is also Wildfly, but Thorntailoffers users to choose which parts they need for their service insteadof getting the full unit. Therefore, the service is lesser in size andgenerates a smaller memory footprint. Furthermore, Thorntail imple-mentsMicroProfile4specification, which includes standards optimizedfor microservices such as health checks, metrics, reactive messaging,OpenApi5 generation, fault tolerance and more.

4.3 Quarkus

Quarkus6 is a new and innovative application platform similar toThorntail in the basic concepts. These include marketing towards mi-croservices and implementing most of Java EE 8 and MicroProfilespecifications. Quarkus is revolutionary in the sense that it enables na-tive Java compilation, which is possible due to a new universal virtualmachine called GraalVM7. GraalVM allows for immensely better boottimes, even smaller memory footprints and overall better performance.However, GraalVM has a considerable number of limitations. Javalibraries and frameworks that use reflection8 will not compile, unlessspecific classes that use reflection are registered ahead of compila-tion. Additionally, dynamic class loading is not supported, which is aroutine approach in Java to achieve modularity. The full list is postedhere9. To counter, Quarkus offers a vast amount of libraries that sufficemost cases and compilation for regular JVM.

3. https://thorntail.io/4. https://microprofile.io/5. https://swagger.io/docs/specification/about/6. https://quarkus.io/7. https://www.graalvm.org/8. https://www.oracle.com/technical-resources/articles/java/javareflection.html9. https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md

22

4. Application platform

Another innovation that Quarkus brings is live coding with devel-opment mode, background compilation10 and overall more comfort-able support for integration testing.

4.4 Decision

As one of the goals of microservice architecture is small and fastdeployment, JBoss EAP is the least suitable due to providing all func-tionality of Java EE 8 in contrast to both Thorntail and Quarkus.

Thorntail has the advantage of beingmuchmorematurewithmorecomprehensive documentation and fewer potential bugs. However,Thorntail ceased proactive development in support of Quarkus.[9]Therefore the decision is to use Quarkus.

10. https://quarkus.io/guides/getting-started#development-mode

23

5 Design

5.1 Requirements

The requirements were defined through a cooperation with possiblefuture users and the aid of the MoSCoWmethod.1

5.1.1 Must have

• Dependency resolution: For a task to start, it has to wait fordependencies. (implemented)

• Remote execution: The scheduler has to have the ability to in-voke execution on a remote entity. (Remote entity could be as aserver where task executes) (implemented)

• Circle detection: Prevent requests that would create a circle ina dependency graph. (implemented)

• Ability to run in cluster (verified)

• Task cancellation: User has to have an option to cancel a task.The scheduler needs to inform the remote entity if the task isrunning and prevent the execution of dependent tasks. (imple-mented)

• Public RESTful API: The scheduler has to provide a public APIthat users can use to submit, cancel and get information abouttasks. (implemented)

• Data consistency: In a state of failure, data can’t be left in aninconsistent state and have to be rolled back. (verified)

5.1.2 Should have

• Grouping: User should have an option to schedule multipletasks in one request. (implemented)

1. https://www.productplan.com/glossary/moscow-prioritization/

25

5. Design

• Notification on state updates: Send a message when a task hastransitioned to a new state. (not implemented)

• Prevention against duplicate execution: The scheduler has toinvoke remote entity exactly once per task. (implemented)

5.1.3 Could have

• Graceful shutdown: If the scheduler is shutdown, it should stopaccepting new tasks andwait for existing ones to complete. (notimplemented)

5.1.4 Remote entity requirements

• REST endpoints for starting and cancelling the task.

• On the invocation to start a task,

– if the task starts successfully, return positive HTTP statuscode (200)

– if the task fails to start, return negative HTTP status code(400)

• Inform the scheduler on successful and failed execution bycallback given by scheduler on the first request.

5.2 Differences and similarities to JBoss MSC

The core of the implementation is inspired by design from the anal-ysis of JBoss MSC, which is considerably adjusted for working withtransactions, remote entities and clusterable cache.

5.2.1 Naming changes

The naming changes are introduced due to an unsuitable meaningoccurring when transitioning from MSC to the scheduler. In MSCServices can be compared to processes where the goal is to have alldependencies in a state of execution to use their resources. In contrast,the scheduler’s goal is to have all dependencies finished. This leads

26

5. Design

to a decision to change naming from Services to Tasks. As Task had ameaning in MSC’s context, another transition was needed. The ade-quate candidate is Job.

Changes:• Service -> Task• Task -> Job

5.2.2 What stayed

• API for Target, Registry, Container, Dependent and Controlleris mostly the same.

• Principles of States, Modes, Transitions and ControllerTasks.• Identification with ServiceName

5.2.3 What changed

• TaskController data moved to a separate entity (Task).• Data consistency is assured by transactions instead of synchro-

nized blocks.• TheTaskRegistry carries data in InfinispanRemoteCache instead

of ConcurrentMap• Dependency interface is removed. MSC used this interface to

place demands on dependencies to start, whereas this situationdoes not occur for the scheduler. (users place the demand)

• Removed ServiceRegistrationImpl.• Removed Service interface as that interface served for local exe-

cution. (moved to remote entities)• The Target supports requests for multiple Tasks.• StartTask renamed to InvokeStartJob, which invokes starting end-

point of remote entity.

27

5. Design

• StopTask renamed to InvokeStopJob, which invokes cancellingendpoint of remote entity.

• Jobs are not async due to a limitation placed by transactions.

• The Controller has two new methods accept() and fail() whichsignify positive or negative response from a remote entity.

• States, Jobs,Modes and Transitions have been remodelled.

IDLE

entry / setMode(Mode.NEVER)NEW

WAITING

RUNNING

entry / InvokeStartJobSTARTING

UP

entry / InvokeStopJobSTOPPING

FINAL

entry / DependencyStoppedJobSTART FAILED

entry / DependencyStoppedJobFAILED

entry / DependencyStoppedJobSTOP FAILED

entry / DependencySuccessfulJobSUCCESSFUL

entry / DependencyCancelledJobSTOPPED

orDependencyStoppedJob

setMode(Mode.ACTIVE)[unfinishedDeps > 0]

accept()

setMode(Mode.CANCEL)

setMode(Mode.CANCEL)

accept()

<<create>>

setMode(Mode.Active),dependencySucceeded()[unfinishedDeps <= 0 &&

mode = Mode.Active]

setMode(Mode.Cancel), dependencyStopped()

fail()

fail()

<<destroy>>

accept()

fail()

Figure 5.1: State-machine diagram of a Task

5.3 States, Transitions, Modes and Jobs

States, Transitions and Jobs described are taken from Figure 5.1.

28

5. Design

5.3.1 Modes

• IDLE: Controller does not attempt to start a Task.

• ACTIVE: Controller is actively trying to start a Task.

• CANCEL: Signal to a Controller to cancel the Task and its de-pendents recursively.

5.3.2 Jobs

• InvokeStartJob: Invokes start endpoint on a remote entity. Ac-cording to the status code in response, Job calls accept() or fail().

• InvokeStopJob: Invokes stop endpoint on a remote entity. Accord-ing to the status code in response, Job calls accept() or fail().

• DependencyStoppedJob: Calls Dependent.dependencyStopped() oneach dependent. The method sets the StopFlag to DEPEN-DENCY_FAILED.Additionally, calling dependencyStopped()willcause a transition to STOPPED state, which will force the de-pendent Task to create anotherDependencyStoppedJob. Therefore,executing one DependencyStoppedJob will recursively stop everydependent.

• DependencyCancelledJob: Calls Dependent.dependencyCancelled()on each dependent. The method sets the StopFlag to CAN-CELLED. It has the same propagating mechanism asDependen-cyStoppedJob.

• DependencySuccessfulJob: CallsDependent.dependencySucceeded()on each dependent. The method decreases the number of un-finishedDependencies by one. If the resulting number is 0, thedependent will attempt to start.

5.3.3 StageGroups

• IDLE: Task is unproductive and either waiting for dependenciesor is in IDLE Mode.

• RUNNING: Task is remotely active.

29

5. Design

• FINAL: Task is in a final state, and it cannot transition further.

5.3.4 States

• NEW: Initial state.

• WAITING: Task is waiting for a dependency to succeed.

• STARTING: Task is attempting to start on a remote entity. Tran-sition into this state causes the invocation of start endpoint ona remote entity.

• START FAILED: Start on the remote entity failed. The Task hasto inform its dependents it failed, therefore it runs Dependen-cyStoppedJob.

• UP: Task is running on a remote entity.

• STOPPING: User has requested a cancellation of a task. TheTask is remotely active, therefore the scheduler has to informthe remote entity to terminate the Task. That means that thescheduler has to execute InvokeStopJob.

• STOP FAILED: Stopping on the remote entity failed. Thereforethe scheduler runs DependencyStoppedJob.

• FAILED: The Task failed to complete on the remote entity. Theremote entity invoked internal endpoint of the scheduler witha negative message. Therefore the scheduler runs Dependen-cyStoppedJob.

• SUCCESSFUL: The Task completes on the remote entity. Theremote entity invoked internal endpoint of the scheduler witha positive message. The Task needs to inform its dependents ithas succeeded.

• STOPPED: TheTaskwas informed to stop. Its direct dependencywas either cancelled or failed. In case the dependency failed,it runs DependencyStoppedJob, and if it was cancelled, it runsDependencyCancelledJob. (the decision is based on StopFlag)

30

5. Design

Figure 5.2: The diagram of package dependencies in the scheduler

5.4 Modules

Project hierarchy:

• common: Classes and enums shared across the whole project.

• core: The package that contains whole business logic and datas-tore communication. It exposes API in the form of interfacesfor other packages to use.

• model: Contents of the module are data models of entities, thatare saved in the datastore.

31

5. Design

• dto: Holds Data transfer object2 entities, used by REST for re-sponses and requests.

• facade: Connects rest and coremodules. Additionally, it includesmappers from the data model to the dto model and vice versa.

• rest: Exposes RESTful endpoints for users.

2. https://www.javaguides.net/2018/08/data-transfer-object-design-pattern-in-java.html

32

6 Implementation

This chapter goes through details of working with the libraries andframeworks such as Quarkus, CDI, Infinispan, JTA, JAX-RS and Map-struct. Furthermore, it mentions problems, obstacles and difficultiesencountered during implementation with a little fallback due to lack-ing documentation. It concludes with a guide on how to start up thescheduler.

6.1 Context dependency injection

Context dependency injection (CDI) is a standard from Java EE. De-pendency injection is a programming technique that makes a classindependent of its dependencies through delegation of object creationto a container. The container makes sure the dependencies are createdbeforehand and injects it to the dependent class. Therefore, the classhas available dependencies, and additionally, it enables the class to beloosely-coupled.

Quarkus provides its own implementation of the CDI frameworkby the name of ArC. The usual implementations of CDI like Weld,Google Guice and Spring are computing the dependencies in run-time.However, Quarkus shifts the computation to compile-time, whichintroduces a couple of difficulties. Firstly, the CDI standard is notimplemented to the fullest1. Secondly, the injection is quirky acrossMaven modules.

6.1.1 Maven module problem

The first version of scheduler was designed with a rest, facade andendpoint packages being in separate maven modules. Modules werelinked by dependency injection. As Quarkus needs every dependencydefined during build-time, it prevents any misconfiguration. Nonethe-less, after a successful build, the scheduler would get stuck on initial-ization without any logs displayed. Therefore, debugging the problemwas complicated and ultimately led to a decision tomerge themodulesinto one.

1. Limitations: https://quarkus.io/guides/cdi-reference#limitations

33

6. Implementation

6.2 Transactions

Transactions are vital to ensure data consistency across clusters. EachREST endpoint invocation is handled by a separate thread, whichmeans that evenwith a single node, there is a possibility for concurrentupdates of the same data.

An example being, there are two running tasks with a commondependent. Both complete at the same time, therefore their associatedremote entities send a notification. The notifications are accepted at thesame time by a single node, and a distinct thread handles each request.The common dependency is loaded from datastore and updated byeach thread. This produces a data inconsistency due to each threadmodifying data of the same origin and version. With transactions,one of the threads would get rolled back, because datastore can de-tect these inconsistencies. However, the rolled back notification aboutcompletion of a Task is necessary for the common dependency ever tostart. Therefore, there needs to be a mechanism which would retry theoperation again. The retried operation would retrieve updated dataand successfully commit.

The most significant benefit of solving transaction conflicts on athread level is that it does not matter from where a thread originates.It would be the same situation if the notifications were sent to differentnodes. One of the transactions would get rolled back, and the threadwould have to retry the operation.

6.2.1 Partial updates

The transaction boundaries have to be set in a way to avoid partial up-dates. For that reason, every request to the scheduler has to be atomic,which implies that either everything happens or nothing happens.

6.2.2 Prevention of duplicate execution

Duplicate execution could occur if a transaction that invoked a starton a remote entity failed and was consequently retried. As retryingrepeats all operations, it would invoke again. To prevent the incidentfrom happening, the invocation has to be separated from the trans-action and only take place if the transaction succeeds. The solution

34

6. Implementation

is to implement a unique type of a Job. This Job executes in a newthread and creates a separate transaction, but only if the transactionfrom a thread that this Job originated from succeeds. Propagatingstatus of the transaction to another thread was possible due to apply-ing Microprofile-context-propagation2 specification. Therefore, for aninvocation to happen, the underlying transaction has to succeed.

6.3 Mapping

Mapping fields from an object to another object can be done by hand orby using a mapping framework. Writing mappers by hand is tediousand error-prone. Additionally, if the schema of the object changes, thenumber of places that a programmer has to modify increases. It istypically more beneficial to leave the work to another library.

Most mapping frameworks like Dozer3 and ModelMapper4 usereflection to inspect the fields of an object, which is problematic withQuarkus. However, libraries like Mapstruct5 use an annotation pro-cessor to generate injectible mappers with just a standard Java.

Mapstruct defines a set of annotations by which you specify whichfield from one object maps to the other. If a field has the same nameand type on both sides, Mapstruct creates mapping automatically.Additionally, Mapstruct automatically maps around collections andnested classes, if there is a mapper available.

6.4 REST

The scheduler exposes two major endpoints, one is internal and servesas a mechanism for remote entities to inform the scheduler about Taskcompletion or failure, and the second is for users of the scheduler torequest scheduling, cancel Tasks and retrieve information about them.The endpoints are implemented with JAX-RS standard.

Additionally, the OpenAPI document is generated for the wholepublic API. It is possible due to an extension called smallrye-open-

2. https://github.com/eclipse/microprofile-context-propagation/3. https://github.com/DozerMapper/dozer4. http://modelmapper.org/5. https://mapstruct.org/

35

6. Implementation

api6 that Quarkus provides. It enables the user to use annotationson their endpoints, from which the OpenAPI document is produced.Furthermore, Quarkus exposes a Swagger UI7, that enables users tovisualize and use the endpoint directly through the browser. Internalendpoint:

• POST “/rest/internal/{TaskID}/finish”: The endpoint requiresa path parameter “TaskID” to identify, which Task is in con-text, and request body with a boolean signifying completion orfailure.

Task endpoint:• POST “/rest/tasks”: The purpose of the endpoint is to schedule

Tasks. The body of the request contains an array of TaskDTOs.Additionally, the scheduler can return 404 Bad Request if thearray creates a circle.

• GET “/rest/tasks”: The endpoint returns all tasks it has in thedatastore. There is an optional body, where the user can specifyfilters by StateGroup.

• GET “/rest/tasks/{TaskID}”: The endpoint returns informationabout specific Task identified by path parameter “TaskID”.

• PUT “/rest/tasks/{TaskID}”: The endpoint cancels the execu-tion of a Task specified in the path parameter.

6.5 Installation

This section describes necessary prerequisites, the process of settingup and configuring an Infinispan cache-store, and how to compile andconsequently run the scheduler.

6.5.1 Prerequisites

1. Apache Maven 3.5.3+ (to compile and build from the sourcecode).

6. https://github.com/smallrye/smallrye-open-api7. https://swagger.io/tools/swagger-ui/

36

6. Implementation

2. OpenJDK 1.8 or OpenJDK 11 of any distribution (OpenJDK 11prefered).

3. Infinispan Server of version 9.4.16.Final available for downloadfrom8.

6.5.2 Setting up an Infinispan server

To start up the scheduler, it has to have available datastore. Addition-ally, the data store has to be pre-configured with a definition of cachefor Tasks. To set up the server, follow these steps:

1. Download the infinispan-server-9.4.16.Final.zip.

wget <downloadable-link-for-infinispan-server>

2. Unzip the file.

unzip infinispan-server-9.4.16.Final.zip

3. Start the server

infinispan-server-9.4.16.Final/bin/standalone.sh-c clustered.xml

(to make it available on LAN, supply the host/ip-address withan additional option -b="ip-address", otherwise it defaults tolocalhost)

4. Wait until the server is operational.

5. Open another terminal on the same path.

6. Use the configuration script available in the scheduler’s reposi-tory. The script configures the needed cache, avoids potentialport conflicts with Infinispan REST server and restarts it.

infinispan-server-9.4.16.Final/bin/ispn-cli.sh--file="path/to/scheduler/server-config.cli"

8. https://downloads.jboss.org/infinispan/9.4.16.Final/infinispan-server-9.4.16.Final.zip

37

6. Implementation

6.5.3 Compilation and execution

During the Maven packaging phase, Quarkus creates a runner jar filewhich is executable with a java -jar command. To run the scheduler,follow these steps:

1. Open the scheduler repository directory in terminal.

2. Compile the scheduler without running the tests.

mvn clean install -DskipTests

3. Execute the scheduler.

java [-options] -jarcore/target/core-0.1-SNAPSHOT-runner.jar

Additional options:

(a) “-Dquarkus.infinispan-client.server-list=<ispn-ip-address>:11222”: Host/ip-address of Infinispan

Hot-Rod server. Default value is “localhost:11222”.(b) “-Dscheduler.baseUrl=<scheduler-url>”: URL address of the

scheduler which is used for callbacks sent to remote enti-ties. Default value is “http://localhost:8080/”.

(c) “-Dquarkus.http.port=<port>”: Port where the applicationis deployed on. Default value is “8080”.

38

7 Testing

Testing is a crucial part of software development. Tests ensure the im-plementation is behaving as expected, meets the requirements and pre-vents unexpected regression from appearing. This chapter describeslocal integration testing and tests done in a simulated cluster by hand.

7.1 Local integration testing

Integration tests are the second level of software testing behind unittests, which test individual components of the system. Integration test-ing is used to determine whether separately developed componentswork together as expected.

To define the tests, JUnit 51 testing framework is utilized. JUnitallows users to set the boundaries of the test and define commonpre-conditions and post-conditions.

Occasionally, tests tend to be complex and confusing, thereforeto achieve fluent and more comprehensible syntax, AssertJ library isused. Examples of AssertJ usage can be found here2.

Furthermore, Quarkus supports dependency injection in integra-tion tests, which makes the testing very convenient. The tests madefor the scheduler verify cases such as:

• simple datastore retrieval and store• correct Task format• dependencies triggering the execution of dependent tasks• Infinispan queries• cycles in a complex graph of Tasks• cancellation propagation to all dependents• execution of sophisticated Task graph ending up with all Tasks

in a successful stateTo run the integration tests, these actions are needed:

1. Start a configured Infinispan Server as described in subsection6.5.2

2. Open a new terminal.

1. https://junit.org/junit5/2. Examples: https://assertj.github.io/doc/

39

7. Testing

3. Change the directory to the repository with the scheduler.4. Build the scheduler and run the tests.

mvn clean install

7.2 Clustered testing

The first step for clustered testing is to set up a simulated clusterof nodes made of the scheduler. The nodes can be either on one ormultiple machines sharing a LAN. In the case of one machine, thetester needs to ensure that the schedulers are deployed on differentports. Furthermore, a separate project with an example of a remoteentity was created3. The entity has two endpoints, one notifies back thescheduler through a callback, and the other does not. Therefore, weare able to test the communication between the services and simulatethe scheduling of Tasks. These steps were taken to simulate a cluster:

1. Start a configured Infinispan Server as described in subsection6.5.2 and set the ip-address on server corresponding to themachine’s local network address.

2. Start several instances (on single or multiple machines) of thescheduler as described in subsection 6.4.4 and specify baseUrl,address of Infinispan server and HTTP port if required.

3. Choose a machine for a mocked remote entity.4. Clone the repository with the entity.

git clonehttps://github.com/michalovjan/mock-remote-entity.git

5. Enter the directory.cd mock-remote-entity

6. Compile the entity.mvn clean install

3. https://github.com/michalovjan/mock-remote-entity.git

40

7. Testing

7. Run the entity. (the entity deploys on port 8090 by default)

java -jartarget/mockremoteentity-1.0.0-SNAPSHOT-runner.jar

Concurrent updates on a single node

Testing of concurrent updates on a single node is done with arequest to schedule three Tasks, where one Task has two depen-dencies. The request must include the mocked remote entity.The invocations to start the Tasks and consequent responses areperformed at the same time. The responses create concurrenttransactions. One of them fails due to a write to the same place(common dependant). The failed transaction is rolled back andretried to ensure data consistency.

Concurrent updates on distinct nodes

Testing of concurrent updates on different nodes is done withthe same request mentioned above. The difference is that the re-mote entity does not inform the scheduler about the completionof a Task. The notifications are sent manually to different nodesat the same time, which can be done with "curl" command andtwo terminals. The behaviour is the same as on a single node.

41

8 Conclusion

The goal of this thesis was to create a general solution for remotetask dependency resolution with an ability to run in a cluster.The solution has to be able to prevent data inconsistencies ina concurrent environment, cancel a remote task and preventduplicate execution. Users have to have an option to access thescheduler through publicly available REST API. Additionally,the implementation should have been published as an open-source project.

To implement the solution, I’ve made thorough and detailedresearch of existing or partial solutions in the form of analyzesof JBossMSC, Infinispan and application platforms, which aredescribed in chapters two, three and four. According to theknowledge gained, I’ve designed a solution and subsequentlycreated an implementation. The implementation was testedand verified, that it conforms the requirements set by futureusers.Afterwards, the solutionwas published as an open-sourceproject on GitHub1.

8.1 Future improvements

Upper limit of running concurrent Tasks

To further configure the scheduler, there could be a cluster-wide upper limit for running Tasks. To implement this feature,I would use a clustered counter provided by Infinispan, thatprovides an atomic API for use in a concurrent environment.Next task to execute, could be chosen by randomly, by creationdate or by a custom algorithm with a heuristic.

1. https://github.com/michalovjan/remote-scheduler

43

8. Conclusion

Graceful shutdown

A graceful shutdown was mentioned in the requirements ascould-have. The feature could be implemented by a cluster-wide flag that could be triggered by any of the nodes in thecluster. The trigger would block incoming requests and preventfurther Tasks from executing.

Notification on state updates

It was mentioned as should-have in the requirements. For eachtransition, therewould be a new Job executed, whichwould sentthe information into message queue or web-sockets. It wouldbe up to users, to intercept the message and act accordingly.Additionally, the mentioned Jobwould have to have the sameprevention against duplicate execution as the InvokeStartJob had.

44

Bibliography

1. LEHMAN, E.; LEIGHTON, F.T.; MEYER, A.R. Mathematics forComputer Science. 2018. Available also from: https://courses.csail.mit.edu/6.042/spring18/mcs.pdf. Chapter 10.5 DirectedAcyclic Graphs and Scheduling.

2. Wildfly 18 - High Availability guide [online] [visited on 2019-12-12]. Available from: https : / / docs . wildfly . org / 18 / High _Availability_Guide.html.

3. Application Clustering For Scalability and High Availability [online][visited on 2019-12-12]. Available from: https://dzone.com/articles/application-clustering.

4. Stateless Over Stateful Applications [online] [visited on 2019-12-12]. Available from: https://medium.com/@rachna3singhal/stateless-over-stateful-applications-73cbe025f07.

5. DRAGONI, Nicola; GIALLORENZO, Saverio; LAFUENTE, Al-berto Lluch;MAZZARA,Manuel;MONTESI, Fabrizio;MUSTAFIN,Ruslan; SAFINA, Larisa. Microservices: Yesterday, Today, and To-morrow. In: Present and Ulterior Software Engineering. Ed. by MAZ-ZARA, Manuel; MEYER, Bertrand. Cham: Springer InternationalPublishing, 2017, pp. 195–216. ISBN 978-3-319-67425-4. Availablefrom DOI: 10.1007/978-3-319-67425-4_12.

6. API Documentation of JBoss MSC [online] [visited on 2019-12-09].Available from: https://jboss- msc.github.io/jboss- msc/apidocs/.

7. User and reference guide of Infinispan [online] [visited on 2019-12-10]. Available from: https://infinispan.org/docs/9.4.x/user_guide/user_guide.html.

8. Red Hat JBoss Enterprise Application Platform technology overview[online] [visited on 2019-12-11]. Available from: https://www.redhat.com/en/resources/resources-red-hat-jboss-enterprise-application-platform-technology-overview-html.

9. Thorntail Community Announcement on Quarkus [online] [visitedon 2019-12-11]. Available from: https://thorntail.io/posts/thorntail-community-announcement-on-quarkus/.

45

A Attached files

scheduler.zip

• A zip file containing maven-based source code of the schedulerand README.md file with instructions for installation.

• The source code open-source and also available on https://github.com/michalovjan/remote-scheduler.

47