Maven just won't die -- WTF is Takari?

WTF is Takari?Jason van Zylhttp://takari.io@takari_io

‣ Incremental build‣ Aggressive parallelization‣ Generations‣ Polyglot for Maven‣ Shell

Agenda

It’s too hard for organizations to efficiently deliver JVM-based software

The problem with delivery

What is an application and how do you reason about it? Why you might be here today?

Internal Components

External Services

3rd Party Components

Configuration

Machine

v1.0

Software Delivery Chain

4

2 3

1

5

6

Provisioner

Repository

IDE SCM CI

Build

Central

Agent Agent

Agent

7

Developer requests componentsfrom Repoman and the components

are delivered to the developer

Developer checks code into the SCM CI checks code out of the SCM

CI instructs build tool to perform a build

Build tool deploys componentsto repository manager

Provisioning system requests componentsand configuration from Nexus.

Provisioning system provisions apps andcomponents to the agents

managing runtimes

‣ Incremental build‣ Aggressive parallelization‣ Generations‣ Polyglot for Maven‣ Shell

Agenda

Incremental build

‣ m2eclipse is capable of operating on a single resource with a single Mojo call -- very efficient‣ We implemented incremental build for Maven on the CLI‣ We implemented a custom Guice scope for Maven’s core to @Inject what we call a BuildContext‣ We Implemented versions the Mojos for the standard lifecycle including and incremental CLI compiler based

on Eclipse JDT

Incremental build

We’ve supported incremental build in m2eclipse for quite a while

public class BuildAvoidanceDemo { private final BuildContext context;

private final File inputDir;

private final File outputDir;

private final Set<String> includes;

private final Set<String> excludes;

// performed automatically by the framework // // 1. delete all stale or orphaned output files produced during previous build(s) // 2. persist BuildContext state such that it can be used during the next build // 3. replay any messages not cleared from previous build(s) // 4. raise build error if any of the inputs processed in this or previous builds had SEVERITY_ERROR message

@Inject public BuildAvoidanceDemo(BuildContext context, File inputDir, Set<String> includes, Set<String> excludes, File outputDir) { this.context = context; this.inputDir = inputDir; this.includes = includes; this.excludes = excludes; this.outputDir = outputDir; }

public void execute() throws IOException { // determine all input files, regardless of their up-to-date status // in this particular case use basedir with ant-style includes/excludes patterns FileSet inputs = new FileSetBuilder(inputDir).addIncludes(includes).addExcludes(excludes).build();

// context.registerInputs registers specified inputs FileSets with the build context, which is necessary to // enable cleanup of stale outputs. it returns collection of input files that require processing for (File inputFile : context.registerInputs(inputs)) { // context.addProcessedInput marks beginning of processing of the input file // it clears all output files and messages associated with the input file for this build context.addProcessedInput(inputFile);

// mapping between input and output files is outside of scope of BuildContext API File outputFile = determineOuputFile(outputDir, inputFile);

// context.newOutputStream associates input and output files, which is necessary to determine input files // that require processing and to perform cleanup of stale output files. OutputStream os = context.newOutputStream(inputFile, outputFile);

try { // generation of output file contents is outside of scope of BuildContext API // likewise, Message object, if any, is application specific and is not part of BuildContext API Collection<Message> messages = generateOutput(inputFile, os);

// record error and warning messages generated during input file processing for (Message message : messages) { context.addMessage(inputFile, message.getLine(), message.getColumn(), message.getText(), BuildContext.SEVERITY_ERROR, null); } } finally { os.close(); } } }}

Aggressive parallelization

Upstream

Existing parallel mode

Upstream

Aggressive parallel mode

Upstream

Aggressive parallel modewith smart scheduling

Aggressive parallel modewith smart scheduling

Build Telemetry

Critical path

Generations & Workspaces

SCMrevision 100:branch X

Generation 100

Generation 100 Generation 101 Generation 102 Generation 103

SCMrevision 100:branch X

SCMrevision 101:branch X

SCMrevision 102:branch X

SCMrevision 103:branch X

As time passes and your product gets better (I hope)

How does a workspace relate to a generation?

Workspaces

jetty-server

javax.servlet

jetty-continuation

jetty-http

jetty-io

jetty-util

jetty-webapp

jetty-xml

jetty-util

jetty-deploy

jetty-webapp

jetty-util

javax.inject

provisio-webserver

jetty-jmx

jetty-util

jetty-util

What if generations were supported across the delivery chain?

4

2 3

1

5

6

Provisioner

Repository

IDE SCM CI

Build

Agent Agent

Agent

7

Consuming a new generation

Producing a new generation

Generations collaboration between build and repository manager

Polyglot for Maven

‣ A lot of work has been done recently on the Ruby and Scala DSLs, but there is support for Groovy, Atom, YAML, and Clojure as well

‣ We cut the 0.9.0 release yesterday and it’s available to try‣ The code and instructions are available at https://github.com/tesla/tesla-polyglot

Polyglot for Maven

Maven can support different formats and even DSLs

project 'Polyglot Tesla :: Aggregator' do

model_version '4.0.0' id 'io.tesla.polyglot:tesla-polyglot:0.0.1-SNAPSHOT' inherit 'io.tesla:tesla:4' packaging 'pom'

properties( 'sisuInjectVersion' => '0.0.0.M2a', 'teslaVersion' => '3.1.0' )

modules [ 'tesla-polyglot-common', 'tesla-polyglot-atom', 'tesla-polyglot-ruby', 'tesla-polyglot-groovy', 'tesla-polyglot-yaml', 'tesla-polyglot-clojure', 'tesla-polyglot-scala', 'tesla-polyglot-cli', 'tesla-polyglot-maven-plugin' ]

overrides do jar 'org.eclipse.sisu:org.eclipse.sisu.inject:${sisuInjectVersion}' jar 'org.eclipse.sisu:org.eclipse.sisu.plexus:${sisuInjectVersion}' jar 'org.apache.maven:maven-model-builder:3.1.0' jar 'org.apache.maven:maven-embedder:3.1.0' jar( 'junit:junit:4.11', :scope => 'test' )

end

plugin 'org.codehaus.plexus:plexus-component-metadata:1.5.4' do execute_goals 'generate-metadata', 'generate-test-metadata' end

build do execute("first", :validate) do |context| puts "Hello from JRuby!" end end

end

project { modelVersion '4.0.0' parent('io.tesla:tesla:4') groupId 'io.tesla.polyglot' artifactId 'tesla-polyglot' version '0.0.1-SNAPSHOT' packaging 'pom' name 'Polyglot Tesla :: Aggregator' modules([ 'tesla-polyglot-common', 'tesla-polyglot-atom', 'tesla-polyglot-ruby', 'tesla-polyglot-groovy', 'tesla-polyglot-yaml', 'tesla-polyglot-clojure', 'tesla-polyglot-scala', 'tesla-polyglot-cli', 'tesla-polyglot-maven-plugin', ]) properties { sisuInjectVersion '0.0.0.M2a' teslaVersion '3.1.0' } dependencyManagement { dependencies { dependency('org.eclipse.sisu:org.eclipse.sisu.inject:${sisuInjectVersion}') dependency('org.eclipse.sisu:org.eclipse.sisu.plexus:${sisuInjectVersion}') dependency('org.apache.maven:maven-model-builder:3.1.0') dependency('org.apache.maven:maven-embedder:3.1.0') dependency('junit:junit:4.11:test') } } build { $execute(id: 'hello', phase: 'validate') { println "" println "hello, I am Groovy inside Maven. What? What am I doing here?? I thought he was my arch nemesis! I'm confused." println "" } plugins { plugin('org.codehaus.plexus:plexus-component-metadata:1.5.4') { executions { execution(goals: ['generate-metadata', 'generate-test-metadata']) } } } }}

import org.sonatype.maven.polyglot.scala.model._ implicit val scalaVersion = ScalaVersion("2.10.2") ScalaModel( "" % "tesla-polyglot-scala", name = "Polyglot Tesla :: Scala", contributors = Seq( Contributor( name = "Christopher Hunt", organization = "Typesafe", organizationUrl = "http://typesafe.com" ) ), parent = Parent("io.tesla.polyglot" % "tesla-polyglot" % "0.0.1-SNAPSHOT"), repositories = Seq( Repository( snapshots = RepositoryPolicy(enabled = false), id = "sonatype-public-grid", url = "http://repository.sonatype.org/content/groups/sonatype-public-grid/" ) ), dependencies = Seq( "io.tesla.polyglot" % "tesla-polyglot-common" % "0.0.1-SNAPSHOT", "com.twitter" %% "util-eval" % "6.3.8", "com.googlecode.kiama" %% "kiama" % "1.5.1", "org.specs2" %% "specs2" % "2.1.1" % "test", "junit" % "junit" % "" % "test" ), build = Build( plugins = Seq( Plugin( "io.tesla.maven.plugins" % "tesla-license-plugin", configuration = Config( header = "../license-header.txt" ) ) ), tasks = Seq(Task("someTaskId", "verify") { ec => println("I'm Scala running during the verify phase. The ec passed in allows me to access the project") }) ), modelVersion = "4.0.0")

modelVersion:  4.0.0

parent:  {artifactId:  tesla,  groupId:  io.tesla,  relativePath:  ../pom.xml,  version:  '4'}groupId:  io.tesla.polyglotartifactId:  tesla-­‐polyglotversion:  0.0.1-­‐SNAPSHOTpackaging:  pomname:  'Polyglot  Tesla  ::  Aggregator'

properties:  {sisuInjectVersion:  0.0.0.M2a,  teslaVersion:  3.1.0}

build:    plugins:    -­‐  artifactId:  plexus-­‐component-­‐metadata        executions:        -­‐  goals:  [generate-­‐metadata,  generate-­‐test-­‐metadata]            id:  default            inherited:  true            priority:  0        extensions:  false        groupId:  org.codehaus.plexus        inherited:  true        version:  1.5.4

modules:  [    tesla-­‐polyglot-­‐common,      tesla-­‐polyglot-­‐atom,      tesla-­‐polyglot-­‐ruby,      tesla-­‐polyglot-­‐groovy,    tesla-­‐polyglot-­‐yaml,      tesla-­‐polyglot-­‐clojure,      tesla-­‐polyglot-­‐scala,      tesla-­‐polyglot-­‐cli,    tesla-­‐polyglot-­‐maven-­‐plugin]

dependencyManagement:    dependencies:    -­‐  {artifactId:  org.eclipse.sisu.inject,  groupId:  org.eclipse.sisu,  optional:  false,  type:  jar,  version:  '${sisuInjectVersion}'}    -­‐  {artifactId:  org.eclipse.sisu.plexus,  groupId:  org.eclipse.sisu,  optional:  false,  type:  jar,  version:  '${sisuInjectVersion}'}    -­‐  {artifactId:  maven-­‐model-­‐builder,  groupId:  org.apache.maven,  optional:  false,  type:  jar,  version:  3.1.0}    -­‐  {artifactId:  maven-­‐embedder,  groupId:  org.apache.maven,  optional:  false,  type:  jar,  version:  3.1.0}    -­‐  {artifactId:  junit,  groupId:  junit,  optional:  false,  scope:  test,  type:  jar,  version:  '4.11'}

project "Polyglot Tesla :: Aggregator" @ "" as pom id: io.tesla.polyglot:tesla-polyglot:0.0.1-SNAPSHOT inherits: io.tesla:tesla:4 properties: [ sisuInjectVersion: "0.0.0.M2a" teslaVersion: "3.1.0" ] overrides: [ org.eclipse.sisu:org.eclipse.sisu.inject:${sisuInjectVersion} org.eclipse.sisu:org.eclipse.sisu.plexus:${sisuInjectVersion} org.apache.maven:maven-model-builder:3.1.0 org.apache.maven:maven-embedder:3.1.0 junit:junit:4.11 ] modules: [ tesla-polyglot-common tesla-polyglot-atom tesla-polyglot-ruby tesla-polyglot-groovy tesla-polyglot-yaml tesla-polyglot-clojure tesla-polyglot-scala tesla-polyglot-cli tesla-polyglot-maven-plugin ]plugin id: org.codehaus.plexus:plexus-component-metadata:1.5.4

Mavenize like a champ.Introduction to Maven Virtual TrainingThursday, February 13, 20141:00PM-7:30PM EST (UTC -5 hours)575CAD/studentSmall classes, an accomplished instructor, top-notch training…all from the comfort of your own pajamas office. Learn more at www.takari.io