Silicon Valley JUG - How to generate customized java 8 code from your database

How to Generate Customized Java 8 Code From Your DatabasePer MinborgCTO, Speedment, Inc

Emil ForslundDeveloper, Speedment, Inc.

Every Decision a Developer Makes is a Trade-off

The best code is no code at all

Using Code Generation• Makes the code efficient and

short• Modifications are done once

and applied everywhere• Minimizes errors• “DRY” (Don’t Repeat

Yourself) vs. ”WET” (We Enjoy Typing)

• “Code your code”

But how can we control the generated code?

About UsPer Minborg• Founder of several IT companies• Lives in Palo Alto • 20 years of Java experience• 15+ US patents • Speaker at Java events• Blog: Minborg’s Java Pot

Emil Forslund• Java Developer• Lives in Palo Alto• 8 years of Java

experience• Speaker at Java events • Blog: Age of Java

Spire• Speedment Open Source mascot• Lives on GitHub • 2 years of mascot experience

Agenda• Problem Description• Code Generation in Database Applications

• How Does it Work?• Code Example

• Controlling the Code Generation• Generate Custom Classes• Add new Method to Existing Classes• Code Example

• Additional Features• Beyond Open Source• Questions & Answers





Do You Recognize This Code?Class.forName("org.postgresql.Driver");try (final Connection conn = DriverManager.getConnection( "jdbc:postgresql://hostname:port/dbname", "username", "password")) { // Database Logic Here... }

Why Creating DB-Apps is So Time Consuming• Even trivial database operations require a lot of

boilerplate code • Mixing SQL and Java is error-prone • ORMs require you to write annotated POJOs for

every table• Creating even a simple DB app can take hours

Open-Source Project Speedment• Stream ORM Java toolkit and runtime • Generate domain-model from the

database • No need for complicated

configurations or setup• All operations are type-safe• Data is accessed using Java 8 Streams• Business friendly Apache 2-license

Speedment on GitHub

Speedment Workflow

customers.stream() .filter(…) .filter(…) .map(…) .collect(toList());

Step 1: Generate Code

Step 2: Write Logic Step 3: Run Application

Step 4: Iterate

Tool

Artifacts• com.speedment:

• runtime• generator• tool• speedment-maven-plugin





So How Do the Generated Code Work?• Code is organized based on database structure• Hash-sums make sure user changes are not overwritten

If the DB structure changes, the code is

updated with the press of a button

Entities, Managers and Applications• An Entity represents a row in a table

• Is a POJO• Customer• CustomerImpl

• A Manager represents a table• Responsible for the CRUD operations• CustomerManager• CustomerManagerImpl

• An Application represents the entire project• Responsible for configuration and settings• SalesApplication • SalesApplicationBuilder

Querying the Database using Streams• Queries are expressed using the

standard Java 8 Stream API• Streams are analyzed to

produce high-performance queries

Expressing Queries as Streams

customers.stream() .filter(Customer.REGION.equal(Region.NORTH_AMERICA)) .filter(Customer.REGISTERED.greaterOrEqual(startOfYear)) .count();

Standard Stream API Generated Enum Constants

Only 1 value is loaded from DB

Full Type-Safety

SELECT COUNT('id') FROM 'customer' WHERE 'customer'.'region' = ‘North America’ AND 'customer'.'registered' >= ‘2016-01-01’;

Querying the Database using Streams

SELECT * FROM 'customer'

REGION.equal(NORTH_AMERICA)

REGISTERED.greaterOrEqual(2016-01-01)

count()

Source

Filter

Filter

Term.

Pipeline


SELECT * FROM 'customer'WHERE 'customer'.'region' = ‘North America’

REGION.equal(NORTH_AMERICA)


count()

Source

Filter

Filter

Term.

Pipeline


SELECT * FROM 'customer'WHERE 'customer'.'region' = ‘North America’

AND 'customer'.'registered' >= ‘2016-01-01’;


count()

Source

Filter

Term.

Pipeline


SELECT COUNT('id') FROM 'customer'WHERE 'customer'.'region' = ‘North America’

AND 'customer'.'registered' >= ‘2016-01-01’;

count()

Source

Term.

Pipeline


SELECT COUNT('id') FROM 'customer'WHERE 'customer'.'region' = ‘North America’

AND 'customer'.'registered' >= ‘2016-01-01’;Source

Pipeline

Expressing Queries as Streams// Gets the second page of customers in North America// sorted by name in the form of a JSON array

customers.stream() .filter(REGION.equal(Region.NORTH_AMERICA)) .sorted(NAME.comparator()) .skip(10) .limit(10) // JVM from here… .collect(toJson(encode.allOf(customers)))

[ {”id”:11, ”name”: …}, {…}, …]

Expressing Queries as Streams// Supports parallelism on custom executors// with full control of thread work item layout

customers.stream() .parallel() .filter(REGION.equal(Region.NORTH_AMERICA)) .forEach(expensiveOperatation());

Application Example• Total Count of Customers

• Located in North America• Registered This Year

Step 1: Getting Speedment using Maven

<plugin> <groupId>com.speedment</groupId> <artifactId>speedment-maven-plugin</artifactId> <version>3.0.3</version></plugin>

Generate source files based on database

Step 2: Initializing Speedment

SalesApplication app = new SalesApplicationBuilder() .withPassword("qwerty") .build();

CustomerManager customers = app.getOrThrow(CustomerManager.class);

These classes are generated automaticallyInstance is configured using Builder-pattern

A manager class is generated for every database table

Step 3: Querying Region fromWhere = Region.NORTH_AMERICA; Instant fromWhen = Instant.parse("2016-01-01");

long count = customers.stream() .filter(Customer.REGION.equal(fromWhere)) .filter(Customer.REGISTERED.greaterOrEqual(fromWhen)) .count();

Step 4: Output System.out.println( "A total of %d customers from %s " + "have registered this year.", count, fromWhere.name() );

Full Application public static void main(String… args) { SalesApplication app = new SalesApplicationBuilder() .withPassword("qwerty") .build();

CustomerManager customers = app.getOrThrow(CustomerManager.class);

Region fromWhere = Region.NORTH_AMERICA; Instant fromWhen = Instant.parse("2016-01-01");

long count = customers.stream() .filter(Customer.REGION.equal(fromWhere)) .filter(Customer.REGISTERED.greaterOrEqual(fromWhen)) .count();

System.out.println( "A total of %d customers from %s " + "have registered this year.", count, fromWhere.name() ); }

OutputPers-MacBook-Pro:~ pemi$ java –jar salesapplication.jar

A total of 354 customers from NORTH_AMERICA have registered this year.





Controlling the Code Generation• So far we have queried a database

with streams• We have used code generation to

create entities and managers• Can it be used for more?

What is Available out of the Box?• MVC oriented code generation• Modular design• Database domain model• JSON configuration (DSL)• Java language namer• Translator and TranslatorDecorator• Maven goals• Type mappers

MVC Oriented Code Generation• Model

• File, Class, Interface, Enum, Field, Method, Constructor,Type, Generic, Annotation, …

• View• Renders a model to Java (or another language)• Set code style using custom views

• Control• AutoImport, AutoEquals, AutoJavadoc, SetGetAdd, FinalParameters• Write custom controllers to automate recurring tasks

MVC Oriented Code Generation• Separation of concerns• Code generation is type safe

• Catch errors compile time• Discover methods directly in the IDE

• Reuse code segments and controllers

Modular Design

Database Domain Model• Project• Dbms• Schema• Table

• Column• PrimaryKey• ForeignKey• Index

List<Table> tables = project.dbmses() .flatMap(Dbms::schemas) .flatMap(Schema::tables) .collect(toList());

JSON Configuration (DSL){ "config" : { "name" : "sales",

"dbmses" : [{ "name" : "db0", "typeName" : "MySQL", "ipAddress" : "127.0.0.1", "username" : "root",

"schemas" : [{ "name" : "sales", "tables" : [ { "name" : "city" }, { "name" : "salesperson" } ] }] }] }}

Java Language Namer• Camel caser: converts from “some_db_name” to “someDbName”• Java naming conventions: user, User and USER• Detects keywords like “final”,”static” and escapes them• Detects collisions• Pluralizer: bag → bags, entity → entities, fish → fish

Translator and TranslatorDecorator• Translator

• Renders a DB entity like a Table to a new Class or an Interface

• TranslatorDecorator• Modifies an existing Class or Interface

Maven Goals• speedment:tool

• Launches the graphical tool• Allows customization of configuration model• Code generation

• speedment:generate• Code generation without launching the tool

• speedment:reload• Reloads database metadata without launching the tool

• speedment:clear• Removes all the generated classes (except manual changes) without launching the tool

Type Mappers• Controls how columns are implemented• Runtime conversion between Database and Java types

java.sql.Timestamp long

Generation vs. Templates• Separation of concerns• Easily change code style• Minimize maintenance• Maximize reusability

Generate a New Custom Class

1. Create a new Translator2. Model how the new class should

look3. Define a Plugin Class4. Include it in project pom.xml

Example: Generate a Point class

Step 1: Create a New Translator Classpublic class PointTranslator extends AbstractJavaClassTranslator<Project, Class> {

public final static TranslatorKey<Project, Class> POINT_KEY = new TranslatorKey.of(“generated_point", Class.class); public PointTranslator(Project document) { super(document, Class::of); }

@Override protected Class makeCodeGenModel(File file) { return newBuilder(file, getClassOrInterfaceName()) .forEveryProject((clazz, project) -> { // Generate Content Here }).build(); }

@Override protected String getClassOrInterfaceName() { return ”Point"; } @Override protected String getJavadocRepresentText() { return "A 2-dimensional coordinate."; }

}

Every translator is identified by a TranslatorKey

Name of generated class

Javadoc

Called every time the translator is invokedforEvery(Project|Dbms|Schema|Table|Column|…)

Step 2: The makeCodeGenModel - methodclazz.public_() .add(Field.of(“x”, int.class) .private_().final_() ) .add(Field.of(“y”, int.class) .private_().final_() ) .add(Constructor.of().public_() .add(Field.of(“x”, int.class)) .add(Field.of(“y”, int.class)) .add(“this.x = x;”, “this.y = y;” ) ) .add(Method.of(“getX”, int.class).public_() .add(“return x;”) ) .add(Method.of(“getY”, int.class).public_() .add(“return y;”) ) .call(new AutoEquals<>()) .call(new AutoToString<>());

Step 3: Define a Plugin Class

public class PointPlugin {

@ExecuteBefore(RESOLVED) void install(CodeGenerationComponent codeGen) { codeGen.put( Project.class, PointTranslator.POINT_KEY, PointTranslator::new ); }}

The key defined earlier

Will execute when Speedment is being initialized

How the translator is constructed

Step 4: Include it in Project pom.xml<plugin> <groupId>com.speedment</groupId> <artifactId>speedment-maven-plugin</artifactId> <version>3.0.3</version>

<dependencies> <dependency> <groupId>com.example</groupId> <artifactId>point-plugin</artifactId> <version>1.0.0-SNAPSHOT</version> </dependency> </dependencies>

<configuration> <components> <component>com.example.pointplugin.PointPlugin</component> </components> </configuration></plugin> This tells Speedment to load the plugin

Make sure our plugin project is on the classpath

Execute

/** * A 2-dimensional coordinate. * <p> * This file is safe to edit. It will not be overwritten by the code generator. * * @author company */public class Point { private final int x; private final int y;

public Point(int x, int y) { this.x = x; this.y = y; }

public int getX() { return x; }

public int getY() { return y; }

The following file is generated: public int hashCode() { int hashCode = 31; hashCode += 41 * x; hashCode += 41 * y; return hashCode; }

public Boolean equals(Object other) { if (this == other) return true; else if (other == null) return false; else if (!(other instanceof Point)) { return false; }

final Point point = (Point) other; return x == point.x && y == point.y; }

public String toString() { return new StringBuilder(“Point{”) .append(“x: “).append(x).append(“, “) .append(“y: “).append(y).append(“}”); }}

A More Concrete Example

1. Create a new Translator2. Model how the new class

should look3. Define a Plugin Class4. Include it in project pom.xml

Example: Generate an Enum of tables in the database

Step 1: Create a New Translator Classpublic class TableEnumTranslator extends AbstractJavaClassTranslator<Project, Enum> {

public final static TranslatorKey<Project, Enum> TABLES_ENUM_KEY = new TranslatorKey.of(“tables_enum", Enum.class); public TableEnumTranslator(Project document) { super(document, Enum::of); }

@Override protected Enum makeCodeGenModel(File file) { return newBuilder(file, getClassOrInterfaceName()) .forEveryProject((clazz, project) -> { // Generate Content Here }).build(); }

@Override protected String getClassOrInterfaceName() { return ”Tables"; } @Override protected String getJavadocRepresentText() { return "An enumeration of tables in the database."; }

}

Every translator is identified by a TranslatorKey

Name of generated class

Javadoc

Called every time the translator is invokedforEvery(Project|Dbms|Schema|Table|Column|…)

Step 2: The makeCodeGenModel - methodDocumentDbUtil.traverseOver(project, Table.class) .map(Table::getJavaName) .map(getSupport().namer()::javaStaticFieldName) .sorted() .map(EnumConstant::of) .forEachOrdered(clazz::add);

Step 3: Define a Plugin Class

public class TableEnumPlugin {

@ExecuteBefore(RESOLVED) protected void install(CodeGenerationComponent codeGen) { codeGen.put( Project.class, TableEnumTranslator.TABLES_ENUM_KEY, TableEnumTranslator::new ); }}

Step 4: Include it in Project pom.xml<plugin> <groupId>com.speedment</groupId> <artifactId>speedment-maven-plugin</artifactId> <version>3.0.3</version>

<dependencies> <dependency> <groupId>com.example</groupId> <artifactId>table-enum-plugin</artifactId> <version>1.0.0-SNAPSHOT</version> </dependency> </dependencies>

<configuration> <components> <component>com.example.tableenumplugin.TableEnumPlugin</component> </components> </configuration></plugin> This tells Speedment to load the plugin

Make sure our plugin project is on the classpath

Execute

/** * An enumeration of tables in the database. * <p> * This file is safe to edit. It will not be overwritten by the code generator. * * @author company */enum Tables { CITY, SALESPERSON;}

The following file is generated:

Generate all the Things• Gson Adapters• Spring Configuration Files• REST Controllers• and much more…

Ext Speeder





Add New Method to Existing Classes• So far we have

• Queried a database with generated classes

• Generated a custom class

• How do we modify the existing generation of classes?

Add New Method to Existing Classes• Fit Into Existing Class Hierarchy• Change Naming Conventions• Optimize Internal Implementations• Add Custom Methods

Add New Method to Existing ClassesExample: Add a getColumnCount method to generated managers

1. Create a new TranslatorDecorator class

2. Write code generation logic3. Add it to Speedment





Step 1: Creating a New Decorator Class

public class ColumnCountDecorator implements TranslatorDecorator<Table, Interface> {

@Override public void apply(JavaClassTranslator<Table, Interface> translator) { translator.onMake(builder -> { builder.forEveryTable((intrf, table) -> { // Code generation logic goes here }); }); }}

Step 2: Write Code Generation Logic

int columnCount = table.columns().count();

intrf.add(Method.of("getColumnCount", int.class) .default_() .set(Javadoc.of("Returns the number of columns in this table.") .add(RETURN.setValue("the column count")) ) .add("return " + columnCount + ";"));

Step 3: Add it to Speedmentpublic final class TableEnumPlugin { @ExecuteBefore(RESOLVED) protected void install(CodeGenerationComponent codeGen) { codeGen.put( Project.class, TableEnumTranslator.TABLES_ENUM_KEY, TableEnumTranslator::new ); codeGen.add( Table.class, StandardTranslatorKey.GENERATED_MANAGER, new ColumnCountDecorator() ); }}

Modify an existing translator key

Our new decorator

The same plugin class as we created earlier

ExecuteWhen the project is regenerated, a new method is added to each manager





Additional Features• Add GUI Tool components for

custom configuration• Extend the JSON DSL

dynamically with plugins• Automate your build

environment with Maven• Add custom data type mappers

Additional Features• Plugin a custom namer• Generate classes that are related to other domain models• Generate classes for other languages• Style the GUI Tool

Database Connectors

Open Source• MySQL• PostgreSQL• MariaDB

Enterprise• Oracle• Microsoft SQL server• DB2• AS400





Beyond Open-Source• Speedment Enterprise• Run your database queries orders of magnitude faster!

• 10x, 100x, 1 000x, 10 000x, …

• Use the same Stream API

How is This Possible?• No changes to user-written code• Alternative stream source• Data is stored in-memory• Generates optimized serializers

for offheap storage

In-JVM-Memory, JMH Benchmarks(*)Benchmark Mode Cnt Score Error UnitsSpeedmentBenchmark.findStartsWith avgt 200 ≈ 10⁻⁵ s/opSpeedmentBenchmark.findStartsWithSql avgt 200 ≈ 10⁻⁴ s/op

SpeedmentBenchmark.join avgt 200 0.476 ± 0.005 s/opSpeedmentBenchmark.joinSql avgt 200 5.174 ± 0.010 s/op

SpeedmentBenchmark.sort avgt 200 ≈ 10⁻⁶ s/opSpeedmentBenchmark.sortSql avgt 200 ≈ 10⁻⁴ s/op

SpeedmentBenchmark.scrollSorted avgt 200 ≈ 10⁻⁵ s/opSpeedmentBenchmark.scrollSortedSql avgt 200 24.661 ± 0.670 s/op

SpeedmentBenchmark.count avgt 180 ≈ 10⁻⁸ s/opSpeedmentBenchmark.countSql avgt 200 5.143 ± 0.012 s/op

(*) Preliminary results, Mac Book Pro, 2.2 GHz i7, 16GB, MySQL 5.7.16 standard with indexes on all relevant columns.

10x

>10x

100x

2,000,000x

500,000,000x

Speedment Enterprise Example

Courtesy of Extremely Heavy Industries

Free Enterprise TrialToday we have released version 1.1.1

Get a free trial! Leave your contact information.

Q&Awww.speedment.org

github.com/speedment/speedment

@speedment

www.speedment.com

Linkedin: Per MinborgEmil Forslund

http://www.speedment.org/

http://www.speedment.com/

Data & Analytics

Silicon Valley JUG - How to generate customized java 8 code from your database