Xtext beyond the defaults - how to tackle performance problems

Xtext beyond the defaults How to tackle performance problems

Who is that guy?

Xtext defaults are a good start!But one size does not fit all!


Amount of DSLs increases



Files get bigger



Files get bigger

Amount of files increases



Files get bigger


Many cross references



Files get bigger


Many cross references

Maybe transitive cross references…

Look what we have build!

Look what we have build!More than 20 DSLs…


and 200.000 lines in one file…


and 200.000 lines in one file…and we can check in broken models…



and we generate the missing stuff…



and we generate the missing stuff…to make them valid again!



We are on Xtext 2.6.2 and can’t update!

and we can check in broken models…and we generate the missing stuff…

to make them valid again!



We are on Xtext 2.6.2 and can’t update!

and we can check in broken models…and we generate the missing stuff…

to make them valid again!

Could you please back-port the changes?

Everything is sooo slow!!! Xtext sucks!!!

Finally think about improving the situation!

Update to newest Xtext Version

2.10.x

Update to newest Xtext Version

2.10.xXte

xt 2.1

1

releas

ed ea

rly 20

17

Why should I update???

Because…we killed bugs…

Because…we killed bugs…

Let wo

rkarou

nds

not la

st for

ever!

…Speed…

…Speed…

Resolv

ing cr

oss-

refere

nces h

as

never

been f

aster…

…new features…

…new features…

Make

Xtext

work

with o

ther

platfo

rms

…let’s work together.

…let’s work together.

It wa

s neve

r easi

er

to con

tribute

!

We are

on G

ithub

:-D

What about Performance…

Most problems happen in…

…Index and scoping…

…Memory consumption.

Validations…

… Complex generators…

I need Xbase in every language…

I need Xbase in every language…… because my language is

complicated…


complicated… … no I can’t explain why…


complicated… … no I can’t explain why…

… but there is this small corner case…

Is a FAST validation really fast…

Is a FAST validation really fast…… it’s triggered every time I stop typing

Scoping is sooo complicated…

Scoping is sooo complicated…… I heard that in nearly every project…

Scoping is sooo complicated…… I heard that in nearly every project……and people stop improving things when

it work’s for the first time…


it work’s for the first time…… code is not touched any more because

who knows…


it work’s for the first time…… code is not touched any more because

who knows…… it worked before you touched it!

What’s the problem?LocalScope

GlobalScopeImportScopeTypeScope

Caches everywhereThe Index

Lazylinking

EcoreUtils.resolveAll should do the trick!

DerivedState

ContainerState

BatchLinkableResource

Scoping APIscope_Element_feature(Element element ,EReference ref)





Called by the PolymorphicDispatcher…


Called by the PolymorphicDispatcher……in a reflective way…



and if grammar changed it might not get called any more……in a reflective way…




…BECAUSE IT’S SLOW!




…BECAUSE IT’S SLOW!Implement

getScope(EObject context, EReference ref) if(context instanceof Element && ref == MyPackage.Literals.ELEMENT_FEATURE)

up to 10 %




…BECAUSE IT’S SLOW!Implement

getScope(EObject context, EReference ref) if(context instanceof Element && ref == MyPackage.Literals.ELEMENT_FEATURE)

up to 10 % Will not

work

with

Xbase

anywa

y…

Use caches in the ScopeProvider

Use caches in the ScopeProviderFor each cross reference the ScopeProvider gets asked…


…and it calculates the very same stuff over and over again…

For each cross reference the ScopeProvider gets asked…




…cache the calculated stuff and invalidate it when necessary!




…cache the calculated stuff and invalidate it when necessary!

The cache of the GlobalScope should be the index and nothing else!

Use the force of the Index

Use the force of the IndexHolds all lightweight representations of

referenceable elements and their resources

Use the force of the IndexHolds all lightweight representations of

referenceable elements and their resources

It’s a HashMap…

It’s super fast since Xtext 2.x and improved further

Holds all lightweight representations of referenceable elements and their resources

Use the force of the Index

It’s a HashMap…

It’s super fast since Xtext 2.x and improved further

It’s YOUR Index - you can put data in…!

Index - how to feed it

Index - how to feed it1. Builder picks up DSL files and loads them one by one


2. Creates ResourceDescriptions through ResourceDescriptionManager



3. ResourceDescriptionManager calls ResourceDescriptionsStrategy to create EObjectDescriptions




You have to bind a custom impl for the ResourceDescriptionsStrategy in the RuntimeModule




You have to bind a custom impl for the ResourceDescriptionsStrategy in the RuntimeModule

The default creates a EObjectDescription for EVERY Element that has a name…

IEObjectDescription create(String simpleName, EObject element, Map<String, String> userData)

Information hiding and userData







Save memory & time

UserData - what for?

UserData - what for?Validation

Don’t load the world to get informations - put them in the index

UserData - what for?Validation

Don’t load the world to get informations - put them in the index

Scoping Don’t resolve proxies for simple informations

Reference non Xtext models

Reference non Xtext modelsWe have our own GlobalScopeProvider that does the trick…


…we load everything and the GlobalScope contains the corresponding EObjectDescriptions…


…we load everything and the GlobalScope contains the corresponding EObjectDescriptions…

…but it’s slow and we cannot navigate those elements in the editor…

You can do better…

Reuse the Xtext infrastructure

Reuse the Xtext infrastructureBind a ResourceServiceProvider…

…so that you can put the elements in the index and make them navigable!

Reuse the Xtext infrastructureBind a ResourceServiceProvider…

…so that you can put the elements in the index and make them navigable!

GenericResourceServiceProviderUse

It will put everything that has a name in the index…… introduce your own ResourceDescriptionStrategy!

But the builder picks them up…and it takes ages!

Put the index in the jar

Put the index in the jarWhy calculating the index of a static set of files?

Put the index in the jar

The index is serialisable…

…put it in the jar and load it.

Why calculating the index of a static set of files?

Index in StandaloneIt’s sooo slow…

Index in StandaloneIt’s sooo slow…

…of course I have my custom Standalone impl…

Index in StandaloneResourceSetGlobalScopeProvider

Long time the

was the default.

Index in StandaloneResourceSetGlobalScopeProvider

Long time the

was the default.

Today we can do better but you need to use it and know that there is something new…

For Maven- and Gradleplugin it’s done.

Index in StandaloneFill the index and install it on the ResourceSet!

Load each resource and ask the ResourceDescriptionManager to create a ResourceDescription…

Index in StandaloneFill the index and install it on the ResourceSet!

Load each resource and ask the ResourceDescriptionManager to create a ResourceDescription…

Install it on the ResourceSet

Make big things run again and save your ass!

Up to 80% more faster in 10 minutes!

CodegeneratorsThey are sooo slow…


… of course I am using mwe2 to invoke it…


… of course I am using mwe2 to invoke it…… even in the IDE!

Incremental generators

Incremental generatorsThe builder triggers the generator for each resource…


…when a file changes the builder computes the impact and retriggers the generator for each affected file!



What about m:n scenarios?Solveable with a little effort - but it is worth the work!



What about m:n scenarios?Solveable with a little effort - but it is worth the work!

There is a ParallelBuilderParticipant……you have to bind it for your language…

… make sure all proxies are already resolved!

Memory consumption

Memory consumption

DisabledClusteringPolicyDo not unload resource…

Memory consumption

DisabledClusteringPolicy

DynamicResourceClusteringPolicy

Do not unload resource…

Unload resources when there is no free memory……can be configured.

The NodeModel is hugeStores the syntax tree with text and positions…


…is used to get the position of a specific element or feature…



…produces a lot of Strings in memory…



…produces a lot of Strings in memory……and during the builder runs you do not really need

it.

Switch off the NodeModel

Switch off the NodeModelThe parser creates it …


…it is used to install proxies - so you have to do that differently…



You want to have it when the editor is used…



You want to have it when the editor is used…… the load options should be an indicator…



You want to have it when the editor is used…… the load options should be an indicator…

…what are the implications?

ImplicationsYou cannot use the NodeModel in validations and

scoping any more…


scoping any more…

…Issues are marked at position 0,0…

…an EditorCallback should revalidate the content.


scoping any more…

…Issues are marked at position 0,0…

…an EditorCallback should revalidate the content.

THIS IS NOT A COMMON THING!

DO IT ONLY WHEN IT IS REALLY NECESSARY!

Order makes a difference

Order makes a differenceFiles might belong together…


…and might reference each other.


…and might reference each other.Putting them in an logical order for the builder…



… and they do not have be loaded more than once.



… and they do not have be loaded more than once.Sometimes it makes sense to not unload them at all.

Now you know!

Professional Support

Engineering

Xtext beyond the defaults - how to tackle performance problems