Implementing Higher-Kinded Types in Dotty

Martin Odersky

Scala Symposium, October 2016, Amsterdam

joint work with

Guillaume Martres, Dmitry Petrashko

Dotty• The new Scala compiler developed at LAMP

• Based on DOT as a theoretical foundation

• Long-standing question: How to implement higher-kinded types on these foundations?

• The paper and talk investigate 4 (!) different solutions to do so.

Timeline

Simple Encoding

ProjectionEncoding

RefinementEncoding

DirectRepr.

Foundations: DOT

The DOT calculus is intended to be a minimal foundation of Scala.

Its type structure is a blueprint for the types used internally in the compiler.

DOT Terms• Translated to Scala notation, the language

covered by DOT is:Value v = (x: T) => t Function

new { x: T => d } Object

Definition d = def a = t Method definitiontype A = T Type

Term t = v Valuex Variablet1(t2) Applicationt.a Selection{ val x = t1; t2 } Local definition.

DOT TypesThe Types covered by DOT are:

Type T = Any Top typeNothing Bottom typex.A Selection(x: T1) => T2 Function{ def a: T } Method declaration{ type T >: T1 <: T2 } Type declarationT1 & T2 Intersection{ x => T } Recursion

DOT TypesThe Types covered by DOT are:

Type T = Any Top typeNothing Bottom typex.A Selection(x: T1) => T2 Function{ def a: T } Method declaration{ type T >: T1 <: T2 } Type declarationT1 & T2 Intersection{ x => T } Recursion

Should Scala have these?

DOT TypesThe Types covered by DOT are:

Type T = Any Top typeNothing Bottom typex.A Selection(x: T1) => T2 Function{ def a: T } Method declaration{ type T >: T1 <: T2 } Type declarationT1 & T2 Intersection{ x => T } RecursionScala has only refinements

T { d1 … dn}

with this as self reference.

Encoding Type Parameters

Lists, Encoded

A Variant of Variance

Variance, Encoded

=

Why Encode Parameterization?

• One fundamental concept (type member selection) instead of two.

• Clarifies interaction between parameterization and membership.

E.g.List[_ <: Number] = List { type T <: Number }

The Simple Encoding

Simple Encoding

Higher-Kinded TypesProblem: How to encode

type C[X] = List[X]

type C[+X <: Number] <: List[X]

?

Idea: Generalize definition of type parameters:• of a class: designated member types

• of an alias type: type params of its alias

• of an abstract type: type params of upper bound

The Simple EncodingProblem: How to encode

type C[X] <: List[X]

type D[X <: Number] = Cons[X]

?

This leads to:type C <: List

type D = List { type T <: Number }

The Simple EncodingProblem: How to encode

type C[X] <: List[X]

type D[X <: Number] = Cons[X]

?

This leads to:type C <: List List[_]

type D = List { type T <: Number } List[_ <: Number]

Note that these encodings can also be seen as wildcard (existential) types!

Limitations of the Simple EncodingUnfortunately, there are things that can’t be encoded that way:

type Rep[T] = T

type LL[T] = List[List[T]]

type RMap[V, K] = Map[K, V]

What To Do?

The Projection Encoding

ProjectionEncoding

The Projection EncodingIdea (originally due to Adriaan Moors):

Simulate type application using type selection.

E.g.type C[X] <: List[X]

becomestype C <: Lambda$P {

type $hk0type $Apply <: List { type T = $hk0 }

}

The Projection EncodingIdea (originally due to Adriaan Moors):

Simulate type application using type selection.

ThenC[String]

becomesC { type $hk0 = String } # $Apply

The Projection EncodingIdea (originally due to Adriaan Moors):

Simulate type application using type selection.

If we instantiate C to List, say, we get:type C = Lambda$P {

type $hk0type $Apply = List { type T = $hk0 }

}

The Projection EncodingHence,

C[String]

= C { type $hk0 = String } # $Apply

= Lambda$P { type $hk0; type $Apply = List { type T = $hk0 }

} { type $hk0 = String } # $Apply

=:=

List { type $hk0 = String }

= List[String]

Where “=:=” is interpreted as “being mutual subtype of”.

Problems with the Projection Enc.• There were several, but the killer was:

It relies on type projection T#U, which is unsound in general and will therefore be phased out.

• See the paper for a discussion in detail.

The Refinement Encoding

RefinementEncoding

The Refinement EncodingIdea:

Encodetype C[X] <: List[X] C[String]

astype C <: { z => List { type T = $hk0 }

type $hk0 }

C { type $hk0 = String }

This also had several problems which are described in the paper.

In the end…… we represented hk types directly, using special type forms for

• Type lambda: [X] -> T TypeLambda

• Higher-kindedtype application: C[T] HKApply

• A beta reduction rule: ([X] -> T)[U] à [X := U]T

• Rules for subtyping, type inference …

Implementation Data

Simple Encoding

ProjectionEncoding

RefinementEncoding

DirectRepr.

Total type-checker + core: 28’000 cloc

Evaluation• The simple encoding is a modest extension for a

language with wildcards or variance that wants to support basic concepts associated with hk types.• Cheap way to get a lot of the advantages of hk.

• Difficulty: Explain what’s legal and what is not.

• Other encodings were less successful than they looked at first.

• A direct representation is needed for robust support of full higher kinded types.

Still Missing• A theory that explains what the compiler did

in terms of DOT.

• Hopefully, we can find something that’s small and fits well.

A Take Away• Compiler hacking is a possible way to validate

ideas

• But it may not be the most efficient one.

• If we would have a better theory how higher-kinded types fit into DOT, we could have avoided some costly experiments.