Ahead of Time Dynamic Translation PreJit/NGEN by any other name George Bosworth Microsoft MRE04 March 21, 2004

Ahead of Time Dynamic Translation

PreJit/NGEN by any other name

George BosworthMicrosoft

MRE’04March 21, 2004

2

Dynamic Translation

• What is it? – a compiler (not)

• What does it do? – produce good fast code (hopefully)

• When does it run? – Before the code is executed. i.e ahead of time.

How much before is the subject of this talk

3

Outline

• Code Generation and execution in the Common Language Runtime (CLR)

• PreJit– What is it, why do it, at what cost– What is it really, and what is hard vs

easy

• Simple example– What the resultant code could look like

4

Common Language Runtime

Co

mm

on

Lan

gu

age

Ru

nti

me

Co

mm

on

Lan

gu

age

Ru

nti

me

FrameworksFrameworks

Class loader and layoutClass loader and layout

IL t

o

IL t

o

nat

ive

cod

e n

ativ

e co

de

com

pile

rsco

mp

ilers

GC, stack walk, code managerGC, stack walk, code manager

Sec

uri

tyS

ecu

rity

Exe

cuti

on

Exe

cuti

on

Su

pp

ort

Su

pp

ort

Base ClassesBase Classes

5

Runtime Control FlowJust in Time

ClassClassLoaderLoader

IL to nativeIL to nativecode compilercode compiler

CPUCPUSecuritySecuritySystemSystem

CodeCodeManagersManagers

ManagedManagedNativeNativeCodeCode

AssemblyAssembly

First call to First call to methodmethod

First First reference to reference to typetype

ExecutionExecutionSupportSupport

6

Execution ModelVBVB VCVC ...... ScriptScript

ILILNativeNativeCodeCode

““Econo”-JITEcono”-JITCompilerCompiler

Standard JITStandard JITCompilerCompiler

NativeNativeCodeCode

PreJit timePreJit timeCode GenCode Gen

Common Language RuntimeCommon Language Runtime

7

Compiling IL To Native• “Econo” JIT

– Generates unoptimized native code– Code can be discarded and regenerated

• “Standard” JIT– Generates optimized native code– Includes verification of IL code

• PreJit code generation– Done at install time– Reduces start-up time– Native code has version checks and reverts

to runtime JIT if they fail

8

Assemblies• Unit of deployment

– One or more files, independent of packaging– Self-describing via metadata (“manifest”)

• Versioning– Captured by compiler– Policy per-application as well as per-machine

• Security boundary– Assemblies are granted permissions– Methods can demand proof that a permission

has been granted to entire call chain• Mediate type import and export

– Types named relative to assembly

9

What Happens at Compile Time?• Compiler reads any imported metadata and builds

internal symbol table• Compiler resolves method overloading using language-

specific type matching rules– Compiler adds any required coercions or casts– Compiler selects and records a precise member reference

• Compiler provides object layout information through its choice of– Layout technique– Non-static fields– Virtual methods, with “new slot” vs “use existing slot”

• Compiler emits metadata by merging incoming metadata with metadata it has generated

• Compiler emits IL with metadata tokens

10

What Happens at Class Load Time?, I

• A type reference occurs and is resolved either to an existing type or to an assembly reference.

• The assembly is loaded and some validation checks occur.

• The assembly is asked to resolve the type reference and a module is located– It is at this point that the security system runs

inheritance permission checks

11

What Happens at Class Load Time?, II

• If no managed native code is available, the IL module is loaded and preliminary validation checks occur.

• The particular class is then validated and the CLR creates in-memory data structures representing the class, including thunks for use when a method is called the first time.

• When a method on the class is called the first time, the code is JITted (and verified if needed).

12

What Happens at JIT Time?• If a member reference appears in the code, the

type is first loaded (see below), which will also load classes from which it inherits.

• If the type does not itself have the member, the inheritance chain is searched to see if it can be resolved higher up in the class hierarchy.

• The member access check, including a possible security check, occurs at this time.

• The JITter may insert calls to security code for declarative security checks in addition to those specified using the imperative security mechanism.

• The code runs.

13

Unmanaged (Traditional) Code

DEPLOYMENTDEPLOYMENT

app. directory, app. directory, Win32 directoryWin32 directoryapp. directory, app. directory, Win32 directoryWin32 directory

public static void Main(String[] args ){ String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(“C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); }} CompilerCompiler

DEVELOPMENTDEVELOPMENT

public static void Main(String[] args ){ String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(“C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); }}

Source codeSource code PE header + PE header + EH Table +EH Table +Native codeNative code

(incl. object layout)(incl. object layout)

PE header + PE header + EH Table +EH Table +Native codeNative code

(incl. object layout)(incl. object layout)

EXECUTIONEXECUTION

Native codeNative codeEH tableEH table

Native codeNative codeEH tableEH table

ServicesServicesExceptionException

14

Managed Code Execution

PEVerifyPEVerify

NGENNGEN


GAC, GAC, app. directory, app. directory,

download cachedownload cache






Source codeSource code AssemblyAssemblyPE header + MSIL + PE header + MSIL + Metadata + EH TableMetadata + EH Table

AssemblyAssemblyPE header + MSIL + PE header + MSIL + Metadata + EH TableMetadata + EH Table

EXECUTIONEXECUTIONAssembly infoAssembly info

Module Module + Class list+ Class list

Assembly infoAssembly infoModule Module

+ Class list+ Class listPolicy Policy

ManagerManager

HostHost

Policy<?xml version="1.0" encoding="utf-8" ?><configuration> <mscorlib> <security> <policy> <PolicyLevel version="1"> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="Nothing" Name="All_Code" Description="Code group grants no permissions and forms the root of the code group tree."> <IMembershipCondition class="AllMembershipCondition" version="1"/> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust"


Granted Granted permissionspermissions


Vtable +Vtable +Class infoClass infoVtable +Vtable +

Class infoClass infoJIT +JIT +

verificationverificationNative codeNative code+ GC table+ GC table

Native codeNative code+ GC table+ GC table

CLR ServicesCLR ServicesGCGCExceptionExceptionClass initClass initSecuritySecurity

(assembly

(assembly))

((classclass))

((methodmethod))

AssemblyAssemblyLoaderLoader

EvidenceEvidence

Permission requestPermission request

15

PreJit: What Is It?An MSIL-to-Native Compiler (Not)

• Pre-compile, pre-load, pre-layout– Validate at execution time

• May factor in processor type, other installed assemblies, etc.

– If invalid, use the normal execution path

• Goal: faster start-up – Because of smaller start-up working set

• Usually as part of assembly installation

ngen.exe, ships in Visual Studio® .Net

16

What Happens at PreJit Time?

• Record version information about any dependents

• Classes are laid out• Internal runtime data structures allocated• Generate managed native code, verifying

as needed• Emit a specially formed PE file (.exe, .dll)

in the code cache and associate it with the input IL assembly

17

NGen Code Creation

PEVerifyPEVerify

NGENNGEN









Source codeSource codeAssemblyAssembly

PE header + MSIL + PE header + MSIL + Metadata + EH TableMetadata + EH Table

AssemblyAssemblyPE header + MSIL + PE header + MSIL + Metadata + EH TableMetadata + EH Table

EXECUTIONEXECUTIONAssembly infoAssembly info

Module Module + Class list+ Class list

Assembly infoAssembly infoModule Module

+ Class list+ Class list

Policy Policy ManagerManager

HostHost





Vtable +Vtable +Class infoClass infoVtable +Vtable +

Class infoClass infoJIT +JIT +

verificationverificationNative codeNative code+ GC table+ GC table



((assembly

assembly))

((classclass))

((methodmethod))


EvidenceEvidence

Permission requestPermission request

18

EXECUTIONEXECUTIONAssembly info

Module + Class list

NGen Code ExecutionDEPLOYMENTDEPLOYMENT





PrebuiltPrebuilt(check(check

Version)Version)

PrebuiltPrebuilt(check(check

Version)Version)

Policy Policy ManagerManager

HostHost



Vtable +Class info

JIT +verification




((assembly

assembly))

((classclass))

((methodmethod))


Permission requestPermission requestGranted

permissionsPrecomputedPrecomputedpermissionspermissions

(check)(check)

PrecomputedPrecomputedpermissionspermissions

(check)(check)

PrebuiltPrebuiltVtable +Vtable +

Class infoClass info

PrebuiltPrebuiltVtable +Vtable +

Class infoClass info

Fixups /Fixups /RelocsRelocs(or JIT)(or JIT)

19

Deployment Model

• Shipping and installation– Compile IL in build lab – Ship IL– PreJit during installation (optional)– Possible JIT during execution

• Patching– Deliver new IL image– Repair native images (all depending

assemblies)

20

Why Add Complexity of PreJit?

• Performance– Remove expensive post-deployment work from app startup

• Signature validation

• Cross-component validity checks

• Type safety verification

• Optimization (distribution -> execution)– Code generation

– Data structures

– Packing

– Disk-based images allow sharing data/code across processes and across machine runs

21

Performance Gains

• Warm startup time – Up to 6x faster

• Private pages– Reduced by as much as 2/3

• Startup working set– Reduced by as much as 1/2

• Basis– Whidbey (the upcoming release of the CLR)– Reference WinForm app with 100s of controls

Caveats: your mileage may very and these are very preliminary based on pre-release builds and hence subject to change

22

Costs • Disk space

– Typical prejit image is ~2-2.5x IL size– Stored in Native Image Cache (alongside GAC)

• Installation time– Typically ~3s per MB of IL– 40s to prejit Framework assemblies

• Steady state performance– Slowdown due to indirections in code– ASP.net

• V1: PreJit 7-12% slower• Whidbey: PreJit 2-5% slower

– Bytemark• Whidbey: same performance

– Future: PreJit will have better performance• Recompilation

23

Scenarios

• Critical: libraries– Loaded into multiple processes

• Warm startup is the norm• Maximize shared pages

– Used by OS like components

• Important: large apps– Large percentage of code is app code– Large impact on machine– Benefit is primarily startup time

• Important: shared apps– Terminal server– Shared pages are key

24

Scenarios

• Questionable: small single instance apps and addins– ~1MB working set improvement– ~15% startup improvement– Relatively small amount of compilation overhead – Likely to be numerous– Recommendation: do not ngen unless heavily used

• Not helpful: transient code– Reflection emit– “Run and (probably) throw away” code– Frequently updated apps (relative to execution)– Code under development (edit-compile-debug)

• Assumption: some processes will be jitting some assemblies

25

PreJit Architectural Background

• IL was designed as CLR deployment format– Verifiable type safety– Small– Self describing– Portable

• PreJit image was designed as internal format– Designed for execution speed

• Working set rather than disk size• Data structures paged off of disk & used directly

– Highly dependent on CLR implementation• Data structure snapshots• Complex codegen requirements• High rate of churn over time

– Specialized for particular scenario

26

Specialization and Recompilation• PreJit images are specialized translations of an IL assembly for a

particular environment– Processor– Version of CLR– Versions of assembly dependencies– Development scenario (debug, profile, IBC instrument)– Other factors

• Processor features• Security policy• Hosting hooks (SQL, IE,???)• AppDomain shareable (state access)• Profiling data

• Different environments require different prejit images– Different machines may have different environments– Servicing can change CLR or assembly dependencies– Environmental changes can invalidate prejit images– App-level configuration can change requirements across apps

• Recompilation is needed to adjust available prejit images over time to adapt to a changing environment

27

Client code generation motivation

• IL Distribution– Verifiable (limited trust)– Small– Self describing

• Specialized native code – Reach

• Processor• CLR Implementation

– CLR Version– Platform (assemblies) Version

• Recompilation

Brittleness okay

Rebuilt as

needed

28

Why is specialization important?

• Innovation– Managed APIs and implementation– CLR implementation

• Compatibility– Application needs to deploy on multiple OS’s– OS update must retain application

compatibility

• Plus performance (of course)

29

Framework evolution & recompilation• We want to be able to recompile application native code as frameworks (platform

assemblies) evolve• Flexible compatible update rules

– Add public stuff– Change any non-public stuff– Rules based on semantics – no implementation driven constraints

• Rearrange and add fields to public structures & classes• Rearrange and add virtual methods• Insert into inheritance heirarchy

– Non-compatible changes are possible and handled relatively gracefully• Versionability is a fundamental assumption when designing managed APIs

– Implementation inheritance• Each layer up the hierarchy may add fields, methods, virtuals• All are rolled into single flat structure• All can version independently from each other and from client

– Typesafe extensibility• Public fields on a structure or class• Set can be extended as needed• Example: delegates on form, properties on window

– Fine grained, version-resilient component interaction• Properties• Can protect access and future evolution with a function call but still get efficiency

– Raises the bar for API quality• Uniform mechanisms with language support• Patterns can be enforced with tools

30

CLR evolution & recompilation• We want to be able to recompile native code as CLR evolves• Much of runtime implementation is “baked” into code

– Code sequences• Helper calls, Dispatch sequences, Global variable access

– Runtime data structures• Highly specialized, Slow to build from IL, Likely to change over time

– Invariants• Garbage collection, Lazy initialization

• Recent Changes– Interface/Virtual dispatch– Working set tuning– Single file loading– Generics– And on and on…

• Future potential– More data structure tuning– Phoenix

The CLR is a great vehicle for delivering improvements into Windows

31

Future Directions

• Phoenix– Changes prejit equation

• Higher compile cost• Better perf for prejit vs JIT

– More cross-assembly analysis

• Client compilation optimizations– Profile data feedback– Active base address management

• Enable some additional specialization conditionally – Allow developer to dial conservative behavior vs. aggressive

optimizations

32

Why it’s Not So Easy

let me count the ways

33

Issue: Compilation Overhead

• Installation cost• Recompilation cost

– Cascaded low level changes have very high cost • Image churn can have secondary costs (OS

optimizations, virus scan)• Goals:

– Ensure compilation performance is well tuned– Minimize recompilation

• Minimize factors which cause recompilation• Allow limited assembly servicing without cascading recompilation

– Ensure that most low-level servicing falls in this limited category

– Be smart about deferring compilation work– However, some compilation cost is fundamental

34

Issue: Managed Watson Dumps

• Native code and CLR data structures are generated on user machine, not available in any shipped dlls

• Much of this data is required to do a stack trace• IL<->Native mapping info is typically not available unless

debug code is in use– Source mapping is described in terms of IL

• Goals:– Leverage repeatability of codegen to optimize ngen image

contents out of process dump• Result is faster dump collection and smaller dumps

– Also leverage repeatability to regenerate IL<->native mapping info post mortem

35

Cross-assembly optimizations

• These optimizations expose internal implementation details to other assemblies’ code generation

• Examples– Method inlining– Constant value inlining– Persisted instance data created at compile time– Class constructors– Struct enregistration

• Likely to be much more in Phoenix

36

NGen Fixup Blobs• Descriptive reference to something in another module• Blobs are grouped by referenced module?

– Module level table describes ranges of blobs– Each table entry has an assembly ref token + file ref token

• Could roughly follow metadata signature byte encoding?– Byte sized tags with fixed following values– Variable length encoded numbers & tokens

• Treat blob is a type signature?– However, scope is in destination assembly– This avoids symbolic lookups, however introduces fragility if token numbers change

• Other possible blob formats– MODULE (no args)– TYPE + type signature– METHOD + <vtable index> + containing type signature– FIELD + <field index> + containing type signature– STRING + <string token>– SIGNATURE + <signature token>

• Blobs are read-only? & interned within a module?• Typically described with an RVA?

DetailsDetailsDetails

37

NGen Fixup Tables• Set of tables which provide fixup cells for code and data to

reference• Header describes ranges of tables?• Table entry initially contains tagged RVA of fixup blob?• Table index implies type of fixup

– Handle (runtime data structure)– Method call address– Static field address– Varargs signature– Others?– Side effects? Which ones? (e.g. type initializations?)

• Entries are fixed up lazily?– All fixups are idempotent?, tag allows atomic update– Eager update is too expensive?– Fixup can fail?

And they never end

38

But when it works

it can work well

An example(take with a grain of salt)

39

Example

Assembly A (a.cs) public class A { static public int a() { ... }}

Assembly B (b.cs) public class InheritFromA : A{ static public int a1() { ... }}

public class B{ public static void b() { A a = new A(); A.a(); InheritFromA.a1(); }}

40

Example: Naive

b() has 4 fixups & requires fixup guarantee before execution{

A a = new A();20002552 mov ecx,[20006180] // Fixup on A's method table pointer20002558 call CLRStub@200040e0 // Call to new2000255d mov ecx,eax2000255f mov eax,[20006190] // Fixup on A.ctor20002565 call dword ptr [eax] // Call to A.ctor

A.a();20002567 mov eax,[20006194] // fixup on A.a 2000256d call dword ptr [eax] // Call to A.a

InheritFromA.a1(); 2000256f call dword ptr [20005cb8] // Indirect call to InheritFromA.a1

// (InheritFromA may need one time init)}20002575 ret

41

Example: In the Limit

b() has no fixups and can be called directly{

A a = new A();2000254a mov ecx,0x6ab95b98 // Note no fixup on A's method table pointer2000254f call CLRStub@200040e0 // call to new20002554 mov ecx,eax20002556 call a!A..ctor() // Note direct call to A.ctor

A.a();2000255b call a!A.a() // Note direct call to A.a InheritFromA.a1(); 20002560 call b!InheritFromA.a1() // Note direct call to

// InheritFromA.a1 (InheritFromA // is pre-inited)

}20002565 ret

42

In Conclusion

Ahead of Time Dynamic Translation (PreJit)

• What is it? – a compiler (not)

• What does it do? – produce good fast code (hopefully)

• When does it run? – Before the code is executed. i.e ahead of time, (just a bit earlier than a JIT).

43

Bottom Line

All that is left is compiler style optimizations

Prejit makes sense in some scenarios, but not all

Documents

Ahead of Time Dynamic Translation PreJit/NGEN by any other name George Bosworth Microsoft MRE04 March 21, 2004