OpenCAPI Power Platform Architectureopenpowerfoundation.org/wp-content/uploads/resources/oppa-1/opp… · 18-01-2019 · This document defines the OpenCAPI Power Platform Architecture

www.openpowerfoundation.org

http://openpowerfoundation.org

http://www.openpowerfoundation.org

OpenCAPI Power Platform Archi-tecture

January 18, 2019 Revision 3.0

OpenPOWER Foundation iiWorkgroup Notes

Non-standard Track

OpenCAPI Power Platform ArchitectureSystem Software Work Group <[email protected]>OpenPower Foundation

Revision 3.0 (January 18, 2019)Copyright © 2018 OpenPOWER Foundation

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except incompliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License isdistributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, eitherexpress or implied. See the License for the specific language governing permissions and limitations underthe License.

The limited permissions granted above are perpetual and will not be revoked by OpenPOWER or itssuccessors or assigns.

This document and the information contained herein is provided on an "AS IS" basis AND TO THEMAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, THE OpenPOWER Foundation AS WELLAS THE AUTHORS AND DEVELOPERS OF THIS STANDARDS FINAL DELIVERABLE OR OTHERDOCUMENT HEREBY DISCLAIM ALL OTHER WARRANTIES AND CONDITIONS, EITHER EXPRESS,IMPLIED OR STATUTORY, INCLUDING BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES, DUTIESOR CONDITIONS OF MERCHANTABILITY, OF FITNESS FOR A PARTICULAR PURPOSE, OFACCURACY OR COMPLETENESS OF RESPONSES, OF RESULTS, OF WORKMANLIKE EFFORT, OFLACK OF VIRUSES, OF LACK OF NEGLIGENCE OR NON-INFRINGEMENT.

OpenPOWER, the OpenPOWER logo, and openpowerfoundation.org are trademarks or registeredtrademarks of OpenPOWER Foundation, Inc., registered in many jurisdictions worldwide. Other company,product, and service names may be trademarks or service marks of others.

Abstract

The purpose of this document is to describe the interactions between software and hardware supportingthe OpenCAPI architecture. The target audience is firmware and operating system developers.

This document is a Non-standard Track, Work Group Note work product owned by the System SoftwareWorkgroup and handled in compliance with the requirements outlined in the OpenPOWER Founda-tion Work Group (WG) Process document. It was created using the Document Development Guideversion 1.1.0. Comments, questions, etc. can be submitted to the public mailing list for this document at<[email protected]>.

http://www.apache.org/licenses/LICENSE-2.0



OpenPOWER Foundation iiiWorkgroup Notes

Non-standard Track

Table of ContentsPreface ......................................................................................................................................... vii

1. Conventions ...................................................................................................................... vii2. Document change history ................................................................................................ viii

1. Overview .................................................................................................................................... 11.1. Who Should Read This Manual ...................................................................................... 11.2. Document Organization ................................................................................................... 11.3. Related Publications ........................................................................................................ 11.4. Conventions and Notation ............................................................................................... 21.5. References to Registers, Fields, and Bits ........................................................................ 21.6. Endian Order .................................................................................................................. 3

2. Introduction to OpenCAPI Power Platform Architecture .............................................................. 42.1. AFU functional definition is outside the scope of the OPPA ............................................. 42.2. Main Storage Addressing ................................................................................................ 4

2.2.1. Main Storage Attributes ........................................................................................ 42.3. Reserved Fields and Registers ....................................................................................... 52.4. Conformance to the OPPA .............................................................................................. 5

3. Programming Model ................................................................................................................... 63.1. Shared Programming Model ........................................................................................... 6

3.1.1. Starting and Stopping an AFU in the Shared Models ............................................ 73.2. Scheduled Processes Area ............................................................................................. 9

3.2.1. Converting BDF/PASID to PE Handle ................................................................... 93.2.1.1. Multi Port AFU ......................................................................................... 10

3.2.2. Process Element Entry ....................................................................................... 123.2.2.1. OSL Configuration State .......................................................................... 133.2.2.2. LPID, PID, TID, AMR .............................................................................. 143.2.2.3. Software State Field Format .................................................................... 14

3.3. Context Management .................................................................................................... 143.3.1. Removing a Context ........................................................................................... 15

4. Interrupts .................................................................................................................................. 164.1. Interrupt Flow ................................................................................................................ 16

4.1.1. Interrupt EA Page Mapping ................................................................................ 164.2. Interrupt Command Flag Formats Supported ................................................................. 164.3. Interrupt Types .............................................................................................................. 174.4. Wake Host Thread ........................................................................................................ 17

5. Storage Addressing .................................................................................................................. 185.1. Error Summary .............................................................................................................. 185.2. Retry Responses ........................................................................................................... 18

6. OSL Privileged Facilities .......................................................................................................... 196.1. Host BAR Registers ...................................................................................................... 19

6.1.1. OSL Registers .................................................................................................... 196.1.2. Configuration Address/Configuration Data .......................................................... 196.1.3. AFU MMIO BAR ................................................................................................. 196.1.4. AFU LPC BAR ................................................................................................... 19

6.2. OSL Privileged 1 Facilities ............................................................................................ 206.2.1. OSL Scheduled Processes Area Pointer Register (OSL_SPAP) .......................... 216.2.2. OSL Context Cache Invalidate Register (OSL_CCINV) ....................................... 226.2.3. OSL Slice Control Register (OSL_SCNTL) ......................................................... 236.2.4. OSL ERAT Invalidate (OSL_INV_ERAT) ............................................................. 24



OpenPOWER Foundation ivWorkgroup Notes

Non-standard Track

6.2.5. OSL Invalidate LPID/PID (OSL_INV_LPP) .......................................................... 266.3. OSL Privileged 2 Facilities ............................................................................................ 27

6.3.1. OSL Data Storage Interrupt Status Register (OSL_DSISR) ................................. 286.3.2. OSL Data Address Register (OSL_DAR) ............................................................ 296.3.3. OSL Translation Fault Control Register (OSL_TFC) ............................................ 306.3.4. OSL Process Element Handle Register (OSL_PEHandle) ................................... 31

7. OCAPI Configuration Environment ........................................................................................... 327.1. Dual Bus Configuration Space ...................................................................................... 327.2. OCAPI Reset ................................................................................................................ 32

7.2.1. Fundamental Reset (PERST) ............................................................................. 32A. Glossary .................................................................................................................................. 33B. OpenPOWER Foundation overview ......................................................................................... 37

B.1. Foundation documentation ............................................................................................ 37B.2. Technical resources ...................................................................................................... 37B.3. Contact the foundation .................................................................................................. 38



OpenPOWER Foundation vWorkgroup Notes

Non-standard Track

List of Figures3.1. Accelerator Invocation Process in the Shared Model ............................................................... 83.2. Structure for Scheduled Processes ....................................................................................... 11



OpenPOWER Foundation viWorkgroup Notes

Non-standard Track

List of Tables1.1. Register References ............................................................................................................... 32.1. Sizes of Main Storage Address Spaces .................................................................................. 43.1. PE Handle Mask Function .................................................................................................... 103.2. Structure of Scheduled Processes Area ................................................................................ 113.3. Process Element Entry Format ............................................................................................. 124.1. Interrupt Command Flags and Field Definition ....................................................................... 164.2. Interrupt Types ...................................................................................................................... 174.3. Wake Host Thread Command Flags and Field Definitions ..................................................... 17



OpenPOWER Foundation viiWorkgroup Notes

Non-standard Track

Preface1. ConventionsThe OpenPOWER Foundation documentation uses several typesetting conventions.

NoticesNotices take these forms:

Note

A handy tip or reminder.

Important

Something you must be aware of before proceeding.

Warning

Critical information about the risk of data loss or security issues.

ChangesAt certain points in the document lifecycle, knowing what changed in a document is important. Inthese situations, the following conventions will used.

• New text will appear like this. Text marked in this way is completely new.

• Deleted text will appear like this. Text marked in this way was removed from the previous versionand will not appear in the final, published document.

• Changed text will appear like this. Text marked in this way appeared in previous versions but hasbeen modified.

Command promptsIn general, examples use commands from the Linux operating system. Many of these are alsocommon with Mac OS, but may differ greatly from the Windows operating system equivalents.

For the Linux-based commands referenced, the following conventions will be followed:

$ prompt Any user, including the root user, can run commands that are prefixed with the $prompt.

# prompt The root user must run commands that are prefixed with the # prompt. You can alsoprefix these commands with the sudo command, if available, to run them.



OpenPOWER Foundation viiiWorkgroup Notes

Non-standard Track

Document linksDocument links frequently appear throughout the documents. Generally, these links include a textfor the link, followed by a page number in parenthesis. For example, this link, Preface [vii],references the Preface chapter on page vii.

2. Document change historyThis version of the guide replaces and obsoletes all earlier versions.

The following table describes the most recent changes:

Revision Date Summary of Changes

January 25, 2018 • Conversion from framemaker source

April 9, 2018 • Minor editing for first draft

July 2, 2018 • More editing

July 26, 2018 • Mark "Review Draft". No other changes

January 18, 2019 • Version 3.0: approved to be published



OpenPOWER Foundation 1Workgroup Notes

Non-standard Track

1. OverviewThis document defines the OpenCAPI Power Platform Architecture (OPPA) for the IBM™POWER9™ systems. The information contained in this document allows various OPPA-compli-ant accelerator implementations to meet the needs of a wide variety of systems and applications.Compatibility with the OPPA allows applications and system software to migrate from one implemen-tation to another with minor changes.

For a specific implementation of the OPPA, see the documentation for that accelerator.

1.1. Who Should Read This ManualThis manual is intended for designers who plan to develop products that use the OPPA. Readersof this manual should be familiar with the documents listed in Section 1.3, “Related Publica-tions” [1]. In addition, individual implementations of the OPPA should have their own documenta-tion for that implementation.

1.2. Document OrganizationThe OPPA document describes the facilities available to an operating system or to hypervisorsoftware. Implementation details and compliance with the architecture for a specific implementationare provided separately.

Document Division Description

Overview Describes this document, related documents, the intended audience, and other generalinformation.

Introduction Provides a high-level overview of the OpenCAPI Power Platform Architecture (OPPA).

Programming Model Describe the programming model used by the AFUs.

OpenCAPI™ Service Layer Describes the additional instructions and facilities, beyond those defined in user-modeenvironment, that are provided by the OPPA. This section covers instructions and facili-ties, not available to the application programmer, that affect storage control, interrupts, andtiming facilities.

The following sections are included:

• Interrupts

• Storage Addressing

• OSL Privileged Facilities

OCAPI Configuration Environment This section describes the OCAPI configuration space for a OPPA-compliant device. Theconfiguration facilities allow system software to set the base address and various otherconfiguration parameters for the OPPA-compliant device.

Glossary Defines terms and acronyms used in this document.

1.3. Related PublicationsThe following documents can be helpful when reading this specification:

• Power ISA Version 3.0

https://openpowerfoundation.org/?resource_lib=power-isa-version-3-0




Non-standard Track

• OpenCAPI 3.0 Transaction Layer Specification

1.4. Conventions and NotationThis document uses standard IBM notation, meaning that bits and bytes are numbered in ascendingorder from left to right. Thus, for a 4-byte word, bit 0 is the most-significant bit, and bit 31 is the least-significant bit.

MS

b

LSb

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Numbers are generally shown in decimal format, unless designated as follows:

• Hexadecimal values are preceded by an “x” and enclosed in single quotation marks or precededby “0x”.

For example: x‘0A00’ or 0x0A00.

• Binary values in sentences are shown in single quotation marks.

For example: ‘1010’.

Note: A bit value that is immaterial, which is called a “don't care” bit, is represented by an “X”.

This document uses the following software documentation conventions:

1. Command or instruction names are written in bold type. For example: afu_wr and afu_rd.

2. Field names and variables are written in italic type. Required parameters are enclosed in anglebrackets. Optional parameters are enclosed in brackets. For example: afu<f,b>_wr[a].

This document uses the following symbols:

& Bitwise AND

| Bitwise OR

~ Bitwise NOT

% Modulus

= Equal to

! = Not equal to

≥ Greater than or equal to

≤ Less than or equal to

x >> y Shift to the right; for example, 6 >> 2 = 1; least-significant y bits are dropped

x << y Shift to the left; for example, 3 << 2 = 12; least-significant y bits are replaced zeros

|| Concatenate

1.5. References to Registers, Fields, and BitsRegisters are referred to by their full name or by their short name (also called the register mnemon-ic). Fields are referred to by their field name or by their bit position. The following table describeshow registers, fields, and bit ranges are referred to in this document and provides examples.

https://opencapi.org/technical/specifications/




Non-standard Track

Table 1.1. Register ReferencesType of Reference Format Example

Reference to a specific register and aspecific field using the register shortname and the field name

Register_Short_Name[Field_Name] MSR[R]

Reference to a field using the fieldname

[Field_Name] [R]

Reference to a specific register and tomultiple fields using the register shortname and the field names

Register_Short_Name[Field_Name1, Field_Name2] MSR[FE0, FE1]

Reference to a specific register and tomultiple fields using the register shortname and the bit positions.

Register_Short_Name[Bit_Number, Bit_Number] MSR[52, 55]

Register_Short_Name[Bit_Number] MSR[52]Reference to a specific register and toa field using the register short nameand the bit position or the bit range. Register_Short_Name[Starting_Bit_Number:Ending_Bit_Number] MSR[39:44]

Register_Short_Name[Field_Name]= n MSR[FE0] = ‘1’

MSR[FE] = x‘1’

Register_Short_Name[Bit_Number]= n MSR[52] = ‘0’

MSR[52] = x‘0’

A field name followed by an equalsign (=) and a value indicates thevalue for that field.

Register_Short_Name[Starting_Bit_Number:Ending_Bit_Number]=n

MSR[39:43] = ‘10010’

MSR[39:43] = x‘11’

Where n is the binary or hexadecimal value for the field or bits specified in the brackets.

1.6. Endian OrderThe Power ISA supports both big-endian and little-endian byte-ordering modes. Book I of the PowerISA describes these modes. However, the OPPA only supports big-endian byte ordering for access-ing the registers defined in this OPPA document except where noted.

Specifically:

• The OpenCAPI service layer (OSL) registers described in the OPPA only support big-endianbyte ordering. The OPPA does not support accessing these registers in little-endian byte-ordering format. To access the OSL registers in little-endian byte ordering, little-endian systemsoftware must perform a byte swap.




Non-standard Track

2. Introduction to OpenCAPI PowerPlatform ArchitectureThe OpenCAPI Power Platform Architecture (OPPA) defines an accelerator interface structure forcoherently attaching accelerators to the IBM Power Systems™ using an OpenCAPI Physical Link.The intent is to allow implementation of a wide range of accelerators to optimally address manydifferent market segments.

2.1. AFU functional definition is outside thescope of the OPPAA OPPA-compliant device includes one or more AFUs. An AFU is a user-defined function foraccelerating applications. An AFU typically processes data and initiates any required data transfersto perform its allocated tasks.

The purpose of an AFU is to provide applications with a higher computational unit density forhardware acceleration of functions to improve the performance of the application and off-load thehost processor. Using an AFU for application acceleration allows for cost-effective processing over awide range of applications.

The management of the AFU is provided by the driver. This document describes the facilities withinthe OSL to configure the OSL to communicate with the AFU.

2.2. Main Storage AddressingAFU access of the memory owned by the attached host builds on the rules provided by thePowerISA and the Power Platform architecture. OPPA provides extensions to allow an AFU toaccess host attached memory and allows the AFU's memory to be mapped in to the host's memory.

The AFU uses effective addresses to access host memory. The effective address is computed bythe AFU and is provided to the OSL. The effective address is translated to a real address accordingto the procedures described in the overview of address translation in Power ISA, Book III. The realaddress is the location in host memory that is referenced by the translated effective address.

2.2.1. Main Storage AttributesThe main storage of a system typically includes both general-purpose and nonvolatile storage. It alsoincludes special-purpose hardware registers or arrays used for functions such as system configura-tion, data-transfer synchronization, memory-mapped I/O, and I/O subsystems.

Table 2.1, “Sizes of Main Storage Address Spaces” [4] lists the sizes of address spaces in mainstorage.

Table 2.1. Sizes of Main Storage Address SpacesAddress Space Size Description

Real Address Space 2m bytes where m ≤ 60




Non-standard Track

Address Space Size Description

Effective Address Space 264 bytes An effective address is translated toa virtual address using the segmentlookaside buffer (SLB).

Virtual Address Space 2 n bytes where 65 ≤ n ≤ 78

A virtual address is translated to a realaddress using the page table.

Real Page (Base) 212 bytes

Virtual Page 2p bytes where 12 ≤ p ≤ 34

Up to eight page sizes can be supportedsimultaneously. A small 4 KB (p = 12) pageis always supported. The number of largepages and their sizes are implementationdependent.

Segment Size 2s bytes where s = 28 or 40

The number of virtual segments is 2 (n - s)

where 65 ≤ n ≤ 78

• The values of “m,” “n,” and “p” are implementation dependent.

2.3. Reserved Fields and RegistersAll reserved fields must be set to zero.

A specific instantiation of the OPPA must handle reserved bits as follows:

• Ignore the reserved bits on writes and return zeros for the reserved bits on reads; or

• All reserved registers must be initialized to zero, unless otherwise indicated.

Software is not required to initialize reserved registers. An implementation must decode all reservedregisters for both reads and writes. The handling of reserved register values in a specific instantia-tion of the OPPA is implementation dependent. For each reserved register, an implementation takesone of the following actions:

• Ignore the value on writes and return zeros for reads of the register.

• Maintain the state of the reserved register.

Fields and registers that are currently defined as reserved might become defined in future versionsof the OPPA. If a reserved register or register field is defined in a future version of the OPPA, a stateof zero should be defined in a manner that is backwards compatible with previous versions of theOPPA.

2.4. Conformance to the OPPAAny implementation of this architecture must adhere to the following set of numbered conformanceclauses to claim conformance to OPPA:

NoteThere may be a conformance section in the future that either defines conformance to thehost register set or the attached device.




Non-standard Track

3. Programming ModelThe Programming Model for an AFU is dependent on the AFU and driver. The host applicationregisters its context for the AFU with the Operating System. The Operating System returns aProcess Address Space ID (PASID) to the application. The process is attached to the AFU in animplementation specific manner where the AFU is notified that a process with a known PASID hasbeen added. The AFU assigns an address context tag (acTag) to a Bus Device Function (BDF) andPASID combination with the assign_actag command as described in the OpenCAPI TL Architecture.The BDF is the physical identifier of the device function, and the PASID is passed from the applica-tion in an implementation specific manner. The AFU passes this acTag with every command associ-ated with this process across the OCAPI link.

The OSL resolves the acTag into a BDF/PASID combination based on the prior assign_acTag thatwas issued by the AFU. The BDF/PASID combination is then converted to a Process Handle. Whilethe Process Handle is implementation specific, the lower 15-bits of the process handle must be theoffset of the process element within the Scheduled Process Area. The details about BDF conversionto a PE Handle is described in Section 3.2.1, “Converting BDF/PASID to PE Handle” [9]

The AFU can be shared by multiple partitions. The shared models require a system hypervisor tovirtualize the AFU so that each operating system can access the AFU. For single-partition systemsnot running a hypervisor, the AFU is owned by the operating system. In both cases, the operatingsystem can virtualize the AFU so that each process or application can access the AFU. A dedicatedprogramming model can be used - it is simply a subset of the shared programming model where only1 process is attached.

3.1. Shared Programming ModelThe shared programming model allows for all processes, or a subset of processes, from allpartitions, or a subset of partitions in the system, to use an AFU. Figure 3.1, “Accelerator InvocationProcess in the Shared Model” [8] shows how an application invokes an AFU under the sharedprogramming model.

For the shared model, the application is required to make an operating-system call with at least thefollowing information:

• An Authority Mask Register (AMR) value

The AMR value is the AMR state to use for the current process. The value passed to the operat-ing system is similar to an application setting the AMR in the processor by using SPR 13 or bycalling a system library.

Upon receiving the system call (syscall), the operating system verifies that the application hasregistered and been given the authority to use the AFU. The operating system then calls the hypervi-sor (hcall) with at least the following information:

• An Authority Mask Register (AMR) value, masked with the current AMOR Register value andoptionally masked with the current UAMOR by the hypervisor.

• A process ID (PID) and optional thread ID (TID)




Non-standard Track

Upon receiving the hypervisor call (hcall), the hypervisor verifies that the operating system hasregistered and been given the authority to use the AFU. The hypervisor then puts the processelement into the Scheduled Process Area. The process element minimally contains the following:

• An Authority Mask Register (AMR) value, masked with the current AMOR

• A process ID (PID) and optional thread ID (TID)

• An OSL Configuration State value

• A logical partition ID (LPID)

The hypervisor initializes the following host proxy registers:

• Real address Scheduled Processes Area Pointer (OSL_SPAP)

Note

The information passed by the application and operating system is not complete. Theapplication may need to indicate the number of interrupts it requires or other information.

The final AMR value that is placed in the process element after any masking is the AMRvalue that will be sent to the nMMU for any translation request.

3.1.1. Starting and Stopping an AFU in the SharedModelsStarting and stopping an AFU process is an AFU implementation-specific procedure.

Note

How the AFU gets the PASID is implementation specific as is adding or removingPASID’s from the AFU. Each AFU can have its own method of doing this.

Existing AFU reference implementations and 'generic' driver are an example of how itcan be done.




Non-standard Track

Figure 3.1. Accelerator Invocation Process in the Shared Model




Non-standard Track

3.2. Scheduled Processes AreaThe host proxy reads process elements from a structure located in system memory called thescheduled processes area (SPA). The SPA contains a list of processes to be serviced by the AFUs.The process elements contain the address context and other state information for the processesscheduled to run on the AFUs assigned to service the SPA structure. The SPA structure consists ofprocess elements with each process element located at a cacheline offset within the SPA. The offsetof each process element within the SPA is defined as a PE Handle.

The PE Handle is determined from the BDF (Bus/Device/Function) and the PASID in an implemen-tation specific manner. The actag (address context tag) field of the TL architecture is assigned to aBDF/PASID combination by the AFU via the assign_actag command, and this actag is used to identi-fy all commands associated with that BDF/PASID.

The actag is the context identifier that is received from the TL. This actag is used to index into a hoststructure that contains the BDF/PASID associated with that actag. The afu must send commands(assign_actag) to the host proxy before sending any command with that actag or the host lookup willfail and result in an error due to a bad context. The host proxy contains a configuration register thatdefines how many actag index bits are supported by the platform, and the afu must be configured viaconfig space with the same number of supported index bits. Host software is responsible for readingmaximum index value from the host proxy and programming this into the AFU.

When the host proxy receives a command with an actag it will resolve the index pointer into a BDF/PASID to be used for the PE Handle.

3.2.1. Converting BDF/PASID to PE HandleThe BDF/PASID needs to be converted to a PE Handle which is an index into the SPA. The PEHandle is a 16 bit index. There are 2 XSLO components, each supporting 2 bricks. The most signif-icant bit of the PE Handle is reserved as a brick ID identifier within the XSL, so only 15 bits areavailable for use by software. A non-overlapping hash is created from 15 bits of the BDF and 15 bitsof the PASID. Only low order 15 bits of the PASID may be assigned to the AFU. The others will beignored by hardware. The BDF is bit reversed before applying the mask since the left most bits areupper bus numbers that can be thrown away and replaced by the PASID.The PE Mask register inthe host proxy determines how many bits are taken from the bit reversed BDF and how many bitsare taken from the PASID to form the PE Handle:

PE Mask(3:0): Valid values: 0 to 15

Indicates how many of pe_handle(14:0) consecutive high-order bits are selected from the bit-reversed BDF with the remaining bits coming from the PASID.

BDF_Reverse(15:0) = BDF(0:15). BDF_MASK(15:1) & BDF_Reverse(15:1) defines the bits usedfrom the Reverse BDF. PASID_MASK(14:0) & PASID(14:0) defines the bits used for the PASID.The register assures the mask values for each are non-overlapping so no aliasing occurs, and themasked results are ORed together to form the PE_Handle(14:0).

The full equation is PE_Handle(14:0) = ((BDF_Reverse(15:1) & BDF_MASK(15:1)) | ((PASID(14:0) &PASID_MASK(14:0)). See Table 3.1, “PE Handle Mask Function” [10] for additional information.




Non-standard Track

The full PE_Handle(15:0) is formed by Brick_ID || PE_Handle(14:0), and this is used to index into thecorrect cacheline within the SPA.

3.2.1.1. Multi Port AFU

If the AFU is dual port (See Section 7.1, “Dual Bus Configuration Space” [32]), the SPAP foreach brick must be configured to point to the same table so that the Process Element entries areshared and can be accessed indifferently from both links. The match on the Brick_ID bit of thePE_Handle should be masked. This is configured via Section 6.2.3, “OSL Slice Control Register(OSL_SCNTL)” [23].

If the AFU is operating in this mode, it should also assign the same actag to the same BDF/PASIDcombination for each brick it is sending commands.

Table 3.1. PE Handle Mask Function

PE_Mask(3:0) bdf_mask(15:1)Applied to bitreversed BDF

pasid_mask(14:0)Applied toPASID

Description

0 000_0000_0000_0000 111_1111_1111_1111 0 bits come from bit-reversedBDF, all bits selected fromPASID(14:0)

1 100_0000_0000_0000 011_1111_1111_1111 1 bit comes from bit-reversedBDF, 14 bits selected fromPASID(13:0)

2 110_0000_0000_0000 001_1111_1111_1111

3 111_0000_0000_0000 000_1111_1111_1111

4 111_1000_0000_0000 000_0111_1111_1111

5 111_1100_0000_0000 000_0011_1111_1111

6 111_1110_0000_0000 000_0001_1111_1111

7 111_1111_0000_0000 000_0000_1111_1111

8 111_1111_1000_0000 000_0000_0111_1111

9 111_1111_1100_0000 000_0000_0011_1111

10 111_1111_1110_0000 000_0000_0001_1111

11 111_1111_1111_0000 000_0000_0000_1111

12 111_1111_1111_1000 000_0000_0000_0111

13 111_1111_1111_1100 000_0000_0000_0011

14 111_1111_1111_1110 000_0000_0000_0001 14 bits come from bit-reversedBDF, 1 bit selected fromPASID(0)

15 111_1111_1111_1111 000_0000_0000_0000 15 bits come from bit_reversedBDF, no PASID bits selected




Non-standard Track

Figure 3.2. Structure for Scheduled Processes

Table 3.2, “Structure of Scheduled Processes Area” [11] defines the various fields and areaswithin the scheduled processes structure. The starting address of the area (SPA_Base) is defined bythe Scheduled Processes Area Pointer Register (OSL_SPAP). The storage must be contiguous inthe real address space and naturally aligned to the size of the scheduled processes area.

Note that there is no size checking for the SPA size, and each brick has an entire 4MB regionassumed to be reserved for the SPA. This is described in Section 6.2.1, “OSL Scheduled ProcessesArea Pointer Register (OSL_SPAP)” [21].

Table 3.2. Structure of Scheduled Processes AreaMnemonic Address (Byte) Description

start_of_SPA_area SPA_Base This is the start of the area in systemstorage used by system software to storethe process elements scheduled for theacceleration function units (AFUs).

end_of_SPA_area SPA_Base +x’3FFFFF This is the end of the area in systemstorage used by system software to storethe process elements scheduled for theAFUs.




Non-standard Track

3.2.2. Process Element EntryTable 3.3. Process Element Entry Format

Hypervisor Process Element EntryWord

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

0 OSL Configuration State (0:31)

1 OSL Configuration State (32:63)

2 Reserved

3 Reserved

4 Reserved

5 Reserved

6 Reserved

7 Reserved

8 Reserved

9 Reserved

10 Reserved

11 Reserved

12 Reserved

13 LPID

14 TID

15 PID

16 Reserved

17 Reserved

18 Reserved

19 Reserved

20 Reserved

21 Reserved

22 Reserved

23 Reserved

24 Reserved

25 Reserved

26 AMR (most significant bits)

27 AMR (least significant bits)

28 Reserved

29 Reserved

30 Reserved

31 Software State




Non-standard Track

3.2.2.1. OSL Configuration State

The OSL Configuration State contains configuration information for the corresponding ProcessElement Entry which will be passed to the nMMU during translation requests

SF Res. HV R. XMD Pg Sz ST Reserved

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Reserved PR Reserved ISL TC Reserved SC DR Reserved

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

Bits Field Name Description

0 SF Sixty-four bit mode.

0 AFU effective addresses are interpreted as 32-bit values.1 AFU effective addresses are interpreted as 64-bit values.

1:2 Reserved Reserved.

3 HV Hypervisor state.

0 AFU is not operating in a hypervisor state.1 AFU is operating in a hypervisor state if PR = ‘0’; otherwise AFU is not operating in

hypervisor state.

NoteThe privilege state of an AFU is determined by HV || PR.

00 AFU is operating in a privileged state.01 AFU is operating in a problem state.10 AFU is operating in a privileged and hypervisor state.11 AFU is operating in a problem state.

4 Reserved Reserved.

5:6 XLAT Mode Translation mechanism mode select.

00 Hashed page table (HPT) mode (AIX on PHYP via single-level translation)01 Reserved10 Radix on HPT mode (Linux on PHYP)11 Radix on Radix mode (Linux on KVM)

These bits equate to the host Radix (HR) and guest Radix (GR) bits in the partitiontable entry for an LPID.

7:9 Page Size Page size.

Page size is used for HPT hash by the host (hypervisor) under a guest using Radix transla-tion.

Encoding is the same as a segment’s L and LP fields.

These bits equate to LPCE

10 BoT Select for the nMMU to use SLB or Segment Table for EA-VA HPT translations.

0 Use the SLB only.1 Use the in-memory segment table.

NoteThis is included for completeness and will cause a segment fault when setto ‘0’.





Non-standard Track


49 PR Problem state.

0 The OSL has privileged-state access to pages.1 The OSL has problem-state access to pages.

NotePOWER9 implementations do not currently use this bit. This bit might beremoved in future specifications.


53 ISL Ignore segment large-page specification.

0 ISL disabled1 ISL enabled

54 TC Translation control.

0 Secondary hash of the page table search is enabled.1 Secondary hash of the page table search is disabled.

55:57 Reserved Reserved

58 SC Segment translation control.

0 Secondary hash of the segment table search is enabled.1 Secondary hash of the segment table search is disabled.

59 DR Relocate.

0 OSL translation is off.1 OSL translation is on.


3.2.2.2. LPID, PID, TID, AMR

These values are passed to the nMMU with a translation request.

3.2.2.3. Software State Field Format

Hardware only uses the valid bit. If the valid bit is not set a translation attempt will not be attemptedand an error will be logged.

V Reserved

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31


0 V Process element valid.

0 Process element information is not valid.1 Process element information is valid.


3.3. Context ManagementThe management of contexts is between the AFU and software.

A 'generic' driver can be implemented. Most AFUs should be able to use it instead of providing theirown.




Non-standard Track

3.3.1. Removing a ContextHardware contains a context cache for storing process element entries. This cache must be invali-dated by software if a process element is updated or removed. This is done through the CCINVregister (see Section 6.2.2, “OSL Context Cache Invalidate Register (OSL_CCINV)” [22]).

Software must poll CCINV[Inv_In_Prog] and only initiate a new invalidate request if this bit is ‘0’.

Software should set the software state of the Process Element to invalid prior to writing the CCINVregister.




Non-standard Track

4. InterruptsInterrupts are EA based with a unique address per interrupt source provided by the AFU. Theintrp_req OpenCAPI command is sent by the AFU with Object_Handle = EA of Interrupt. Each EA iseither 4kB or 64kB page aligned.

To prevent buggy or malicious problem state code from interfering with or causing a denial of serviceissue with the proper usage of the overall XIVE architecture by hypervisor and operating systemcode, the operating system should only allow the mapping of the even ESB page of the even/oddESB page pair in the routing engine that only allows trigger operations into the address space usedby a process element.

If multiple interrupts are used, their respective EAs may, or may not, be contiguous. It is up tothe application and AFU developer to agree on how the host process passes the interrupt ObjectHandles to the AFU.

4.1. Interrupt Flow1. AFU accesses local process context and obtains interrupt Object Handle and Command Flag

2. AFU generates interrupt request providing acTag, Command Flag and Object Handle

3. acTag from interrupt request is used to resolve BDF:PASID and generate PE Index

4. PE Index is used to look up translation context to resolve EA->RA

Note

Note that EA->RA translation can result in a fault

5. CI (Cache-Inhibited) Write is sent to RA which triggers the interrupt

4.1.1. Interrupt EA Page MappingSince the interrupt is translated to a write operation the page must be allocated with write permis-sions or the EA->RA translation will fail.

4.2. Interrupt Command Flag FormatsSupportedThe command flag field in the TL packet identifies the platform specific format of the object handle.See Table 4.1, “Interrupt Command Flags and Field Definition” [16]

Table 4.1. Interrupt Command Flags and Field Definition

cmd_flag(3:0) obj_handle(63:0) Description

‘0000’ EA of interrupt Standard interrupt format. Obj_handle contains EA of interrupt




Non-standard Track

4.3. Interrupt TypesTable 4.2, “Interrupt Types” [17] list the types of interrupts that can be received by systemsoftware. In the comment column, registers containing additional information about the interrupt areprovided. A register resides in the host processor which is configured for the Real Address destina-tion for translation faults.

Host proxy detected errors are handled via FIRs, and additional interrupts are not required.

Table 4.2. Interrupt TypesOSL_DSISRInterrupt Type

TF

Comment

Translation Fault TF = ‘1’ Translation fault.

The faulting Effective Address is reported in OSL_DAR. The process handle for the faultingprocess is contained in the OSL_PEHandle Register.

The OSL_DSISR[TF] status bit is reset when the corresponding OSL_TFC Register iswritten. Additional information about the cause of the translation fault is provided in theOSL_DSISR[CO_RSP] field.

AFU Interrupt - AFU interrupt

The signification of the interrupt is application-dependent and defined by its EffectiveAddress

The RA for fault interrupts will be in a host register.

4.4. Wake Host ThreadThe wake_host_thread command is used by an AFU to wake a thread on the host. There is anObject Handle field defined for this command for host specific information. There may be hostspecific implementations for Power Platforms where the AFU must be configured with the informa-tion to place in the Obj_handle and which 4 bit command flag to use for the wake_host_threadcommand. For example if an AFU is programmed to indicate the thread ID to wake vs the threadID within the Process Element Entry it may need to specify a different command flag where thethread ID is specified in the obj_handle. Definitions of possible command flags and their host specif-ic obj_handle bit definitions are shown in Table 4.3, “Wake Host Thread Command Flags and FieldDefinitions” [17]

Table 4.3. Wake Host Thread Command Flags and Field Definitionscmd_flag(3:0) obj_handle(67:0) Description

‘0000’ Not Used TID taken from Process Element Entry

‘0001’ TID is taken from (15:0) TID is communicated to the AFU through some implementationdependent method.




Non-standard Track

5. Storage AddressingIn OPPA, the virtual-to-real address conversion is performed by a unit in the host processor. Thedefinition of this unit is outside the scope of the OPPA. For more information, see the implementa-tion-specific documentation for the host processor.

Addresses sent by the AFU towards the host proxy that have a dot-BE attribute must not map to anMMIO address. If a read maps to MMIO space the length must be 32 bytes or less. If the AFU doesnot understand the difference between Main memory and MMIO memory an EA should never begiven to the AFU that maps to MMIO memory.

It is believed that it is always true that mmio memory (I=1) has RA(13:14)=0b11 in the PTE.

5.1. Error SummaryThe general AFU error strategy is for the AFU to send an interrupt and have that error serviced bythe device driver when possible. Errors detected by the host proxy will most likely bring the link downwith error information captured within the host proxy for debug.

It is recommended that a system be designed with some remote debug capability via I2C or othermechanism for lab bringup. This includes the ability to gather error information from the remote chipfor link debug as well as a way to gather information from the AFU.

Note

Secure Memory Facilities

OPPA 3.0 does not support operating in SMF mode, but there is a configuration bit withinthe host proxy to guarantee that the SMF address bit will never get set. This allowsOPPA to be used within a SMF system, but OPPA itself is not SMF.

5.2. Retry ResponsesThe TL architecture defines a rty_req response encode for some responses. These are consideredlong back-off events since they indicate a software response is needed to continue. The long back-off timer is configurable in config space.

It should be noted that for power systems a rty_req response delivered with the xlate_done orintrp_rdy indicates that software has responded to a storage interrupt and indicated it will take longerto resolve the page mapping. An afu design for power systems may wish to use a larger multiplier forthe long back-off timer for these responses.




Non-standard Track

6. OSL Privileged FacilitiesThe registers in these sections should only be accessed by system software. The hypervisor canaccess the Privileged 1 registers, and the O/S can access the Privileged 2 registers.

6.1. Host BAR RegistersAccess to all OpenCAPI registers is done via BAR registers set up in the host processor.

6.1.1. OSL RegistersThe OpenCAPI Service Layer (OSL) Registers are part of the overall register set offered by the hostprocessor. These registers are accessed as offsets from BAR regions configured in the host proces-sor. See the OpenCAPI programming guide for the host processor for information.

6.1.2. Configuration Address/Configuration DataEach brick has a configuration address and data register to provide an indirect access to configura-tion space of an AFU.

6.1.3. AFU MMIO BARAn AFU MMIO BAR is needed to determine which addresses to forward to the AFU MMIO Space.The Physical Address (PA) also needs to be programmed into the AFU configuration space.

6.1.4. AFU LPC BARAn AFU can contain memory to be mapped to system main memory. This is sometimes called AFULowest Point of Coherency (LPC) Memory. An LPC Memory BAR is needed to determine whichaddresses to forward to AFU LPC Memory. LPC memory is intended to start at PA 0 within the AFU,so no BAR is needed by the AFU.




Non-standard Track

6.2. OSL Privileged 1 FacilitiesThe OCAPI Service Layer (OSL) privilege 1 facilities include the following registers:

• OSL Scheduled Processes Area Pointer Register (OSL_SPAP) (page 21)

• OSL Context Cache Invalidate Register (OSL_CCINV) (page 22)

• OSL Slice Control Register (OSL_SCNTL) (page 23)

• OSL ERAT Invalidate (OSL_INV_ERAT) (page 24)

• OSL Invalidate LPID/PID (OSL_INV_LPP) (page 26)




Non-standard Track

6.2.1. OSL Scheduled Processes Area Pointer Register(OSL_SPAP)The Scheduled Process Area for each brick is always 4MB in size, and there is 1 SPAP register foreach brick. Each brick can use 15 bits of the PE Handle for 32k PE entries (4MB)

The entire 4MB region for each brick is assumed to be reserved for the SPA, so there is no sizevalidation check on the calculated PE Handle.

There are 2 XSLO regions each with 2 bricks, so there will be 2 SPAP registers per XSLO. The mostsignificant bit of the PE Handle is used as a brick ID and determines which SPAP register within anXSLO is used to determine the SPAP to use for the Process Element fetch.

The OSL Scheduled Processes Area Pointer Register (OSL_SPAP) contains the physical address inmain storage of the Scheduled Process Area.

For a dual port AFU, the OSL_SPAP register for each brick must be configured with the sameaddress.

There is one register for each brick. Access to these registers shall be privileged. These registersshall be accessed using a single 64-bit operation.

Access Type Read/Write

Reserved SPA Real Address

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

SPA Real Address Reserved V

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63



4:41 SPA

Real Address

Real address of the scheduled processes area.

NoteThe lower 22-bits of the 60-bit real address pointer are always ‘0’ (that is,4MB aligned).


63 V When set, indicates that the scheduled process area’s real address is valid. If the SPA realaddress is not valid an error will be logged.




Non-standard Track

6.2.2. OSL Context Cache Invalidate Register(OSL_CCINV)The management of process elements is unique to an AFU and managed by the driver. The CCINVregister is only used for management of the context cache within the host proxy. When a processelement is removed or altered a store shall be issued to this register to remove the cached entrywithin the OSL.

There is only 1 register per XSL with the most significant bit of the PE_Handle representing the brickid within that XSL.


Reserved T IP Reserved

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Reserved ID PE_Handle

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63



15 Terminate Remove the Process Element Entry identified by the pe_handle field from the contextcache.

16 Inv_In_Prog Invalidate in Progress

Software should ensure this bit is ‘0’ before writing a new context invalidate command


48 Brick_ID Brick ID to remove Process Element Entry

49:63 PE_Handle Process element handle.

The process element handle is the offset from the SPA_Base of the process element tooperate on, shifted right by 7 bits.




Non-standard Track

6.2.3. OSL Slice Control Register (OSL_SCNTL)The OSL Slice Control Register (OSL_SCNTL) allows system software to govern the operation of anXSL. This register controls both bricks within an XSL

Setting the Suspend Control (Sc) bit causes the corresponding OSL to stop executing transla-tions for the corresponding OTL. To continue normal operation from a suspended state, theOSL_SCNTL[Sc] must be set to zero.

Setting the Purge (Pc) bit in this register behaves like Suspend Control. Purging operations requiresa sequence of recovering the host proxy.

These registers shall be accessed using a single 64-bit operation.


MA Reserved

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Reserved Ps Reserved Pc Reserved Ss Reserved Sc

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63


0 Multi_AFU The same AFU resides on both bricks. Don’t compare the Brick ID bit of the PE Handlewhen searching the ERAT

0 Same AFU Context Space on both bricks - don’t match the Brick ID within the PEHandle

1 Unique AFU’s exist on each brick - match the Brick ID within the PE Handle


38:39 Ps OSL transaction purge status. This is a read-only field. Data written to this field is ignored.

00 Purge request not outstanding.01 Purge of OSL transactions in process (some implementations can choose not to

implement this state).10 Purge of OSL transactions is complete.


48 Pc Purge all OSL transactions from the associated AFU.

0 No purge of OSL transactions requested.1 Purge OSL transactions request.


54:55 Ss OSL transaction queue suspend status. This is a read-only field. Data written to this field isignored.

00 Normal OSL transaction operation.01 Suspend of OSL operation in process (some implementations can choose not to

implement this state).10 OSL operation suspended.


63 Sc Suspend control. Suspend OSL operation.

0 Normal OSL operation.1 Suspend OSL operation request.




Non-standard Track

6.2.4. OSL ERAT Invalidate (OSL_INV_ERAT)A write to the OSL ERAT Invalidate Register causes an invalidation of all ERAT entries that matchthe parameters set in this register and in OSL_INV_LPP.

The OSL_INV_LPP register must be written prior to writing the OSL_INV_ERAT register.

Writing to this register sets the invalidation attributes, reading from it returns the invalidate inprogress status bit.


Register write bit formatBits Field Name Description

0 MLPID Match LPID - If set, the invalidate will match the LPID set in the INVLPID field in theXSL_INV_LPP register.

1 MPID Match PID - If set, the invalidate will match the PID set in the INVPID field in theXSL_INV_LPP register. Valid only if INVR or INVSLB are set.

2 MADDR Match ADDR - If set, the invalidate will match the address provided in the INVADDR fieldvs. the EA or VA of each ERAT entry.

3 INVR Radix - Selects whether the invalidate will invalidate HPT or Radix translations.

4 PRS PRS - Sets the PRS bit for Radix invalidations

5:7 PAGESIZE Encoded Page size.

This control only selects the amount of address bits to compare, it will not be matched vs.the page size attribute of the ERAT entry.

Valid encodings are:

3’b000 - 4k - Match address bits [0:51]

3’b001 - Reserved

3’b010 - 64k - Match address bits [0:47]

3’b011 - 2M - Match address bits [0:42]

3’b100 - 16M - Match address bits [0:39]

3’b101 - 1G - Match address bits [0:33]

3’b110 - Reserved

3’b111 - 16G - Match address bits [0:29]

Valid only if INVESID=’b0

8 INVALL Invalidate All - Invalidates all ERAT entries for Radix or HPT (Based on the INVR bitsetting).

All other attributes are ignored if this bit is set.

9 Reserved Reserved

10 SEGSIZE Segment size - Selects the segment size for HPT invalidates.

1’b0 - 256MB Segment

1’b1 - 1TB Segment

Valid only if INVESID=’b1

11 INVESID Invalidate ESID - If set, the address in the INVADDR field will be matched to the EA in theERAT entry.




Non-standard Track

Bits Field Name DescriptionValid only if MADDR is set.

12:63 INVADDR Address to be invalidate.

If address is a VA or Radix EA (INVESID=’b0), bits [12:63] == VA[0:51]. That actual bitscompared depend on the PAGESIZE field.

If address is an ESID (INVESID=’b1), bits [12:47] == EA[0:35] Valid only if MADDR is set.

Register Read bit formatBits Field Name Description


63 INV_IN_PROG Invalidate in progress

When set, XSL is processing a previous invalidate request.

Writing XSL_INV_ERAT while this bit is set will cause an error.




Non-standard Track

6.2.5. OSL Invalidate LPID/PID (OSL_INV_LPP)OSL_INV_LPP contains the process identifier (PID) and the logical partition ID (LPID) used to selectwhich ERAT entries to invalidate.


Reserved INVPID

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Reserved INVLPID

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63



12:31 INVPID Process Identifier (PID)


52:63 INVLPID Logical Partition ID (LPID)




Non-standard Track

6.3. OSL Privileged 2 FacilitiesThe OSL Privileged 2 facilities are Fault Control Registers.:

• OSL Data Storage Interrupt Status Register (OSL_DSISR) (page 28)

• OSL Data Address Register (OSL_DAR) (page 29)

• OSL Translation Fault Control Register (OSL_TFC) (page 30)

• OSL Process Element Handle Register (OSL_PEHandle) (page 31)




Non-standard Track

6.3.1. OSL Data Storage Interrupt Status Register(OSL_DSISR)The OSL Data Storage Interrupt Status Register (OSL_DSISR) contains the status that defines thecause of the OSL data storage interrupt. The function of this register is not related to the Power ISA,Book III of the same name.

The content of this register is set by the OSL when a translation fails due to a data segment fault(segment table miss) occurs or a data storage fault (page table miss) occurs. The register is resetto zero when the current outstanding fault is either restarted, continued, or indicated as an addresserror by a write to the OSL Translation Fault Control Register (OSL_TFC).

There is one register for each brick. Access to these registers should be privileged. These registersshall be accessed using a single 64-bit operation.

Access Type Read Only

Reserved TF Reserved

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Reserved S Reserved CO_RSP

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63


0:2 Reserved Set to zero.

3 TF Set to '1' if the OSL has detected a translation fault. The cause of the translation fault iscontain in

OSL_DSISR[CO_RSP].

This bit is reset to '0' when the corresponding OSL_TFC Register is written.

4:37 Reserved Set to zeros.

38 S Translation fail for a write operation

Set to ‘1’ if the access type was a write operation.

39:55 Reserved Set to zeros

56:63 CO_RSP Checkout Response Status




Non-standard Track

6.3.2. OSL Data Address Register (OSL_DAR)The OSL Data Address Register (OSL_DAR) contains the effective address associated with an OSLtranslation fault. The function of this register is similar to the POWER Data Address Register (DAR).For more information about the DAR, see the Power ISA, Book III.

There is one register for each brick. Access to these registers should be privileged. These registersshall be accessed using a single 64-bit operation.


Effective Address

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Effective Address

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63


0:63 Effective Address Effective address associated with the data segment or data storage interrupt.




Non-standard Track

6.3.3. OSL Translation Fault Control Register (OSL_TFC)The OSL Translation Fault Control Register (OSL_TFC) allows system software to govern theresponsecode sent to the AFU for the translation completion.

Setting the Restart (R) bit of this register causes a xlate_done to be sent to the AFU. The xlate_doneRespCode = ‘0000’ (Complete).

Setting the Continue (C) bit of this register causes a xlate_done to be sent to the AFU. Thexlate_done RespCode = ‘0010’ (Retry Request)

Setting the Address Error (AE) bit of this register causes a xlate_done to be sent to the AFU. Thexlate_done RespCode = ‘1111’ (Address Translation Error)

One and only 1 of the R,C or AE bits should be written to ‘1’ with a store to this register. If multiplebits are set or if none of the bits are set an indeterminate xlate_done response will be sent to theAFU.

There is one register for each brick. Access to these registers should be privileged. These registersmust be accessed using a single 64-bit store operation.


Reserved C AE R

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Reserved

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63



29 C Continue. Current translation fault is not resolved and must be retried at a later time.

NoteThis bit must be set to ‘1’ to cause a xlate_done to be sent to the AFUwith RespCode = ‘0010’ (Retry Request). This bit is automatically reset byhardware once the xlate_done has been sent.

30 AE Address error on the transaction caused the translation fault.

NoteThis bit must be set to ‘1’ to cause a xlate_done to be sent to the AFUwithRespCode = ‘1111’ (Address Translation Error). This bit is automaticallyreset by hardware once the xlate_done has been sent.

31 R Restarts the AFU transaction that caused the translation fault.

NoteThis bit must be set to ‘1’ to cause a xlate_done to be sent to the AFU.withRespCode = ‘0000’ (Complete). This bit is automatically reset by hardwareonce the xlate_done has been sent.





Non-standard Track

6.3.4. OSL Process Element Handle Register(OSL_PEHandle)The OSL Process Element Handle Register (OSL_PEHandle) contains the current process handleand the AFU command tag. This register is automatically loaded by the OSL with the correspond-ing process handle of the process interrupting the system. When an interrupt is not pending, the datareturned when reading this register is indeterminate.

There is one register for each brick. Access to these registers should be privileged. These registersmust be accessed using a single 64-bit store operation.


Reserved CMD_Tag

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Reserved PE_Handle

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63



16:31 AFUTag Command tag.

The command tag is a 16-bit tag corresponding to the AFUcommand that caused the interrupt. This field is implementationdependent and might not be supported by some implementations.See the implementation-specific documentation for more details.


48 Brick ID Brick ID identifies which link the PE_Handle was received on

49:63 PE_Handle Process element handle.

The process element handle is the 15-bit pointer to thecorresponding process element.




Non-standard Track

7. OCAPI Configuration EnvironmentOCAPI-compliant devices can be discovered and configured by the host software through a config-uration space. Full specification can be found in the OpenCAPI 3.0 Discovery and ConfigurationSpecification document, managed by the OpenCAPI consortium.

Configuration cycles are generated via the Config Address & Config Data registers within the hostproxy. This results in Configuration read and Configuration write packets sent across the OpenCAPIlink. The configuration space is a separate address space. Legal lengths of configuration reads orwrites are 1, 2 or 4 bytes. The AFU contains a header that provides vendor and device information.The vendor/device ID is used to determine a driver to load. Other configuration fields are located inVendor Specific Extended capabilities. These are used to determine both overall OpenCAPI featuresand Platform specific information. The AFU driver that is loaded is responsible for the programmingmodel of the AFU itself which includes process management.

The OCAPI configuration space is only used for discovery of the device and its capabilities andto program certain settings that are needed for correct operation with a host proxy. The completeconfiguration space is not necessary, and there is no physical link status or control available withinthe configuration space.

7.1. Dual Bus Configuration SpaceThe OPPA allows for a single OPPA-compliant device to support multiple independent OCAPI ports.For this type of configuration, the configuration space will indicate the presence of a primary andsecondary port

7.2. OCAPI ResetAn OCAPI device must support a PCIe equivalent of PERST to provide a fundamental reset of thePHY, TLX, DLX and AFU logic. After a fundamental reset link training can begin again along withdevice discovery.

There are some implementation dependent choices that can be made on an FPGA when a PERSTis detected. Once example is to reload the FPGA image from Flash onboard the card. If the entireFPGA is reloaded this is essentially the same as a Fundamental Reset.

7.2.1. Fundamental Reset (PERST)Fundamental reset is triggered when PERST is applied. There are two types of fundamental resets:cold and warm. A cold reset is when power is applied to the device, and a warm reset is whenPERST is asserted with power already applied to the device.

Asserting PERST has the same effect for both cold and warm resets. The device is restored to thepower-on state.




Non-standard Track

Appendix A. GlossaryAFU Accelerator function unit.

architecture A detailed specification of requirements for a processor or computer system. It does not specify detailsof how the processor or computer system must be implemented; instead it provides a template for afamily of compatible implementations.

AUI Accelerator unit interface.

big endian A byte-ordering method in memory where the address n of a word corresponds to the most-significantbyte. In an addressed memory word, the bytes are ordered (left to right) 0, 1, 2, 3, with 0 being the most-significant byte. See little endian.

cache High-speed memory close to a processor. A cache usually contains recently-accessed data or instruc-tions, but certain cache-control instructions can lock, evict, or otherwise modify the caching of data orinstructions.

caching inhibited A memory update policy in which the cache is bypassed, and the load or store is performed to or fromsystem memory.

A page of storage is considered caching inhibited when the ‘I’ bit has a value of ‘1’ in the page table.Data located in caching inhibited pages cannot be cached at any memory hierarchy that is not visibleto all processors and devices in the system. Stores must update the memory hierarchy to a level that isvisible to all processors and devices in the system.

OPPA OpenCAPI Power Platform Architecture.

Defines an architecture for loosely coupled coherent accelerators. The OpenCAPI Power PlatformArchitecture provides a basis for the development of accelerators coherently connected to a POWERprocessor.

CAPI Coherent Accelerator Processor Interface.

coherence Refers to memory and cache coherence. The correct ordering of stores to a memory address, andthe enforcement of any required cache write-backs during accesses to that memory address. Cachecoherence is implemented by a hardware snoop (or inquire) method, which compares the memoryaddresses of a load request with all cached copies of the data at that address. If a cache contains amodified copy of the requested data, the modified data is written back to memory before the pendingload request is serviced.

CRC Cyclic redundancy check.

DAR Data Address Register.

DMA Direct memory access. A technique for using a special-purpose controller to generate the source anddestination addresses for a memory or I/O transfer.

EA Effective address.

An address generated or used by a program to reference memory. A memory-management unittranslates an effective address to a virtual address, which it then translates to a real address (RA) thataccesses real (physical) memory. The maximum size of the effective-address space is 2 64 bytes.

ECC Error correction code.

A code appended to a data block that can detect and correct bit errors within the block.

ERAT Effective-to-real-address translation, or a buffer or table that contains such translations, or a table entrythat contains such a translation.

ESID Effective segment ID.

exception An error, unusual condition, or external signal that can alter a status bit and will cause a correspondinginterrupt, if the interrupt is enabled. See interrupt.

fetch Retrieving instructions from either the cache or system memory and placing them into the instructionqueue.

guarded Prevented from responding to speculative loads and instruction fetches. The operating system typicallyimplements guarding, for example, on all I/O devices.

HAUR Hypervisor accelerator utilization record.

HCA Hot/cold assist.




Non-standard Track

hypervisor A control (or virtualization) layer between hardware and the operating system. It allocates resources,reserves resources, and protects resources among (for example) sets of AFUs that might be runningunder different operating systems.

implementation A particular processor that conforms to the architecture but might differ from other architecture-compliantimplementations for example in design, feature set, and implementation of optional features.

INT Interrupt.

A change in machine state in response to an exception.

ISA Instruction set architecture.

ISN Interrupt source number.

IVT Interrupt vector table.

IVTE Interrupt vector table entry.

KB Kilobyte.

LA A local storage (LS) address of a OSL list. It is used as a parameter in a OSL command.

LISN Logical interrupt source number.

little endian A byte-ordering method in memory where the address n of a word corresponds to the least-significantbyte. In an addressed memory word, the bytes are ordered (left to right) 3, 2, 1, 0, with 3 being the most-significant byte. See big endian.

LPAR Logical partitioning.

A function of an operating system that enables the creation of logical partitions.

LPC Lowest point of coherency.

LPID logical-partition identity.

LSb Least-significant bit.

The bit of least value in an address, register, data element, or instruction encoding.

main storage The effective-address space. It consists physically of real memory (whatever is external to the memory-interface controller), local storage, memory-mapped registers and arrays, memory-mapped I/O devices,and pages of virtual memory that reside on disk. It does not include caches or execution-unit registerfiles.

mask A pattern of bits used to accept or reject bit patterns in another set of data. Hardware interrupts areenabled and disabled by setting or clearing a string of bits, with each interrupt assigned a bit position in amask register.

MB Megabyte.

memory coherency An aspect of caching in which it is ensured that an accurate view of memory is provided to all devicesthat share system memory.

memory mapped Mapped into the Coherent Attached Accelerator’s addressable-memory space. Registers, local storage(LS), I/O devices, and other readable or writable storage can be memory-mapped. System softwaredoes the mapping.

MMIO Memory-mapped input/output. See memory mapped.

MMU Memory management unit. A functional unit that translates between effective addresses (EAs) usedby programs and real addresses (RAs) used by physical memory. The MMU also provides protectionmechanisms and other functions.

MSb Most-significant bit.

The highest-order bit in an address, registers, data element, or instruction encoding.

MSR Machine State Register.

no-op No-operation. A single-cycle operation that does not affect registers or generate bus activity.

OS Operating system.

page A region in memory. The Power Architecture defines a page as a 4 KB area of memory, aligned on a 4KB boundary or a large page size that is implementation dependent.

PBA Pending bit array.

PBT Push block transfer.

PCE PCIe Configuration Environment.

PCIe Peripheral Component Interface Express.




Non-standard Track

PEAM Process element authority mask.

POR Power-on reset.

POWER Of or relating to the Power ISA or the microprocessors that implement this architecture.

Power ISA A computer architecture that is based on the third generation of reduced instruction set computer (RISC)processors.

privileged mode Also known as supervisor mode. The permission level of operating system instructions. The instructionsare described in PowerPC Architecture, Book III and are required of software that accesses system-critical resources.

problem state The permission level of user instructions. The instructions are described in Power ISA, Books I and IIand are required of software that implements application programs.

OSL OpenCAPI Service Layer.

It is the interface logic for a coherently attached accelerator and provides two main functions: movesdata between accelerator function units (AFUs) and main storage, and synchronizes the transfers withthe rest of the processing units in the system.

PTE Page table entry.

A table that maps virtual addresses (VAs) to real addresses (RAs) and contains related protectionparameters and other information about memory locations.

RA Real address.

An address for physical storage, which includes physical memory, local storage (LS), and memorymapped I/O registers. The maximum size of the real-address space is 2 62 bytes.

ROM Read-only memory.

SLB Segment lookaside buffer. It is used to map an effective address to a virtual address.

slbia SLB invalidate all instruction.

slbie SLB invalidate entry instruction.

slbmte SLB move to entry instruction.

SST Storage segment table.

STABORG Segment table origin.

STE Segment table entry.

storage model A OPPA-compliant accelerator implements a storage model consistent with the Power ISA.

sync Synchronize instruction.

synchronization The process of arranging storage operations to complete in the order of occurrence.

system software Software that has access to the privileged modes of the architecture. System software is a reference tohypervisor and operating system software.

TAG OSL command tag.

tag group A group of OSL commands. Each OSL command is tagged with an n-bit tag group identifier. An AFU canuse this identifier to check or wait on the completion of all queued commands in one or more tag groups.

TLB Translation lookaside buffer. An on-chip cache that translates virtual addresses (VAs) to real addresses(RAs). A TLB caches page-table entries for the most recently accessed pages, thereby eliminating thenecessity to access the page table from memory during load-store operations.

tlbie Translation lookaside buffer invalidate entry instruction.

VA Virtual address. An address to the virtual-memory space, which is typically much larger than the realaddress space and includes pages stored on disk. It is translated from an effective address by asegmentation mechanism and used by the paging mechanism to obtain the real address (RA). Themaximum size of the virtual-address space is 2 65 bytes.

VPD Vital product data.

VPN Virtual page number. The number of the page in virtual memory.

VSEC Vendor-Specific Extended Configuration Capability

VSID Virtual segment ID.

WIMG bits Four bits in the page table, also called a page-table entry, which control the processor's accesses tocache and main storage. ‘W’ stands for write through, ‘I’ for cache inhibit, ‘M’ for memory coherence, and‘G’ for guarded storage.




Non-standard Track

word Four bytes.




Non-standard Track

Appendix B. OpenPOWER FoundationoverviewThe OpenPOWER Foundation was founded in 2013 as an open technical membership organiza-tion that will enable data centers to rethink their approach to technology. Member companies areenabled to customize POWER CPU processors and system platforms for optimization and innova-tion for their business needs. These innovations include custom systems for large or warehousescale data centers, workload acceleration through GPU, FPGA or advanced I/O, platform optimiza-tion for SW appliances, or advanced hardware technology exploitation. OpenPOWER members areactively pursing all of these innovations and more and welcome all parties to join in moving the stateof the art of OpenPOWER systems design forward.

To learn more about the OpenPOWER Foundation, visit the organization website atopenpowerfoundation.org.

B.1. Foundation documentationKey foundation documents include:

• Bylaws of OpenPOWER Foundation

• OpenPOWER Foundation Intellectual Property Rights (IPR) Policy

• OpenPOWER Foundation Membership Agreement

• OpenPOWER Anti-Trust Guidelines

More information about the foundation governance can be found at openpowerfoundation.org/about-us/governance.

B.2. Technical resourcesDevelopment resouces fall into the following general categories:

• Foundation work groups

• Remote development environments (VMs)

• Development systems

• Technical specifications

• Software

• Developer tools

The complete list of technical resources are maintained on the foundation Technical Resources webpage.

http://openpowerfoundation.org

https://members.openpowerfoundation.org/document/dl/635




http://openpowerfoundation.org/about-us/governance/

http://openpowerfoundation.org/about-us/governance/

http://openpowerfoundation.org/technical/working-groups/

http://openpowerfoundation.org/technical/technical-resources/development-environmentvm/

http://openpowerfoundation.org/technical/technical-resources/development-systems/

http://openpowerfoundation.org/technical/technical-resources/technical-specifications/

http://openpowerfoundation.org/technical/technical-resources/software/

http://openpowerfoundation.org/technical/technical-resources/openpower-developer-tools/

http://openpowerfoundation.org/technical/




Non-standard Track

B.3. Contact the foundationTo learn more about the OpenPOWER Foundation, please use the following contact points:

• General information -- <[email protected]>

• Membership -- <[email protected]>

• Technical Work Groups and projects -- <[email protected]>

• Events and other activities -- <[email protected]>

• Press/Analysts -- <[email protected]>

More contact information can be found at openpowerfoundation.org/get-involved/contact-us.

http://openpowerfoundation.org/get-involved/contact-us/

Documents

OpenCAPI Power Platform Architectureopenpowerfoundation.org/wp-content/uploads/resources/oppa-1/opp… · 18-01-2019 · This document defines the OpenCAPI Power Platform Architecture