Cloud Key ManagementService—Deep Dive

2

Table of contents

1. Introduction 3

2. Encryption concepts and key management at Google 5

2.1. Keys, key versions, and key rings 5

2.2. Key hierarchy 6

2.3. Operations 7

3. Cloud KMS platform overview 7

3.1. Environment and dependencies 9

3.1.1. Cloud KMS Borg jobs 9

3.1.1.1. Cloud KMS API serving jobs 9

3.1.1.2. Cloud KMS batch jobs 9

3.1.1.3.CloudKMSkeysnapshotter 10

3.1.2.Client-servercommunications 10

4. Cloud KMS platform architectural details 11

4.1. Security of key materials 11

4.2. Datastore protection 12

4.2.1. Master Keys 12

4.2.2. Rotation policy 12

4.2.3. Data residency 13

4.3.Keyavailabilityaftercreation 12

4.4.CloudKMSsoftwarebackend:SOFTWARE

protection level 13

4.4.1. Algorithmic implementations 13

4.4.2. Random number generation and entropy 13

4.5.CloudKMSHSMbackend:HARDWARE

protection level 14

4.5.1. Cavium HSMs 14

4.5.2. HSM key hierarchy 14

4.5.3. Datastore protection 15

4.5.4. Rotation policy 15

4.5.5. Provisioning and handling process 15

4.5.6.Vendor-controlledfirmware 16

4.5.7. Regionality 16

4.6.CloudKMS:Keyimport 16

4.7. Customer-managed encryption keys (CMEK) 17

4.8. Lifecycle of a Cloud KMS request 18

4.9.Destructionofkeymaterial 20

5. Cloud KMS platform operational environment 20

5.1.Softwareengineers,sitereliability

engineers,andsystemoperators 20

5.2. Regionality of Cloud KMS resources 21

5.2.1. Cloud regions and data centers 21

5.2.2. Regionality 22

5.2.3. Regionality monitoring 23

5.3. Authentication and authorization 23

5.4. Logging 23

5.4.1. Cloud audit logs 23

5.4.2.AccessTransparencylogs 23

5.4.3. Internal request logs 24

5.5. Datastore backend 24

6. Further reading 25

Authors: SonalShah,Il-SungLee,WalterPourpore,HunterFreyer,HonnaSegel,DwightWorley

Acknowledgements: AdrianSears,JohnRandolph,TimDierks,ChrisRezek,StanleyMcKenzie,

KevinPlybon,DavidHale,SergeySimakov,DavidU.Lee,CarrieMcDaniel,AntonChuvakin,DaveTonisson

Cloud Key Management Service — Deep Dive

3

1. Introduction

GoogleCloudworksoffthefundamentalpremisethatGoogleCloudcustomersowntheirdataandshouldcontrolhowitisused.

WhenyoustoredatawithGoogleCloud,yourdataisencrypted at restbydefault.WhenyouuseourCloud KeyManagementService(CloudKMS)platform,you cangaingreatercontroloverhowyourdataisencryptedatrestandhowyourencryptionkeysaremanaged.

TheCloudKMSplatformletsGoogleCloudcustomersmanage cryptographic keys in a central cloud service for either direct use or use by other cloud resources and applications.Forthesourceofkeys,CloudKMSprovidesthefollowingoptions:

TheCloudKMSsoftwarebackendgivesyoutheflexibilitytoencryptyourdatawitheitherasymmetric or asymmetric key that you control (Cloud KMS).

Ifyouneedahardwarekey,youcanuseCloud HSM to help ensure that your symmetric and asymmetric keys are only used in FIPS140-2Level3–validatedHardwareSecurityModules(HSMs).

Cloud KMS lets you importyourowncryptographickeys in case you need to use keys that you generate yourself.

YoucanchoosetousekeysgeneratedbyCloudKMSwithotherGoogleCloudservices.Suchkeysareknownascustomer-managed encryption keys (CMEK).TheCMEKfeatureletsyougenerate, use, rotate, and destroy encryption keys that are used to help protect data in other Google Cloud services.

WithCloudExternalKeyManager(CloudEKM), you can create and manage keys in a key manager externaltoGoogleCloud,andthenyousetuptheCloudKMSplatformtousetheexternalkeystohelpprotectyourdataatrest.Youcanusecustomer-managedencryptionkeyswithaCloudEKMkey.CloudEKMisbeyondthescopeofthiswhitepaperatthistime.

Cloud Key Management Service — Deep Dive

4

Googlealsosupportscustomer-supplied encryption keys (CSEK) for Compute Engine and Cloud Storage, wherebydataisdecryptedandencryptedusingakeythat’sprovidedontheAPIcall.CSEKisbeyondthescopeofthispaper.Formoreinformation,seetheonline documentation.

WithCloudKMS,Google’sfocusistoprovideascalable,reliable,andperformantsolution,withthewidestspectrumofoptionsthatyoucancontrol,onaplatformthatisstraightforwardtouse.CloudKMSissupported byfivedesignpillars:

Customer control.CloudKMSletsyoumanagesoftwareandhardwareencryptionkeysorsupplyyourownkeys.

Access control and monitoring.WithCloudKMS,youcanmanagepermissionsonindividualkeysandmonitorhowthekeysareused.

Regionality.CloudKMSoffersregionalizationoutofthebox.Theserviceisconfiguredtocreate,store,andprocesssoftwarekeysonlyintheGoogleCloudregionthatyouselect.

Durability.CloudKMSmatchesthehighestdurabilitystandardsonGoogleCloud.Tohelpguard against data corruption and to verify that data can be decrypted successfully, Cloud KMS periodically scans and backs up all key material and metadata.

Security.CloudKMSoffersstrongprotectionsagainstunauthorizedaccesstokeysandisfullyintegratedwithIdentityandAccessManagement(IAM)andCloudAuditLogscontrols.

Current Google Cloud portfolio

DEFAULT ENCRYPTION

Google’sdefaultdata-at-rest

encryption. Customer has no access to keys or control of key rotation.

CLOUD KMS

Customer can manege keys

generated and stored by Google.

CLOUD HSM

Customer can manege keys

generated and stored by Google

and stored in a Google-ownedand

operated HSM

EXTERNAL KEY MANAGER

Keys stored in a management

system of customer’schoice,

even outside of Google Cloud.

CUSTOMER-SUPPLIED

ENCRYPT. KEYS

Keysownedbycustomer and provided

on each API call to be used ephemerally

to access datat.

KEY IMPORT

Customer can importkeysused

and managed outside of Google,

and use key material in a

Google HSM.

SOURCE OF KEY MATERIAL

GOOGLE CONTROLS KEY MATERIAL CUSTOMER CONTROLS KEY MATERIAL

ThispaperfocusesontheinnerworkingsoftheCloudKMSplatformthatareintheGeneralAvailability(GA)release.TheseoptionshelpyouprotectthekeysandothersensitivedatathatyoustoreinGoogleCloud.ThispaperisintendedfortechnicaldecisionmakerswhoneeddetailsaboutCloudKMSarchitecture,infrastructure,andfeatures.Thispaperassumesthatyouhaveabasicunderstandingofencryptionconceptsandcloudarchitectures.

Cloud Key Management Service — Deep Dive

5

2. Encryption concepts and key management at Google

ThissectiondefinessomeofthetermsanddefinitionsforkeymanagementinthecontextofGoogle’smulti-layered key management infrastructure.

2.1. Keys, key versions, and key ringsThissectiondescribeskeys,keyversions,andthegroupingofkeysintokeyrings.Thefollowingdiagramillustrateskeygroupings.Relateddefinitionsfollowthediagram.

Key: Anamedobjectrepresentingacryptographickeythatisusedforaspecificpurpose.Thekeymaterial— theactualbitsusedforcryptographicoperations—canchangeovertimeasyoucreatenewkeyversions.

Keypurposeandotherattributesofthekeyareconnectedwithandmanagedusingthekey.Thus,thekeyis themostimportantobjectforunderstandingKMSusage.

CloudKMSsupportsbothasymmetrickeys and symmetric keys. A symmetric key is used for symmetric encryption to protect some corpusofdata—forexample,usingAES-256 inGCMmodetoencryptablockofplaintext. An asymmetric key can be used for asymmetric encryption, or for creating digital signatures. Supportedpurposesandalgorithms are described in the Cloud KMS documentation.

Key ring: A grouping of keys for organizational purposes. A key ring belongs to a Google Cloud projectandresidesinaspecificlocation.Keysinherit IAM policies from the key ring that containsthem.Groupingkeyswithrelatedpermissions in a key ring lets you grant, revoke, or modify permissions to those keys at the key ringlevelwithoutneedingtoactoneachkeyindividually. Key rings provide convenience and categorization, but if the grouping of key rings is not useful to you, you can manage permissions directly on keys.

Cloud Key Management Service — Deep Dive

6

Topreventresourcenamecollisions,akeyringcannotbedeleted.Keyringsandkeysdonothavebillablecosts orquotalimitations,sotheircontinuedexistencedoesnotaffectcostsorproductionlimits.Formoredetailsondeletion of key and key materials, see the section about deletion later in this paper.

Key metadata:Resourcenames,propertiesofKMSresourcessuchasIAMpolicies,keytype,keysize,key state,andanydataderivedfromtheabove.Keymetadatacanbemanageddifferentlythanthekeymaterial.

Key version:Representsthekeymaterialassociatedwithakeyatsomepointintime.Thekey version is the resourcethatcontainstheactualkeymaterial.Versionsarenumberedsequentially,beginningwithversion1.Whenakeyisrotated,anewkeyversioniscreatedwithnewkeymaterial.Thesamelogicalkeycanhavemultipleversionsovertime,thuslimitingtheuseofanysingleversion.Symmetrickeyswillalwayshaveaprimaryversion.Thisversionisusedforencryptingbydefault.WhenCloudKMSperformsdecryptionusingsymmetrickeys,itautomaticallyidentifieswhichkeyversionisneededtoperformthedecryption.

2.2. Key hierarchyThefollowingdiagramillustratesthekeyhierarchyofGoogle’sinternalKeyManagementService.CloudKMSleveragesGoogle’sinternalKMSinthatCloudKMS-encryptedkeysarewrappedbyGoogleKMS.CloudKMSusesthesamerootoftrustasGoogleKMS.Relateddefinitionsfollowthediagram.

Data encryption key (DEK): A key used to encrypt data.

Key encryption key (KEK):Akeyusedtoencrypt,orwrap,adataencryptionkey.AllCloudKMSplatformoptions(software,hardware,andexternalbackends)letyoucontrolthekeyencryptionkey.

KMS Master Key: Thekeyusedtoencryptthekeyencryptionkeys(KEK).Thiskeyisdistributedinmemory. TheKMSMasterKeyisbackeduponhardwaredevices.Thiskeyisresponsibleforencryptingyourkeys.

Root KMS:Google’sinternalkeymanagementservice.

Cloud Key Management Service — Deep Dive

7

2.3 Operations Project: Cloud KMS resources belong to a Google Cloud project, just like all other Google Cloudresources.Youcanhostdatainaprojectthatisdifferentfromtheprojectinwhich yourCloudKMSkeysreside.Thiscapabilitysupportsthebestpracticeofseparation of dutiesbetweenthekeyadministratorsanddataadministrators.

Locations:Withinaproject,CloudKMSresourcesarecreatedinalocation.Formoreinformation, see Geography and regions.

3.CloudKMSplatformoverview

TheCloudKMSplatformsupportsmultiplecryptographicalgorithmsandprovidesmethodstoencryptanddigitallysignusingbothhardware-andsoftware-backedkeys.TheCloudKMSplatformisintegratedwithIdentity and Access Management (IAM) and Cloud Audit Logs so that you can manage permissions onindividualkeysandaudithowtheyareused.

Cloud KMS Platform

Cloud KMS API

Independent backup systemDatastore

HSM

Cloud HSM

Cloud KMS software backend

Application

Google Cloud Console

REST, gRPC, APIs

gcloud

Cloud Key Management Service — Deep Dive

8

TheprecedingdiagramshowsthemaincomponentsoftheCloudKMSplatform.Administratorsaccesskey management services by using the Google Cloud Console or the gcloud command-line tool, or throughapplicationsimplementingtheRESTorgRPCAPIs.Applicationsaccesskeymanagementservices using a RESTAPI or gRPC. Applications can use Google services that are enabled to use customer-managedencryptionkeys(CMEK).CMEKinturnusestheCloudKMSAPI.TheCloudKMS APIletsyouuseeithersoftware(CloudKMS)orhardware(CloudHSM)keys.Bothsoftware-andhardware-basedkeysleverageGoogle’sredundantbackupprotections.

WiththeCloudKMSplatform,youcanchooseaprotection levelwhencreatingakeytodeterminewhichkeybackendcreatesthekeyandperformsallfuturecryptographicoperationsonthatkey.TheCloudKMSplatformprovidestwobackends(excludingCloudEKM),whichareexposedintheCloudKMS API as the SOFTWARE and HSMprotectionlevels.TheprotectionlevelSOFTWARE applies to keys that maybeunwrappedbyasoftwaresecuritymoduletoperformcryptographicoperations.TheprotectionlevelHSMappliestokeysthatcanonlybeunwrappedbyHardwareSecurityModulesthatperformallcryptographicoperationswiththekeys.

GoogleCloudsupportsCMEKforseveral services, including Cloud Storage, BigQuery, and Compute Engine.CMEKletsyouusetheCloudKMSplatformtomanagetheencryptionkeysthattheseservicesuse to help protect your data.

CloudKMScryptographicoperationsareperformedbyFIPS140-2–validatedmodules.Keyswithprotection level SOFTWARE,andthecryptographicoperationsperformedwiththem,complywithFIPS140-2Level1.KeyswithprotectionlevelHSM,andthecryptographicoperationsperformedwiththem,complywithFIPS140-2Level3.

Cloud Key Management Service — Deep Dive

9

3.1. Environment and dependenciesThissectionprovidesdetailsabouttheGoogleinfrastructurethattheCloudKMSplatformruns on and the communications protocols that the infrastructure uses.

3.1.1. Cloud KMS Borg jobsCloudKMSplatformelements run as production Borg jobs. Borg isGoogle’slarge-scaleclustermanager for handling API serving jobsandbatchjobs.Fordetailsabout the security of our physical and production infrastructure, see Google Infrastructure Security DesignOverview.

Counting active keys for customer billing.

AggregatingresourcesfromCloudKMS’publicprotocolbufferAPIandforwardingthemetadata to Cloud Asset Inventory. ResourcesinthiscontextareanyresourcesthatCloudKMSmanages—forexample,keysandkeyrings.

Aggregatingallresourcesandreportinginformationforbusinessanalytics.

Snapshottingdataforhighavailability.

Validating that all data stored in the underlying datastore is uncorrupted.

Re-encryptingcustomerkeymaterialwhentheKMSMasterKeyisbeingrotated.

3.1.1.2. Cloud KMS batch jobsTheCloudKMSplatformalsomaintainsanumberofbatchjobsthatrunasproductionBorg jobsonvariousschedules.Batchjobsareregion-specificandonlyaggregatedatafrom,andrunwithin,theassociatedGoogleCloudregion.Tasksthatthesejobsperformincludethefollowing:

3.1.1.1. Cloud KMS API serving jobsCloud KMS API serving jobs are production Borg jobs thatservecustomers’requeststomanageandusetheirkeys.TheseservingjobsrunineveryGoogleCloudregionwhereCloud KMS is available. Each job isassociatedwithasingleregionandisconfiguredtoonly access data from systems geographically located withintheassociatedGoogleCloudregion.Formoreinformation on key residency, see Regionality later in this paper.

Cloud Key Management Service — Deep Dive

10

3.1.1.3. Cloud KMS key snapshotterTomaintainhighavailability,theCloudKMSplatformmaintainsaredundantdatastoreineach instance of a shared service hosting the Cloud KMS API server tasks. Each service deploysitsowninstanceofthesnapshotterservice.Thisredundancymakesserviceshighly independent so that a failure in one zonehasalimitedeffectonotherzones.WhentheCloudKMSAPIjobneedstoperformacryptographicoperation,itqueriestheprimarydatastorealongwiththelocalsnapshotterjobsinparallel.TheCloudKMSAPIthenuseswhicheverjobsuccessfullycompletestherequestfirst.Topreventadelayinthesnapshottingpipelinethatmightcauseexcessivelystaledatatobeserved, the API server does not use results from thesnapshotterjobsifthedataismorethan twohoursold.Snapshotsarecreatedastheoutput of a batch job that runs continuously for each region. Snapshots reside in the cloudregionassociatedwiththekeymaterial.

A peer-to-peer authentication protocol for applications

A record encryption protocol

AnAPIthatexposestheauthenticateduserforauthorization

3.1.2. Client-server communicationsGoogleusesApplicationLayerTransportSecurity(ALTS) to provide security for client-server communications (encryption in transit) that use Google’sremoteprocedurecall(RPC)mechanism.ALTSprovidesthefollowing:

TheALTShandshakingandrecordencryptionprotocols aresimilartotheTransportLayerSecurity(TLS)handshaking and recordprotocols.ALTSmakesdifferenttradeoffstooptimizeforGoogle’sproductionenvironment,andALTS isfullyintegratedintotheentireproductionhardwareandsoftwarestack.Formoredetails,seeCloudKMSplatformoperational environment later in this paper.

Cloud Key Management Service — Deep Dive

11

4.CloudKMSplatformarchitecturaldetails

Thissectiondivesintoarchitecturaldetails,startingwiththesecurityofkeymaterialand datastore protection.Itthendetailstheoptionsforthesourceofkeymaterial:

● TheCloudKMSsoftwarebackend

● TheCloudKMSHardwareSecurityModule(HSM)backend

● TheCloudKMSkeyimport

Thissectionalsodescribestheoptionsforcustomer-managed encryption keys (CMEK).

Toprovideadynamicviewofhowthearchitecturalstructuresintroducedsofarareused,thispaperdescribes the lifecycle of a Cloud KMS request, including a discussion of the destruction of key material.

Whenthecustomerisusingit.

WhenGoogle’sinternalkeyused to protect customer key material is being rotated or checked for integrity.

Customer requests to Cloud KMS are encrypted intransitbyusingTLSbetweenthecustomerand the GoogleFrontEnd(GFE). Application LayerTransportSecurity(ALTS),describedearlier in the section about client-server communications, is used for encryption betweenCloudKMSjobsanditsbackendsindifferentdatacenters.ALTSandencryptionintransit are described in more detail in Lifecycle of a Cloud KMS request later in this paper.

4.1. Security of key materialsInCloudKMS,thekeymaterialisalwaysencrypted at rest and in transit. Key material isdecryptedonlyinthefollowingcases:

AuthenticationoccursbetweenallKMSjobs,bothwithinandbetweenGoogledatacenters.Google’spolicy is to ensure that jobs use only source code thathasbeenbuilt,tested,andreviewedinaverifiablemanner. Binary Authorization for Borg (BAB) enforces this policy at the operational level and is described in more detail in this security documentation.

Cloud KMS API jobs can access key metadata (for example,theIAMpolicyortherotationperiod).AGoogleemployeewithavalid,documentedbusinessjustification(suchasabugoracustomerissue) can also access key metadata. Accesses are logged internally,andlogspertainingtodatacoveredbyAccessTransparencyarealsoavailabletoqualifiedcustomers.

Keymaterial,however,cannotbeaccessedbyCloudKMSAPIjobs,andkeymaterialcannotbeexported orviewedthroughtheAPIinterfaceoranotheruserinterface.NoGoogleemployeehasaccesstounencrypted customer key material. Key material is additionallyencryptedwithaMasterKeyinRootKMS,whichcannotbedirectlyaccessedbyanyindividual.

Cloud Key Management Service — Deep Dive

12

4.2. Datastore protectionThissectiondescribeshowkeydataisprotectedbyGoogleKMS.

4.3. Key availability after creationCloudKMSallowsakeytobeusedimmediatelyaftertheCloudKMSdatastorecommitsthetransactionthatcreatesthekeyandafterthestoragelayeracknowledgesitscreation.

4.2.1. Master KeysCloudKMSusesaMasterKeytowrapallcustomerkeysatrest.EachCloudKMSserverfetchesacopyoftheMasterKeyduringstartupasaharddependency,andanewcopyoftheMasterKeyisretrievedeveryday.TheMasterKeyisre-encrypted periodically.

4.2.2. Rotation policyKeyrotationispartofthegenerallyacceptedbestpracticesforthehandling ofkeymaterial.ArotationpolicyexistsforthekeysusedtoprotectCloudKMSdatastores. A key rotation policy is also applied to the Google internal KMS MasterKeysthatwraptheCloudKMSkeys.TheKMSMasterKeyhasascheduledciphertextmaximumageof90dayswithaclientcachedkeymaximumageofoneday.CloudKMSrefreshestheMasterKeysinKMS tasks every 24 hours and re-encrypts all customer keys on a monthly basis.

4.2.3. Data residencyThedataunderlyingeachCloudKMSdatastoreremainsexclusivelywithintheGoogleCloudregionwithwhichthedataisassociated.Thisappliestolocationsthataremulti-regionsaswell,forexample,theusmulti-region.Formoredetailson data residency, see Regionality later in this paper.

Cloud Key Management Service — Deep Dive

13

4.4. Cloud KMS software backend: SOFTWARE protection levelTheprotectionlevelSOFTWAREappliestokeyswhosecryptographicoperationsareperformedinsoftware.Thissectiondescribesthedetailsofhowthislevelisimplemented.

4.4.1. Algorithmic implementationsForsoftwarekeys,CloudKMSusestheBoringCryptomodule(BCM)withinGoogle’sBoringSSL implementation for all related cryptographicwork.TheBCMisFIPS140-2validated.TheCloudKMSbinaryisbuiltagainstFIPS140-2 Level 1–validated Cryptographic Primitivesofthismodule.Themostcurrentalgorithms covered by this module are listed on our documentation page, and include encrypt, decrypt,andsignoperationswithAES256-GCMsymmetricandRSA2048,RSA3072,RSA4096,EC P256, and EC P384 asymmetric cryptographic keys.

4.4.2. Random number generation and entropyWhengeneratingencryptionkeys,CloudKMSusesBoringSSL.FIPS140-2requiresthatits ownPRNGsbeused(alsoknownasDRBGs). InBoringCrypto,CloudKMSexclusivelyusesCTR-DRBGwithAES-256.ThisprovidesoutputforRAND_bytes,theprimaryinterfacebywhichthe rest of the system gets random data. ThisPRNGisseededfromtheLinuxkernel’sRNG,whichitselfisseededfrommultipleindependententropysources.Thisseedingincludes entropic events from the data centerenvironment,suchasfine-grainedmeasurements of disk seeks and inter-packet arrival times, and Intel’sRDRANDinstruction whereavailable.Formoreinformationaboutthebehavior of the random number generator for BoringSSL(FIPS-compliantmode),pleasereferto the RNGdesign.

Cloud Key Management Service — Deep Dive

14

4.5. Cloud KMS HSM backend: Hardware protection levelTheCloudHSMserviceprovideshardware-backedkeystoCloudKMS.Itofferscustomerstheabilitytomanage and use cryptographic keys that are protected byfullymanagedHardwareSecurityModules(HSMs)in Google data centers.Theserviceishighlyavailableand auto-scales horizontally. Keys are cryptographicallyboundtotheKMSregioninwhichthekeyringwasdefined.WithCloudHSM,thekeysthatyou create and use cannot be materialized outside of theclusterofHSMsbelongingtotheregionspecifiedat the time of key creation. Using Cloud HSM, you canverifiablyattestthatyourcryptographickeysarecreatedandusedexclusivelywithinahardwaredevice.NoapplicationchangesarerequiredforexistingCloudKMScustomerstouseCloudHSM:theCloudHSMservices are accessed using the same API and client librariesastheCloudKMSsoftwarebackend.

Cloud HSM uses HSMs that are FIPS140-2Level3–validatedandarealwaysrunninginFIPSmode.TheFIPSstandardspecifiesthecryptographicalgorithmsand random number generation used by the HSMs.

4.5.1. Cavium HSMsTheCaviumHSMPCIecardisvalidatedbythevendortobeFIPS140-2Level3–compliant.Thecurrentcertificateisavailable on request.

4.5.2. HSM key hierarchyInthefollowingdiagram,weseetheCloud KMS in the top half of the diagram. CloudHSMwrapscustomerkeys,andthenCloudKMSwrapstheHSMkeysthatarepassedtoGoogle’sdatastore.

Cloud Key Management Service — Deep Dive

15

TheCloudHSMhasakey(notshown)whichcontrolsthemigrationofmaterialinsidetheCloud HSM administrative domain. A region might have multiple HSM administrative domains.

TheHSMRootKeyhastwocharacteristics:

CustomerkeyswrappedbyHSMRootKeyscanbeusedontheHSM,buttheHSMneverreturns theplaintextofthecustomerkey;theHSMonlyusesthecustomerkeyforoperations.

4.5.3. Datastore protectionHSMsarenotusedaspermanentstorageforkeys;theystorekeysonlywhiletheyarebeingused.Because HSM storage is constrained, the HSM keys are encrypted and stored in the Cloud KMS keydatastore.ThissubjectisdescribedindetailinDatastore backend later in this paper.

4.5.4. Rotation policySeveraltypesofkeysareinvolvedinCloudHSM’skeyprotectionstrategy.Customersrotatetheir ownkeysandrelyonCloudKMStorotatetheHSMkeys.

4.5.5. Provisioning and handling processProvisioningofHSMsiscarriedoutinalabequippedwithnumerousphysicalandlogicalsafeguards,including,forexample,multi-partyauthorizationcontrolstohelppreventsingle-actorcompromise.

ThefollowingareCloudHSMsystem-levelinvariants:

ItisgeneratedonanHSMandthroughoutitslifetimeneverleavesthewell-definedboundaries of HSMs. It may be replicated on other HSMs, or on backup HSMs.

ItcanbeusedasakeyencryptionkeytodirectlyorindirectlywrapcustomerkeysthatHSMs use.

1

2

Customerkeyscannotbeextractedasplaintext.

Customer keys cannot be moved outside the region of origin.

AllconfigurationchangestoprovisionedHSMsareguardedthroughmultiplesecuritysafeguards.

Administrativeoperationsarelogged,adheringtoseparationofdutiesbetweenCloudHSM administrators and logging administrators.

HSMsaredesignedtobeprotectedfromtampering(suchasbytheinsertionofmalicioushardwareorsoftwaremodifications,orunauthorizedextractionofsecrets)throughoutthe operational lifecycle.

1

2

3

4

5

Cloud Key Management Service — Deep Dive

16

4.5.6. Vendor-controlled firmwareHSMfirmwareisdigitallysignedby the HSM vendor. Google cannot createorupdatetheHSMfirmware.Allfirmwarefromthevendorissigned,includingdevelopmentfirmwarethatis used for testing.

4.6. Cloud KMS: Key import Youmaywanttoimportyourownkeysintoyourcloudenvironment.Forexample,youmighthavearegulatoryrequirementthatthekeysusedtoencryptyourclouddataaregeneratedinaspecificmannerorenvironment.CloudKMSletsyouimportyourowncryptographickeysthatyoucreatedonpremises orinanExternalKeyManager.Keyimportletsyouimportthesekeysandmeettheseregulatoryobligations.Youcanalsoimportasymmetrickeystoextendyoursigningcapabilitiesto the cloud.

Aspartofkeyimport,CloudKMSgeneratesawrappingkeywhichisapublic/privatekeypair,usingone of the supportedimportmethods.Encryptingyourkeymaterialwiththiswrappingkeyprotectsthekeymaterial in transit.

ThisCloudKMSpublicwrappingkeyisusedtoencrypt,ontheclient,thekeytobeimported.TheprivatekeymatchingthispublickeyisstoredwithinGoogleCloudandisusedtounwrapthekeyafteritreachestheGoogleCloudproject.Theimportmethodyouchoosedeterminesthealgorithmusedtocreatethewrappingkey.Afteryourkeymaterialiswrapped,youcanimportitintoanewkeyorkeyversionon Cloud KMS.

ImportedkeyscanbeusedwiththeHSM or SOFTWARE protection level.

ForCloudHSM,theprivatekeyportionofthewrappingkeyisavailableonlywithinCloudHSM.RestrictingtheprivatekeyportiontoCloudHSMpreventsGooglefromunwrappingyourkeymaterialoutsideofCloud HSM.

4.5.7. RegionalityCustomerkeysareassignedtospecificgeographic regionsasaresultoftheirbindingtoaspecificHSMRootKey.Forexample,akeycreatedspecificallyintheus-west1 region cannot migrate into the us-east1 region, or a us multi-region. Similarly , a key created in the us multi-region cannot migrate into or out of the us-west1 region.

Cloud Key Management Service — Deep Dive

17

4.7. Customer-managed encryption keys (CMEK)By default, Google Cloud encrypts all customer data stored at rest,withGooglemanagingthekeysusedforencryption.ForsomeGoogleCloudproducts,customerscaninsteaduseCloudKMStomanagethekeysusedforencryption.CMEKcanbeusedforbothsoftware-andhardware-backedkeys.CloudKMSletsthecustomercontrolmanyaspectsofkeys(suchascreating,enabling/disabling,rotating,anddestroyingkeys)andmanageappropriate IAM permissions on them. Cloud KMS includes severalpredefinedIAMrolesthathavespecificprivilegesandlimitationsandcanbegrantedtospecificusersandserviceaccountsinordertoenforcearecommended separation of duties.

ThefollowingtopicsprovidedetailsonproductsintegratedwiththeCloudKMSplatformthatsupportCMEK:

BigQuery Compute Engine

Cloud Storage

Dataproc Container Registry

TheimpactofkeyrotationonthekeyversionuseddependsonhowaproductimplementsCMEKs.Ingeneral,rotatingtheCloudKMSkeydoesn’tre-encryptthedataintheassociatedCMEKservice.Forexample,BigQuerydoesnotautomaticallyrotateatableencryptionkeywhentheCloudKMSkeyassociatedwiththetablerotates.ExistingBigQuerytablescontinuetousethekeyversionwithwhichtheywerecreated.NewBigQuerytablesusethecurrentkeyversion.Formoreinformation,seethedocumentationforeachproduct.

See this documentation for the full list of CMEK-integrations and CMEK-compliant services. Google continues to invest in CMEK integration for other Google Cloud products.

Cloud Key Management Service — Deep Dive

18

4.8. Lifecycle of a Cloud KMS requestToprovideadynamicviewofhowthearchitecturalstructuresintroducedsofarareused,thissectiondescribes the lifecycle of a Cloud KMS request, including a discussion of the destruction of key material.

Cloud Key Management Service — Deep Dive

19

A customer, or a job running on behalf of a customer, composes a request to the Cloud KMS service,https://cloudkms.googleapis.com.DNSresolvesthisaddresstoananycastIPaddressoftheGoogleFrontEnd(GFE).

TheGFEprovidespublicIPhostingofitspublicDNSname,DenialofService(DoS)protection,andTLStermination.Whenthecustomersendstheirrequest,itisgenerallyroutedtoaGFEneartothecustomer,regardlessofwhichlocationthecustomer’srequestistargetedfor.GFEshandletheTLSnegotiationand,usingtherequestURLandparameters,determinewhatGlobalSoftwareLoadBalancer(GSLB)routestherequest.

ThereisaseparateGSLBtargetforeachCloudKMSregion.IftherequestarrivesatGoogle in us-east1, and the request is destined for us-west1,therequestisroutedbetweentheus-east1 and us-west1datacenters.Allcommunicationbetweendatacentersisencrypted intransitusingALTS,whichmutuallyauthenticatestheGFEandCloudKMSjobs.

WhentherequestreachestheCloudKMSjob,itisfirstprocessedbyaframeworkthathandlesmuchoftheworkcommontoallGoogleCloudservices.Thisframeworkauthenticatestheuserandperformsanumberofcheckstoverifythefollowing:

○ Thecustomerhasavalidcredentialandcanbeauthenticated. ○ TheGoogleCloudprojecthastheCloudKMSAPIenabledandhasavalidbillingaccount. ○ Thequotaissufficienttoruntherequest. ○ ThecustomerisontheallowlisttousetherelevantGoogleCloudregion. ○ TherequestwillnotberejectedbyVPC-ServiceControls.

Afterthesecheckspass,theframeworkforwardstherequestandcredentialtoCloudKMS.CloudKMSparsestherequesttodeterminewhatresourcesarebeingaccessed,andthencheckswithIAMtoseewhetherthecallerisauthorizedtomaketherequest.IAMalsotellsCloudKMSwhethertherequestoccurrenceshouldbewrittentoauditlogs.

Once the request is authenticated and authorized, Cloud KMS calls its in-region datastore backends to fetch the requested resource. Once fetched, the customer key material is decrypted for use.

Withthekeymaterial,CloudKMSthenperformsthecryptographicoperation,forwarding thewrappedversionofthekeytoeithertheCloudKMSsoftwarebackendortheCloudHSM,depending on the protection level of the key.

Afterthecryptographicoperationiscompleted,theresponseissentbacktothecustomeralongthesametypeofGFE-to-KMSchannelastherequest.

Astheresponseisreturned,CloudKMSalsotriggerssomeeventsasynchronously:

○ Auditandrequestlogsarefilledandqueuedtobewritten. ○ Reportsaresentforbillingandquotamanagementpurposes. ○ If the request updated the metadata of a resource, the change is sent to

Cloud Asset Inventory through batch job updates.

1

2

3

4

5

9

8

7

6

The steps in this lifecycle include the following:

Cloud Key Management Service — Deep Dive

20

4.9. Destruction of key materialDeletion of data on Google Cloud is described in this whitepaper. Key material is considered to be customerdata,sotheapproachthewhitepaperdescribesappliestoCloudKMS.Also,becauseGoogleneversharesthekeymaterialoutsideofCloudKMS,thekeymaterialgetsdestroyedonrequestwhenthe24-hourpendingdeleteperiodiscompleteandbackupsareexpired.ThisprocessisnotguaranteedforcopiesofimportedkeysthatexistoutsideofCloudKMScontrol.

Whilekeymaterialisdestroyedasdescribedabove,keysandkeyringscannotbedeleted.Keyversionsalso cannot be deleted, but key version material can be destroyed so that the resources can no longer beused.Keyringsandkeysdonothavebillablecostsorquotalimitations,sotheircontinuedexistencedoesnotaffectcostsorproductionlimits.

Afterit’sscheduledfordestruction,akeyversionisnotavailableforcryptographicoperations.Withinthe 24-hour period, the user can restore the key version so that it is not destroyed.

5.CloudKMSplatformoperationalenvironment

TheoperationalenvironmentfortheCloudKMSplatformincludesdatastorageandsecuritypolicies,access restrictions, and risk mitigation strategies that are designed to optimize reliability, durability, andavailabilitywhilesafeguardingkeymaterial.Thissectiondiscussestheservice’soperatingstructure,responsibilities for operations team members, authentication mechanisms, and auditing and logging protocols.Thediscussionthatfollowsappliestoboththesoftware-andhardware-backedkeycapabilities.

5.1. Software engineers, site reliability engineers, and system operatorsSoftwareengineersareresponsibleforpartneringwithotherstakeholderssuchasproductmanagersandsitereliabilityengineers(SREs)todesignthesystemandultimatelywritemuchofthecodeandtestsfortheCloudKMSservice.Thecodeincludesthemainjobthatservescustomerrequests,as wellassecondaryjobsforactivitiessuchasdatareplicationandbilling.SREsalsomightwritecode.However,softwareengineersandSREdutiesareseparated;SREsareprimarilyresponsibleformaintaining the production environment for Cloud KMS jobs. SREs measure and achieve reliability throughengineeringandoperationswork.

Systemoperatorsareresponsibleforthebuild-and-releaseprocess,monitoring,alerting,andcapacityplanningforCloudKMS.TheyarethefirstresponderstocustomerissuesandoutagesforCloudKMS.Asanexample,systemoperatorsleveragetoolingandautomationtominimizemanualsystemswork sothattheycanfocusoneffortsthatbringlong-termvalue.

Cloud Key Management Service — Deep Dive

21

5.2. Regionality of Cloud KMS resourcesYou can create Cloud KMS resources in one of four types of Google Cloud locations:

Regional locations. A regional location consists of zonesinaspecificgeographicalplace,suchasIowa.

Dual-regional locations. A dual-regional locationconsistsofzonesintwospecificgeographicalplaces,suchasIowaandSouthCarolina.

Multi-regional locations. A multi-regional location consists of zones spread across a general geographical area, such as the United States.

The global location.Theglobal locationconsistsofzonesspreadaroundtheworld.WhenyoucreateyourCloudKMSresourcesinthegloballocation,theyareavailablefromzonesworldwide.

5.2.1. Cloud regions and data centersAGoogleCloudregioncontainsaspecificdatacenteroraspecifiedset of data centers, determined by its designation as a single-region, dual-region,multi-region,orgloballocation.FormoreinformationonGoogle Cloud regions, see Google Cloud locations.

Each data center contains racks of machines for computing, networking,orstorageofdata.ThesemachinesrunGoogle’s Borg infrastructure.

Google data centers have strict physical security requirements. Any physicalspacethatmightcontainuserdatarequiresentrywayswithbadgereadersand/oririsscannersusedtoauthenticateentrants.Doorsarenottobeheldopenformultiplepeople;eachpersonmustauthenticatethemselvesindividually.Bagsarenotpermittedtobebrought in or out of these areas, unless they are clear bags that are explicitlyauthorizedafterinspectionbysecuritypersonnel.Specialpermission is required to bring in or out any electronic device that might transmit or record data.

LocationsrepresentthegeographicalregionswhererequeststoCloudKMSforagivenresourcearehandled,andwherethecorrespondingcryptographickeysarestored.

FormoreinformationonavailableCloudKMSlocations,seetheCloud KMS documentation.

Cloud Key Management Service — Deep Dive

Cloud KMS data residency At rest, in use (single region) Atrest,inuse(dual/multi-region)

Softwarekeymaterial ResidentResident in the regions that constitutethedual/multi-region.

Hardwarekeymaterial(HSM) ResidentResident in the regions that constitutethedual/multi-region.

22

5.2.2. RegionalitySomeindustriesrequirethatcryptographickeysresideinaspecificlocation.AsintroducedaboveinRegionality of Cloud KMS resources,CloudKMSoffersfourtypesofregionallocationstohelpyou meet those requirements.

Forsingle,dual,ormulti-regionlocations,CloudKMScreates,stores,andprocessesyourcustomersoftware-andhardware-backedkeysandkeymaterialonlyinthatlocation.StorageanddataprocessingsystemsthatCloudKMSusesareconfiguredtoonlyuseresourceswithintheGoogleCloudregionassociatedwiththekeymaterial.Keymaterialcreatedindual-ormulti-regionallocationsdoesnot leave the boundaries of the selected locations.

Fortheglobalregion,thereisnoregionalityguaranteespecified.

Cloud KMS does not guarantee residency for key metadata, usage logs, or for key material in transit. Keymetadataincludesresourcenames;propertiesofCloudKMSresourcessuchaskeytype,keysize,andkeystate;IAMpolicies;andanydataderivedfrommetadata.

WhenusingCMEK,thefollowingCloudKMSgeographicrestrictionsapplytoyourkeysregardlessofthe custom, dual- or multi-regional locations that you choose in other Google Cloud products that you wanttousewithCMEK:

For a specific region:Becausetheregionofacustomer-managedKMSkeymustalways correlate to the region of the resource it protects in a given Google Cloud service, residency restrictions for single-region keys and resources are guaranteed to be compatible and enforced.

For dual- or multi-regional locations:UserscanselectcorrelatingGoogle-definedmulti-regionsfortheirkeysandGoogleCloudresourcestoguaranteeresidencycompliance.Toensurethisgeographicresidency,makesurethatyourregionsinotherproductslineupwithyourchosenCloud KMS regional location.

ThefollowingtablesummarizesresidencyofkeymaterialforCloudKMS.

Cloud Key Management Service — Deep Dive

23

5.2.3. Regionality monitoringGoogle’sinternaldatamonitoringservicesactivelymonitorkeyresidency.CloudKMSsendsalertstoSREteammembersifitdetectsaccidentalmisconfigurations,orifkeymaterialleavestheboundaries ofitsconfiguredregion,isstoredinthewrongregion,orisaccessedfromthewrongregion.

5.3. Authentication and authorizationCloudKMSacceptsavarietyofauthenticationmechanisms,suchasOAuth2,JWT,andALTS.Whateverthemechanism,CloudKMSresolvestheprovidedcredentialtoidentifytheprincipal(theentitywhichisauthenticatedandisauthorizingtherequest),andcallsIAMtoseewhethertheprincipalisauthorizedtomaketherequestandwhetheranauditlogiswritten.

Cloud KMS uses an internal version of the public Service Control API for many aspects of service management.BeforeaCloudKMSjobhandlesarequest,itfirstsendsacheckrequesttotheServiceControlAPI,whichenforcesmanycontrolscommontoallGoogleCloudservices,suchasthefollowing:

CheckingwhetherthecustomerhasactivatedtheCloudKMSAPIandhasanactivebillingaccount.

Checkingwhetherthecustomerhasnotexceededtheirquota,andreportingquotausage.

Enforcing VPC Service Controls.

Checkingwhetherthecustomerisontheallowlistforapplicableprivatecloudregions.

5.4. LoggingThreetypesoflogsareassociatedwithCloudKMS:CloudAuditLogslogs,AccessTransparencylogs,and internal request logs.

5.4.1. Cloud audit logsCloud KMS records customer activity in Cloud Audit Logslogs.CustomerscanviewtheselogsintheCloudConsole.Alladminactivity—forexample,creatingordestroyingakey—isrecordedintheselogs.Customerscanalsoelecttoenableloggingforallotheractionsthatuseakey—forexample,usingakeytoencryptordecryptdata.Customerscontrolhowlongtheywishtoretainthelogsaswellaswhomayviewthem.

Cloud Key Management Service — Deep Dive

24

5.4.2. Access Transparency logsEligible customers may optionally choose to enable AccessTransparencylogs,whichprovide themwithlogsofactionsthatGoogleemployeestakeinyourGoogleCloudorganization.AccessTransparencylogs,alongsidethelogsfromCloudAuditLogs,canhelpyouanswerquestionsaboutwhodidwhat,where,andwhen.

YoucanintegrateAccessTransparencylogswithyourexistingsecurityinformationandeventmanagement(SIEM)toolstoautomateyourauditsoftheseactions.TheselogsareavailableintheCloud Console alongside your Cloud Audit Logs logs.

AccessTransparencylogentriesincludethefollowingtypesofdetails:

5.4.3. Internal request logsRequestlogsstorearecordforeveryrequestsenttotheCloudKMSplatform.Thisrecordcontainsdetails about the type of request (such as API method or protocol), and the resource being accessed (suchasresourcename,keyalgorithm,orkeyprotectionlevel).Theselogsdon’tstorecustomerplaintext,ciphertext,orkeymaterial.Beforenewtypesofdataareaddedtotheselogs,ateamthatspecializesinprivacyreviewsmustapprovechangestothedatathatislogged.

Logentriesarepermanentlydeletedwhentheconfiguredtimetolive(TTL)hasexpired.

Cloud KMS SREs and engineers in the on-call rotation can access request logs. Human access to logs andanyaccessthatreturnscustomerdatarequiresavalidanddocumentedbusinessjustification. Withsomespecificexceptions, human access is logged and accessible to qualifying customers in AccessTransparencylogs.

Theaffectedresourceandaction.

Thetimeoftheaction.

Thereasonsfortheaction.Examplesofreasonsincludecustomer-initiatedsupport(withacasenumber),Google-initiatedsupport,third-partydatarequests,andGoogle-initiatedreviewrequests.

Dataaboutwhoisactingonthedata(forexample,theGooglestaffmember’slocation).

Cloud Key Management Service — Deep Dive

25

5.5. Datastore backendThedatastoreforCloudKMSishighlyavailable,durable,andisprotected.

Availability. CloudKMSusesGoogle’sinternaldatastore,whichishighlyavailableandsupportsanumberofGoogle’scriticalsystems.

Durability. Cloud KMS uses authenticated encryption to store customer key material in the datastore. Additionally, all metadata is authenticated using a hash-based message authentication code (HMAC) to ensure it has not been altered or corrupted at rest. Every hour, a batch job scans all key material and metadataandverifiesthattheHMACsarevalidandthatthekeymaterialcandecryptsuccessfully.Ifanydataiscorrupted,CloudKMSengineersareimmediatelyalertedsothattheycantakeaction.

CloudKMSusesseveraltypesofbackupsforthedatastore:

Bydefault,thedatastorekeepsachangehistoryofeveryrowforseveralhours.Inanemergency,thislifetimecanbeextendedtoprovidemoretimetoremediateissues.

Everyhour,thedatastorerecordsasnapshot.Thesnapshotcanbevalidatedandusedforrestoration,ifneeded.Thesesnapshotsarekeptforfourdays.

Every day, a full backup is copied to disk and to tape.

TheCloudKMSteamhasdocumentedproceduresforrestoringbackupsinemergencies.

Residency.CloudKMSdatastorebackupsareresidentintheirassociatedGoogleCloudregion.Thesebackupsareallencryptedatrest.Accesstodatainbackupsisgatedbasedonaccessjustifications(suchasaticketnumberyoufiledwithGooglesupport),andhumanaccessisloggedinAccess Transparencylogs.

Protection. At the Cloud KMS application layer, customer key material is encrypted before it is stored. Datastoreengineersdonothaveaccesstoplaintextcustomerkeymaterial.Additionally,thedatastoreencryptsalldataitmanagesbeforewritingtopermanentstorage,soaccesstounderlyingstoragelayers,includingdisksortape,wouldnotallowaccesstoeventheencryptedCloudKMSdatawithoutaccesstothedatastoreencryptionkeys,whicharestoredinGoogle’sinternalKMS.

Cloud Key Management Service — Deep Dive

26

6.Furtherreading

Tolearnmore,explorethefollowingresources:

● Cloud KMS documentation

● Cloud HSM documentation

Whitepapers: ● Google security

● Googleinfrastructuresecuritydesignoverview

● TrustingyourdatawithGoogleCloud

● Data encryption at rest

● Data encryption in transit

● Data deletion on Google Cloud

Otherdocumentation: ● Binary Authorization for Borg (BAB) (code provenance)

● AccessTransparency

● Governmentrequestsforcustomerdata:controlling

access to your data in Google Cloud

● Data residency, operational transparency, and privacy

for European customers on Google Cloud

● Cloud Identity and Access Management

● Cloud Audit Logs

● Cloud Asset Inventory

Cloud Key Management Service — Deep Dive