Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
© 2018 IBM Corporation
Db2 for z/OS: What’s New?
Mark Rader Tadas Varaneckas Dominique Parker
Db2 for z/OS, IBM WSC Db2 for z/OS, IBM Cloud Db2 for z/OS, IBM WSC
Chicago Z Council
December 4, 2018
© 2018 IBM Corporation2 © 2018 IBM Corporation
Db2 for z/OS: What’s New?
Transparent Data Set Encryption
Mark Rader [email protected]
Db2 DevOps Experience for z/OS
Tadas Varaneckas [email protected]
Db2ZAI – Db2 and Machine Learning
Dominique Parker [email protected]
Please note
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice and at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
3
© 2018 IBM Corporation
Db2 for z/OS: Transparent Data Set
Encryption
Mark Rader
Db2 for z/OS, IBM Washington Systems Center
Chicago Z Council
December 4, 2018
IBM z Analytics
5
Today’s security challenges
European Union General Data Protection Regulation
Payment Card Industry Data Security Standard
Health Insurance Portability and Accountability Act
Source: 2017 Ponemon Cost of Data Breach Study: Global Analysis --http://www.ibm.com/security/data-breach/
1,2
$ 3.62MAverage cost of a data breach in 2017 1
27.7%Likelihood of an organization having a data breach over the next two years 2
IBM z Analytics
6
Multiple layers of encryption of data
Coverage
IBM z Analytics
7
Db2 support of z/OS data set encryption and pervasive encryption
• Db2 11 and Db2 12 support transparent data set encryption (TDE)
• Without database downtime or requiring the administrator to redefine objects which could cause disruption to operations
• No application changes required
• Encrypt active and archive log datasets
• Encrypt catalog and directory table spaces
• Encrypt user table spaces
• Utilizes new z/OS DFSMS data set encryption support delivered in z/OS 2.3 and z/OS 2.2 (with OA50569 and OA53951)
• Extended format data sets only
• Db2 12 V12R1M502 adds additional controls to set up encryption policies using Db2 interfaces
IBM z Analytics
8
Understanding DFSMS policy-based dataset encryption (reference)
Data sets are defined as encrypted by specifying a key label during the creation of a new data set:
• RACF data set profile
• JCL, Dynamic Allocation, TSO Allocate, IDCAMS DEFINE
• SMS DATACLAS
During data set open, DFSMS:
• Checks the user access to the key label
• Specifies the key label to ICSF to retrieve the secure / protected key from the CKDS
ICSF:
• Locates the secure key in the CKDS using the key label specified by DFSMS
• Calls the adapter to unwrap the key value from the Master key
• Rewraps the key value under a CPACF wrapping key to make it a protected key
• Protected key stored in ICSF cache
IBM z Analytics
9
DFSMS dataset encryption
•Application transparency
• Data remains encrypted during backup/recovery, migration/recall
• In memory system or application data buffers remain in the clear
• Access to the key label is controlled through SAF permissions, in addition to traditional data set permissions
•Segregation of duties
• Storage administrators need access to the data set but not access to the key label
IBM z Analytics
10
Steps to enable encryption
Generates an encryption key and key label.Stores it in the ICSF CKDS
ICSF AdminSets up RACF for use of key label
Allows secure key to be used as protected key via ICSF segment- SYMCPACFWRAP- SYMCPACFRET
Permits Db2 use of the key label
– AND –
In RACF, permit access to new resource in FACILITY class
Security Admin
Online REORG
Migrate to encrypted data
Non-disruptive
Database Admin
Associate the key label with the desired data set.
In RACF, alter DFP segment in data set profile - DATAKEY()
IDCAMS DEFINE, TSO Allocate, etc. (V12 only) In Db2, set key label for system objects and user objects
– OR –
In DFSMS, assign to data class
– OR –
Security / Database / Storage Admin
1 2 3 4
IBM z Analytics
11
Encrypting Db2 system objects (see prior page)
•Options to define a key label used by Db2 (precedence order):
• 1. Security Admin can set a key label in the DFP segment of RACF data set profile using the new DATAKEY keyword
• 2. Database System Admin can set a key label using ENCRYPTION_KEYLABEL system parameter (V12R1M502 only)
• -SET SYSPARM command is required for the zparm value to take effect
• Group scope: Takes effect on all the members of a data sharing group immediately
• Security related parameter: Requires installation SYSADM or SECADM authority to set the zparm
• Db2 DBM1 and MSTR address space IDs must be permitted access to the key label
OR
• 2. Storage Admin can set a key label using IDCAMS DEFINE
• Only option to encrypt active logs; ENCRYPTION_KEYLABEL will not apply
• 3. Storage Admin can set a key label in the DFSMS data class
3
In RACF, alter DFP segment in data set profile - DATAKEY()
– OR –
In DFSMS, assign to data class
In Db2, set key label using system parameter
ORIDCAMS DEFINE, etc.
– OR –
Security / Database System Admin / Storage Admin
IBM z Analytics
12
Encrypting Db2 system objects
•Active logs
• Encrypt new active logs
• Define active log data set as encrypted and issue the SET LOG command NEWLOG option to add the newly defined active log data set to the active log inventory without stopping Db2
• Encrypt all active logs
• Stop Db2. Copy the contents of the active log data set to an encrypted data set. Restart Db2.
•Archive logs
• New archive logs automatically encrypted based on the key label setting
• Must be defined on disk
•Catalog and directory table spaces
• Execute REORG TABLESPACE utility to encrypt table spaces and index spaces in DSNDB06 and DSNDB01
• Encrypt DSNDB01.SYSUTILX – Execute RECOVER utility followed by REBUILD INDEX ALL.
Online REORG
Database Admin
4
Can use ENCRYPTION_KEYLABELsystem parameter
IBM z Analytics
13
Encrypting user objects
•Options to define a key label for user objects encryption (precedence order):
• 1. Security Admin can set a key label in the RACF data set profile DFP segment using the new DATAKEY keyword
• 2. Storage Admin (or Database Admin) can set a key label via IDCAMS, TSO, etc…
OR
• 2. Application Database Admin can set a key label using SQL interfaces:CREATE / ALTER with STOGROUP / TABLE (V12R1M502 only)
• Enabled with APPLCOMPAT V12R1M502
• 3. Storage Admin can set a key label in the DFSMS data class
In RACF, alter DFP segment in data set profile - DATAKEY()
– OR –
In DFSMS, assign to data class
Security / Database / Storage Admin
– OR –
IDCAMS DEFINE, etc.OR
In Db2, set key label using SQL interfaces
3
IBM z Analytics
14
Utilities considerations•Db2 managed table spaces and index spaces
• Utilities used to convert to encrypted data sets (except when REUSE option is specified)
• REORG TABLESPACE or REORG INDEX
• LOAD REPLACE
• REBUILD INDEX
• RECOVER from image copies – PIT or full recovery
• PART or DSNUM option to encrypt / decrypt at the partition level
•User managed table spaces and index spaces
• IDCAMS DELETE / DEFINE with the KEYLABEL option
• Execute RECOVER and/or REBUILD INDEX utilities to restore the data
•FlashCopy image copies (FCIC), DFSMSdss concurrent image copies, shadow data sets
• Allocated with the same key label as the table space or index
IBM z Analytics
15
Db2 data set encryption considerations
Compression
• Db2 compression works seamlessly with data set encryption
• Compression is performed first
Performance
• There will be some CPU cost for encryption
• Internal benchmarks show significant savings on z14 vs. z13
UI55093
IBM z Analytics
16
Data set encryption summary
•For Db2 11 and Db2 12, a key label can be defined by the security administrator or storage administrator, a data base administrator can use Db2 REORG utility to seamlessly migrate Db2 data sets to encrypted data sets with no application outages
•For Db2 12, function level V12R1M502 provides new Db2 interfaces to configure and manage Db2 key labels
•Make sure all disaster recover user ids and sites have access to any key labels used to protect Db2 data sets and the key management system is fully deployed across the enterprise
•Recommendation: plan and implement enterprise security and encryption strategy
Thank you
Further reference:
Db2 V11 APAR PI81900 (UI51358)Db2 V12 base APAR PI81907 (UI51499)
https://www.ibm.com/support/knowledgecenter/SSEPEK_11.0.0/seca/src/tpc/db2z_dfsmsencryptionsupport.html
Db2 V12 V12R1M502 APAR PI95511 (UI55093)
https://www.ibm.com/support/knowledgecenter/en/SSEPEK_12.0.0/wnew/src/tpc/db2z_fl_v12r1m502.html#db2z_fl_v12r1m502__48635
Systems Technical University 2018 / © 2018 IBM Corporation
17
© 2018 IBM Corporation
Db2 DevOps Experience for z/OS
Tadas Varaneckas
Db2 for z/OS, IBM Cloud
Chicago Z Council
December 4, 2018
© 2018 IBM Corporation
• Enterprises need to innovate rapidly to capitalize on new market opportunities, and reducethe cycle time to collect and react to customer feedback.
• DevOps methods provide the ability for continuous software delivery and management.
What is DevOps?
for faster time to value
for increased capacity to innovate
for improved customer experience
Process
Culture
Technology
Business and Practitioner Adoption of Lean and Agile principles
Business are Seeking to: Driving Changes to:
19
© 2018 IBM Corporation
Why DevOps matters
Source: Forrester20
© 2018 IBM Corporation
• Lines of Business (LOB) owners can respond quickly to customer needs and competition with faster delivery of new or changed Db2 applications
• IT can establish and enforce the criteria for App Dev to create their own Db2 for z/OS test environments, and retains control of rules and standards that protect Db2 databases and data
• App Dev avoids the wait time for IT to create Db2 for z/OS test environments, and can work in teams using more modern agile development methods
IBM Db2 DevOps Experience for z/OSBrings Db2 applications to market faster with lower costs and less risk
21
© 2018 IBM Corporation
Where does the Db2 DevOps Experience for z/OS fit?
Db2 for z/OS
• Allows Db2 for z/OS to participate in customer DevOps pipelines
• Db2 DDL as code• Application and Infrastructure have
been managed as code• Now Db2 fits this model
• On demand, self service provisioning and deployment of Db2 objects and data
• Enable the upstream and downstream portions of DevOps
• With controls to enable object handling, limits, and approvals for deployment
• UI and REST API support• Fits into customer’s existing DevOps
tooling
22
© 2018 IBM Corporation
• Need DBA to create Db2 environment• Creates / changes applications• Need DBA to reset test environment• Application deadlines missed
• Create mainframe test environment for application development
• Cannot service # of app dev requests• Critical DBA tasks not addressed• Application deadlines missed
• Business rules• Enterprise data
App Dev
DBA
Before IBM Db2 DevOps Experience for z/OS
DB2
DBA controls DB2 rules and data
23
© 2018 IBM Corporation
IBM Db2 DevOps Experience for z/OS – Features
• Ability to create and delete Db2 for z/OS sandbox environment for testing without IT time and assistance
• Make Db2 syntax changes using object-aware Data Definition Language (DDL) editor
• Integrate Db2 database object management into existing continuous delivery development processes
App Dev/Engineer DBA
• Define teams of users, environments, and application components for developers to use in their application development
• Discover and select application’s Db2 database components to be managed and processed together
• Set rules and storage limits for the defined teams
• Review and approve developer schema changes
zowe.org© 2018 IBM Corporation24
© 2018 IBM Corporation
IBM Db2 DevOps Experience for z/OS
Ready
© 2018 IBM Corporation25
© 2018 IBM Corporation
• Create own Db2 test environment• Resets test environment when needed• Delete test environment when complete• Application delivered on time!
• Set up policies for APP DEV to self-create Db2 test environment
• Policy limits frees DBA from app service requests• Can focus on critical Db2 tasks• Retain control over Db2 databases• Application delivered on time!
• Business rules• Enterprise data
App Dev
DBA
After IBM Db2 DevOps Experience for z/OS
DB2
DBA controls DB2 rules and data
26
© 2018 IBM Corporation
IBM Db2 DevOps Experience for z/OS
• A platform for the definition administration and operation of DevOps services in support of Db2 for z/OS.
• Establish the Users, Teams, and Subsystems that will participate in DevOps
• Define Site Rules to be used in the naming and management of objects under DevOps control
• Register the Subsystem & establish Environments which will be the targets for provisioning operations
• Define Applications as set of Db2 for z/OS objects- Establishes a master branch in Git for those objects
Users
Teams
Subsystems
Environments
Applications
Site Rules / Limits
Instances
27
© 2018 IBM Corporation
IBM Db2 DevOps Experience for z/OS
• Provision instances of the Application into defined Environments
- Will create an Instance of the Application tables- Optionally copy data into the provisioned Instance
objects• DDL for the Provisioned instance can be modified
and deployed to the Instance for testing• Deprovision object to start over (fail fast)• Commit changes to the master branch when
preparing to deploy to other landscapes- The UI/API will issue a Git pull request where
Approvals can be added to the process before being accepted
Users
Teams
Subsystems
Environments
Applications
Site Rules / Limits
Instances
28
© 2018 IBM Corporation
Additional References
Db2 Tools for z/OS News – Announcement Replayhttps://ibm.co/2R31zwD
IBM Db2 DevOps Experience for z/OS White Paperhttps://bit.ly/2znfTcx
IBM Marketplacehttps://www.ibm.com/us-en/marketplace/db2-devops-experience
29
© 2018 IBM Corporation
Machine Learning and Db2 for z/OS:
the Db2 Optimizer
Dominique Parker
Db2 for z/OS, IBM Washington Systems Center
Chicago Z Council
December 4, 2018
© 2018 IBM Corporation31 © 2018 IBM Corporation
Agenda
The query optimization challenge
A brief overview of machine learning
Db2 AI for z/OS: exploiting machine learning for query optimization
© 2018 IBM Corporation32 © 2018 IBM Corporation
The query optimization challenge
© 2016 IBM Corporation33 © 2018 IBM Corporation
One aspect of the optimization challenge
Determining the degree of row-filtering for each predicate (i.e., the filter factor) is a
key input to the optimizer’s access path cost formulas
Question: how many of you were born before… parameter marker?
It depends…
• If the substitution value is ‘1900-01-01’, the answer is: nobody
• If the substitution value is ‘2018-10-01’, the answer is: everybody
When a query has a range predicate with a parameter marker or host variable…
…the Db2 optimizer uses a default filter factor (these are documented)
BIRTHDATE < ?
© 2016 IBM Corporation34 © 2018 IBM Corporation
The trouble with default filter factors
They may not ALWAYS predict the way an organization accesses its Db2 data
Analogy: transportation recommendation for a local trip
To get from point A to point B, should you take a personal vehicle, bus, commuter rail?
Answer depends not only on factors such as distance to be traveled and time of day, but also on
the particular area in which the trip will be taken
• Best answer for New York may not be best for Dallas, best for Los Angeles may not be best for Charlotte
• It would be nice if we could provide a location-specific answer…
© 2018 IBM Corporation35 © 2018 IBM Corporation
Tuning just ONE query: the user challenge
Access path tuning can be a very complex, time-consuming task
Depends on several factors (user skill level, available tooling, etc.)
Solutions that we like to see:
Determine right RUNSTATS options for targeted database objects
• Sources: SYSIBM.SYSSTATFEEDBACK (system-level) or DSN_STAT_FEEDBACK (query-level)
Indexing – add or modify
Add OPTIMIZE FOR n ROWS to query (“paging-style” SQL, only part of result set fetched)
Solutions that are less preferred:
BIND/REBIND with APREUSE or OPTHINTS
Query rewrites
Manipulating catalog statistics
Query tricks (OR 0=1 for example)
Skill often missing
Organizations often revert to these
© 2018 IBM Corporation36 © 2018 IBM Corporation
REBIND – user challenge
Suppose you REBIND 1000 static packages using APCOMPARE(WARN)
After migration or maintenance apply
If 90% show no change, and 10% (100 packages) have differences, how do you
validate that changes are acceptable?
Review changes identified in PLAN_TABLE.REMARKS due to APCOMPARE(WARN)
• Do access paths look better (e.g., increase in MATCHCOLS)? Any red flags?
If access path change looks OK, go with new instance of package
What if access path change does NOT look OK?
• APREUSE, based on prior instance of package?
• Create OPTHINT?
• Try to tune affected queries?
What if access path change validation work has to be done for several hundred
packages (or more)?
Approaches most commonly taken
© 2018 IBM Corporation37 © 2018 IBM Corporation
Users want performance, but also stability
Db2 for z/OS optimizer
Is a cost-based model
Has evolved and improved over 35+ years.
• User feedback
• Internal analysis by Db2 development
Provides stability and reliability
• RUNSTATS utility provides inputs to the optimizer cost model
• Static SQL, Dynamic Statement Cache, and (with Db2 12) dynamic plan stability persist access path
choices
• Plan management (for static SQL) enables quick and easy switch to earlier instance of a package
in the event of an access path change that negatively impacts performance
© 2018 IBM Corporation38 © 2018 IBM Corporation
Optimizer development goal
Improve performance and/or
stability of access path
choices…
…without compromising the
quality and stability of existing
access path choices
© 2018 IBM Corporation39 © 2018 IBM Corporation
A brief overview of machine learning
© 2018 IBM Corporation40 © 2018 IBM Corporation
What is machine learning?
A twist on traditional data processing The basic areas of activity related to
exploitation of machine learning technology
Identify
patterns in
datanot readily
discerned by
humans
Build
predictive
modelsof behavior
from those
patterns
Obtain “scores” from
deployed modelsthat predict behavior
Computer
data
programoutput
Traditional data processing
Computer
data
outputprogram*
Machine learning
* How can you get from these data input values to those data output values?
© 2018 IBM Corporation41 © 2018 IBM Corporation
How does Db2 optimizer exploit machine learning?
Machine learning…• Constantly learns and adapts
• Avoids making the same mistakes
• Provides faster, deeper, improved insights
Resulting in… More-favorable business outcomes
Reduced risks and costs
New opportunities
© 2018 IBM Corporation42 © 2018 IBM Corporation
A TrainOps (DevOps) story
Model trainingFeature
engineering
Feature
engineering Scoring
Labeled examples
Training
Scoring
Newdata
Model
ModelPredicted
data
Deploy
Data Scientist/Data analyst
Operational system
Dev
Ops
Training, deployment, scoring
© 2018 IBM Corporation43 © 2018 IBM Corporation
Db2 AI for z/OS: exploiting machine learning
for query optimization
© 2018 IBM Corporation44 © 2018 IBM Corporation
Long-term goal for machine learning within Db2 for z/OS
Make Db2 self-tuning and self-managing
Query optimization
Rebinds
Memory and storage allocations (buffer pools, work files, RID pool, sort pool, etc.)
Utility execution (REORG, RUNSTATS,…)
Table space and index PCTFREE/FREEPAGE specifications based on insert patterns
Fast traverse block (FTB) usage
Analytics Accelerator query offload decisioning
Application of maintenance (how, when, with what expected benefits?)
…
© 2018 IBM Corporation45 © 2018 IBM Corporation
Initial use case – access path selection
Construct machine learning models to assist Db2 optimizer in choosing best
access path for a query
Models are based on actual costs observed for queries that are executed on
target system
What that means: the optimizer is customized for the needs of a particular environment and
workload
Machine learning models stay current and adapt to changing conditions
© 2018 IBM Corporation46 © 2018 IBM Corporation
Machine learning and access path selection – phase one
Initial focus on optimizer cost model – could it be improved via machine learning?
Analysis using supervised learning techniques (neural networks, random forest, linear regression)
Analysis was used to validate the existing optimizer cost model
Multiple costing deficiencies found in optimizer base code – will be addressed through future APARs
After that work, focus shifted to using machine learning to improve the quality and
effectiveness of inputs to the optimizer cost model
Machine learning does not replace optimizer (any more than IBM Watson replaces an oncologist)
Watson provides an oncologist with better inputs to patient care decisions – we use machine
learning to provide the Db2 optimizer with better inputs to query access path decisions
© 2018 IBM Corporation47 © 2018 IBM Corporation
Phase two: focusing on what the optimizer doesn’t know
What inputs are currently “estimated” or unknown?
Two of the most important of those factors are:
Execution time values for host variables / parameter markers
Number of rows that will actually be fetched by the application
• Sometimes OPTIMIZE FOR n ROWS has been added to a query – often not…
Among others…
© 2018 IBM Corporation48 © 2018 IBM Corporation
Db2 AI for z/OS (Db2ZAI) enables the Db2 optimizer to determine the best-performing
query access paths based on your workload characteristics, using machine learning
New offering: IBM Db2 AI for z/OS
• Learns patterns from workload data collected in an
organization’s unique operating environment and uses
derived insight in determining optimal access paths for SQL
statements
• Built on top of the IBM Machine Learning for z/OS stack (MLz)
o Leverages all MLz services without requiring data scientist
support
o Db2 generates model training data, deploys and monitors
and retrains models via MLz services
Db2
© 2018 IBM Corporation49 © 2018 IBM Corporation
Db2ZAI – prerequisites
Db2 12 for z/OS
z/OS Db2 JDBC driver (FMID JDBCC12)
Function level V12R1M503 must be activated (APAR PH00506)
Db2ZAI product ID: 5698-CGN
IBM Machine Learning for z/OS, V1.2 (5698-ML1)
Fix pack 1.2.0.1
IBM Open Data Analytics for z/OS (IzODA)
© 2018 IBM Corporation50 © 2018 IBM Corporation
Db2ZAI – architecture
Note: data collected to build predictive models is stored in user-managed tables, but Db2 takes care of “cleaning up” the tables as needed
SQLApplications
“Learning”Task(zIIP Offloaded*)
ModelLifeCycleAutomationSubtasks
PerformanceHistory
Db212forz/OS
- Db2ZAI- MachineLearningforz/OS- z/OS- x86*zIIP Offload
MachineLearningforz/OSV1.2
MLz ModelServicesDb2ZAIUserInterface
MLScoringService*
Db2Optimizer
IBMOpenDataAnalytics*(IZODA)
-x86 or Linux on z * zIIP offload
© 2018 IBM Corporation51 © 2018 IBM Corporation
What’s next?
We’re evaluating the next targeted use cases
Including soliciting customer input
Related, parallel efforts: improving the REBIND experience and boosting
regression avoidance
There is NO commitment on timeline or deliverables – “this is a journey”
Important to know:
The Db2 for z/OS development organization is committed to the safe and cost-effective
exploitation of machine learning technology, for the benefit of Db2-using organizations
© 2018 IBM Corporation52 © 2018 IBM Corporation
Recap
The optimizer’s goal for each new release/enhancement of Db2 for z/OS:
Provide new access path choices that improve performance and/or stability, without
compromising the quality and stability of existing choices
It is understood that improving the BIND/REBIND experience is CRITICAL to
ensuring that optimizer exploitation of machine learning technology does not
degrade performance and/or stability while we’re “learning”
© 2018 IBM Corporation53 © 2018 IBM Corporation
Where to find more information on Db2ZAI
The announcement letter:
https://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/1/897/ENUS218-
341/index.html&request_locale=en
The IBM Knowledge Center:
https://www.ibm.com/support/knowledgecenter/SSGKMA_1.1.0/src/ai/ai_home.html
Overview video:
https://www.youtube.com/watch?v=t5fTNxfehQA
© 2018 IBM Corporation54 © 2018 IBM Corporation
Thank you!
Question?
Contact us:
• Mark [email protected]
• Tadas [email protected]
• Dominique [email protected]