46
Page 1 Knowledge Acquisition, Representation, and Reasoning Prof. Rushen Chahal Prof. Rushen Chahal

Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

Embed Size (px)

Citation preview

Page 1: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 1/46

Page 1

Knowledge Acquisition,Representation, and Reasoning

Prof. Rushen Chahal

Prof. Rushen Chahal

Page 2: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 2/46

Page 2

L earning Objectives

Understand the nature of knowledge.L earn the knowledge engineering processes.Evaluate different approaches for knowledge acquisition.

Examine the pros and cons of different approaches.Illustrate methods for knowledge verification andvalidation.Examine inference strategies.Understand certainty and uncertainty processing.

Prof. Rushen Chahal 11-2

Page 3: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 3/46

Page 3

D evelopment of a Real-Time Knowledge-BasedSystem at Eli L illy Vignette

Problems with fermentation process ± Quality parameters difficult to control ± Many different employees doing same task ± High turnover

Expert system used to capture knowledge ± Expertise available 24 hours a day

Knowledge engineers developed system by: ± Knowledge elicitation

Interviewing experts and creating knowledge bases ± Knowledge fusion

Fusing individual knowledge bases ± Coding knowledge base ± Testing and evaluation of system

Prof. Rushen Chahal 11-3

Page 4: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 4/46

Page 4

Knowledge Engineering

Process of acquiring knowledge fromexperts and building knowledge base

± Narrow perspectiveKnowledge acquisition, representation, validation,inference, maintenance

± Broad perspective

Process of developing and maintaining intelligentsystem

Prof. Rushen Chahal 11-4

Page 5: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 5/46

Page 5

Knowledge Engineering

ProcessAcquisition of knowledge ± General knowledge or metaknowledge ± From experts, books, documents, sensors, files

Knowledge representation ± Organized knowledge

Knowledge validation and verificationInferences ± Software designed to pass statistical sample data to

generalizations

Explanation and justification capabilities

Prof. Rushen Chahal 11-5

Page 6: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 6/46

Page 6Prof. Rushen Chahal 11-6

Page 7: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 7/46

Page 7

Knowledge

Sources ± D ocumented

Written, viewed, sensory, behavior

± UndocumentedMemory

± Acquired fromHuman senses

Machines

Prof. Rushen Chahal 11-7

Page 8: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 8/46

Page 8

Knowledge

L evels ± Shallow

Surface levelInput-output

± D eepProblem solving

D ifficult to collect, validateInteractions betwixt system components

Prof. Rushen Chahal 11-8

Page 9: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 9/46

Page 9

Knowledge

Categories ± D eclarative

D escriptive representation

± ProceduralHow things work under different circumstancesHow to use declarative knowledge ± Problem solving

± MetaknowledgeKnowledge about knowledge

Prof. Rushen Chahal 11-9

Page 10: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 10/46

Page 10

Knowledge Engineers

Professionals who elicit knowledge from experts ± Empathetic, patient ± Broad range of understanding, capabilities

Integrate knowledge from various sources ± Creates and edits code ± Operates toolsBuild knowledge base ± Validates information ± Trains users

Prof. Rushen Chahal 11-10

Page 11: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 11/46

Page 11Prof. Rushen Chahal 11-11

Page 12: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 12/46

Page 12

Elicitation Methods

Manual ± Based on interview ± Track reasoning process ± ObservationSemiautomatic ± Build base with minimal help from knowledge

engineer ± Allows execution of routine tasks with minimal expert

inputAutomatic ± Minimal input from both expert and knowledge

engineer

Prof. Rushen Chahal 11-12

Page 13: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 13/46

Page 13

Manual Methods

Interviews ± Structured

Goal-orientedWalk through

± UnstructuredComplex domains

D ata unrelated and difficult to integrate ± Semistructured

Prof. Rushen Chahal 11-13

Page 14: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 14/46

Page 14

Manual Methods

Process tracking ± Track reasoning processes

Protocol analysis ± D ocument expert¶s decision-making ± Think aloud process

Observation ± Motor movements ± Eye movements

Prof. Rushen Chahal 11-14

Page 15: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 15/46

Page 15

Manual Methods

Case analysisCritical incidentUser discussions

Expert commentaryGraphs and conceptual modelsBrainstormingPrototypingMultidimensional scaling for distance matrixClustering of elementsIterative performance review

Prof. Rushen Chahal 11-15

Page 16: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 16/46

Page 16

Semiautomatic Methods

Repertory grid analysis ± Personal construct theory

Organized, perceptual model of expert¶s knowledgeExpert identifies domain objects and their attributes

Expert determines characteristics and opposites for eachattributeExpert distinguishes between objects, creating a grid

Expert transfer system ± Computer program that elicits information from

experts ± Rapid prototyping ± Used to determine sufficiency of available knowledge

Prof. Rushen Chahal 11-16

Page 17: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 17/46

Page 17Prof. Rushen Chahal 11-17

Page 18: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 18/46

Page 18

Semiautomatic Methods, continued

Computer based tools features: ± Ability to add knowledge to base

± Ability to assess, refine knowledge ± Visual modeling for construction of domain ± Creation of decision trees and rules ± Ability to analyze information flows ± Integration tools

Prof. Rushen Chahal 11-18

Page 19: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 19/46

Page 19

Automatic Methods

D ata mining by computersInductive learning from existing recognized

casesNeural computing mimicking human brainGenetic algorithms using natural selection

Prof. Rushen Chahal 11-19

Page 20: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 20/46

Page 20

Multiple Experts

Scenarios ± Experts contribute individually ± Primary expert¶s information reviewed by secondary

experts ± Small group decision ± Panels for verification and validation

Approaches

± Consensus methods ± Analytic approaches ± Automation of process through software usage ± D ecomposition

Prof. Rushen Chahal 11-20

Page 21: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 21/46

Page 21

Automated Knowledge Acquisition

Induction ± Activities

Training set with known outcomesCreates rules for examplesAssesses new cases

± Advantages

L imited applicationBuilder can be expert ± Saves time, money

Prof. Rushen Chahal 11-21

Page 22: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 22/46

Page 22

Automated Knowledge

Acquisition ± D ifficultiesRules may be difficult to understandExperts needed to select attributes

Algorithm-based search process produces fewer questionsRule-based classification problemsAllows few attributesMany examples neededExamples must be cleansed

L imited to certaintiesExamples may be insufficient

Prof. Rushen Chahal 11-22

Page 23: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 23/46

Page 23

Automated Knowledge

AcquisitionInteractive induction ± Incrementally induced knowledge

General models ± Object Network

± Based on interaction with expertinterviews

± Computer supportedInduction tablesIF-THEN-E L SE rules

Prof. Rushen Chahal 11-23

Page 24: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 24/46

Page 24

Evaluation, Validation,

VerificationD ynamic activities ± Evaluation

Assess system¶s overall value

± ValidationCompares system¶s performance to expert¶sConcordance and differences

± VerificationBuilding and implementing system correctlyCan be automated

Prof. Rushen Chahal 11-24

Page 25: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 25/46

Page 25Prof. Rushen Chahal 11-25

Page 26: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 26/46

Page 26

Production Rules

IF-THENIndependent part, combined with other pieces, to produce better resultModel of human behavior Examples ± IF condition, THEN conclusion

± Conclusion, IF condition ± If condition, THEN conclusion1 (OR) E L SE

conclusion2

Prof. Rushen Chahal 11-26

Page 27: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 27/46

Page 27

Artificial Intelligence Rules

Types ± Knowledge rules

D eclares facts and relationshipsStored in knowledge base

± InferenceGiven facts, advises how to proceed

Part of inference enginesMetarules

Prof. Rushen Chahal 11-27

Page 28: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 28/46

Page 28

Artificial Intelligence Rules

Advantages ± Easy to understand, modify, maintain ± Explanations are easy to get. ± Rules are independent.

± Modification and maintenance are relatively easy. ± Uncertainty is easily combined with rules.L imitations ± Huge numbers may be required ± D esigners may force knowledge into rule-based entities ± Systems may have search limitations; difficulties in evaluation

Prof. Rushen Chahal 11-28

Page 29: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 29/46

Page 29

Semantic Networks

GraphicaldepictionsNodes and linksHierarchicalrelationshipsbetweenconceptsReflectsinheritance

Prof. Rushen Chahal 11-29

Page 30: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 30/46

Page 30

Frames

All knowledge about objectHierarchical structure allows for inheritanceAllows for diagnosis of knowledge independence

Object-oriented programming ± Knowledge organized by characteristics andattributes

SlotsSubslots/facets

± Parents are general attributes ± Instantiated to childrenOften combined with production rules

Prof. Rushen Chahal 11-30

Page 31: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 31/46

Page 31

Knowledge Relationship

RepresentationsD ecision tables ± Spreadsheet format ± All possible attributes compared to conclusionsD ecision trees ± Nodes and links ± Knowledge diagrammingComputational logic ± Propositional

True/false statement ± Predicate logic

Variable functions applied to components of statements

Prof. Rushen Chahal 11-31

Page 32: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 32/46

Page 32Prof. Rushen Chahal 11-32

Page 33: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 33/46

Page 33

Reasoning Programs

Inference Engine ± Algorithms ± D irects search of knowledge base

Forward chaining ± D ata driven ± Start with information, draw conclusions

Backward chaining ± Goal driven ± Start with expectations, seek supporting evidence

± Inference/goal treeSchematic view of inference process

± AND /OR/NOT nodes ± Answers why and how

Rule interpreter

Prof. Rushen Chahal 11-33

Page 34: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 34/46

Page 34

Explanation Facility

Justifier ± Makes system more understandable ± Exposes shortcomings ± Explains situations that the user did not anticipate

± Satisfies user¶s psychological and social needs ± Clarifies underlying assumptions ± Conducts sensitivity analysis

Types ± Why ± How ± Journalism based

Who, what, where, when, why, howWhy not

Prof. Rushen Chahal 11-34

Page 35: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 35/46

Page 35

Generating Explanations

Static explanation ± Preinsertion of textD

ynamic explanation ± Reconstruction by rule evaluation

Tracing records or line of reasoning

Justification based on empiricalassociationsStrategic use of metaknowledge

Prof. Rushen Chahal 11-35

Page 36: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 36/46

Page 36

Uncertainty

WidespreadImportant componentRepresentation ± Numeric scale

1 to 100

± Graphical presentation

Bars, pie charts ± Symbolic scales

Very likely to very unlikely

Prof. Rushen Chahal 11-36

Page 37: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 37/46

Page 37

Uncertainty

Probability Ratio ± D egree of confidence in conclusion ± Chance of occurrence of eventBayes Theory ± Subjective probability for propositions

ImpreciseCombines values

D empster-Shafer

± Belief functions ± Creates boundaries for assignments of probabilitiesAssumes statistical independence

Prof. Rushen Chahal 11-37

Page 38: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 38/46

Page 38

Certainty

Certainty factors ± Belief in event based on evidence ± Belief and disbelief independent and not

combinable ± Certainty factors may be combined into one

rule

± Rules may be combined

Prof. Rushen Chahal 11-38

Page 39: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 39/46

Page 39

Expert System D evelopment

Phases ± Project initialization ± Systems analysis and design ± Prototyping ± System development ± Implementation ± Postimplementation

Prof. Rushen Chahal 11-39

Page 40: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 40/46

Page 40

Project Initialization

Identify problemsD etermine functional requirementsEvaluate solutions

Verify and justify requirementsConduct feasibility study and cost-benefitanalysisD etermine management issuesSelect teamProject approval

Prof. Rushen Chahal 11-40

Page 41: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 41/46

Page 41

Systems Analysis and D esign

Create conceptual system designD etermine development strategy ± In house, outsource, mixedD

etermine knowledge sourcesObtain cooperation of expertsSelect development environment ± Expert system shells ± Programming languages ± Hybrids with tools

General or domain specific shellsD omain specific tools

Prof. Rushen Chahal 11-41

Page 42: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 42/46

Page 42

Prototyping

Rapid productionD emonstration prototype

± Small system or part of system ± Iterative ± Each iteration tested by users ± Additional rules applied to later iterations

Prof. Rushen Chahal 11-42

Page 43: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 43/46

Page 43

System D evelopment

D evelopment strategies formalizedKnowledge base developed

Interfaces createdSystem evaluated and improved

Prof. Rushen Chahal 11-43

Page 44: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 44/46

Page 44

Implementation

Adoption strategies formulatedSystem installed

All parts of system must be fullydocumented and security mechanismsemployedField testing if it stands alone; otherwise,must be integratedUser approval

Prof. Rushen Chahal 11-44

Page 45: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 45/46

Page 45

Postimplementation

Operation of systemMaintenance plans

± Review, revision of rules ± D ata integrity checks ± L inking to databases

Upgrading and expansionPeriodic evaluation and testing

Prof. Rushen Chahal 11-45

Page 46: Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

8/3/2019 Management Support Systems - Knowledge Acquisition, Representation, And Reasoning

http://slidepdf.com/reader/full/management-support-systems-knowledge-acquisition-representation-and-reasoning 46/46

Page 46

Internet

Facilitates knowledge acquisition anddistribution

Problems with use of informal knowledgeOpen knowledge source

Prof Rushen Chahal 11-46