Complex problem solving (CPS) definition

Latent Problem Solving Analysis (LPSA): A

computational theory of representation in complex, dynamic problem solving

tasks

2

Complex problem solving (CPS) definition

• dynamic, because early actions determine the environment in which subsequent decision must be made, and features of the task environment may change independently of the solver’s actions;

• time-dependent, because decisions must be made at the correct moment in relation to environmental demands; and

• complex, in the sense that most variables are not related to each other in one-to-one manner

3

‘Despite 10 years of research in the area, there is neither a clearly formulated specific theory nor is there an agreement on how to proceed with respect to the research philosophy. Even

worse, no stable phenomena have been observed’

(Funke, 1992, p. 25)

4

"How similar are two participant's solutions?"

• For CPS there is no common, explicit theory to explain why a complex, dynamic situation is similar to any other situation or how two slices of performance taken from a

problem solving task can possibly be compared quantitatively.

• This lack of formalized, analytical models is slowing down the development of theory in the field.

5

Example of a complex, dynamic task: Firechief (Omodei and Wearing 1995)

6

No. Command/ Gen Perf App. App. Position Destination/ Landscape/ Event Code Type Upper Left Lower Right Wind Change 0 100.00 Wind Strength = 6 Wind Direction = East Mature Fire 0 100.00 (10, 10) Mature Fire 0 100.00 (6, 9) Mature Fire 0 100.00 (6, 8) Mature Fire 0 100.00 (9, 10) 1 Move 17 100.00 4 Copter (11, 4) (11, 9) Forest 2 Move 31 100.00 2 Truck (4, 11) (17, 7) Clearing 3 Drop Water 38 100.00 4 Copter (11, 9) Forest 4 Move 54 100.00 3 Copter (8, 6) (10, 11) Forest 5 Move 70 99.77 1 Truck (4, 14) (18, 10) Forest 6 Drop Water 77 99.42 3 Copter (10, 11) Forest 7 Move 99 99.18 4 Copter (11, 9) (21, 8) Dam 8 Move 113 99.18 3 Copter (10, 11) (12, 14) Dam 9 Control Fire 122 98.95 2 Truck (17, 7) Clearing 10 Control Fire 131 98.95 1 Truck (18, 10) Forest 11 Move 152 98.95 4 Copter (21, 8) (12, 10) Clearing 12 Drop Water 177 98.71 4 Copter (12, 10) Clearing 13 Move 187 98.71 3 Copter (12, 14) (11, 11) Clearing 14 Move 222 98.48 4 Copter (12, 10) (21, 8) Dam 15 Move 236 98.48 3 Copter (11, 11) (12, 14) Dam 16 Move 267 98.25 3 Copter (12, 14) (10, 12) Forest 17 Drop Water 273 98.01 3 Copter (10, 12) Forest 18 Move 296 98.01 2 Truck (17, 7) (8, 5) Forest 19 Move 319 96.85 4 Copter (21, 8) (7, 6) Forest 20 Move 341 96.61 3 Copter (10, 12) (12, 7) Forest 21 Drop Water 347 96.50 4 Copter (7, 6) Forest 22 Drop Water 352 96.50 2 Truck (8, 5) Forest 23 Drop Water 361 96.26 3 Copter (12, 7) Forest

7

Problems with the classic 'problem space’ approach!

Most of the theories about cognitive skill acquisition and procedural learning are based in two principles:

– The problem space hypothesis– Representation of procedures as productions

8


1. The problem with the ‘generation’ of the problem space

2. The utility of the state space representation for tasks with inner dynamics is reduced because in most CPS environments it is not possible to undo the actions, and prepare a different strategy:

9


3. The classic problem solving theory used mainly verbal protocols as data. However, TALK ALOUD INTERFERES PERFORMANCE IN COMPLEX DYNAMIC TASKS (Dickson, McLennan & Omodei, 2000)

4. Independence (or very short-term dependences) of actions/states is assumed in some of the methods for representing performance. That is, the features that represent performance are local

What is LPSA and how it relates to these problems and

other theories

11

Latent Problem solving Analysis(LPSA)

• m(trial) = f{m(sa1), m(sa2),….. m(san), context}

• Simplifying assumptions:m(trial1) = m(sa11) + m(sa21) +….. + m(san1) m(trial2) = m(sa12) + m(sa22) +….. + m(san2)…. m(trialk) = m(sa1k) + m(sa2k) +….. + m(sank)

• Where sa is a ‘state or action’

12

Latent Problem solving Analysis(LPSA)

• Complexity reduction: Reducing the number of dimensions in the space reduces the noise

13

Cow

Cheetahcalf

LSA LPSAThe problem space is a metric space, where states and trials are represented as vectors

14

LPSA as a theory of representation in CPS tasks

(1) human similarity judgments

(2) ‘Strategy’ changes

(3) Expertise effects of amount of practice

(4) Expertise effects of amount of environmental structure

(5) Applications: Automatic landing technique assessment

15

Approaches to complexity: The ant and the beach parable

(Simon, 1967,1981)

16


(Simon, 1967,1981)

17


(Simon, 1967,1981)

18


(Simon, 1967,1981)

?

19

• Unsupervised learning• Empirical adjustment of a problem space• Definition of a productivity mechanism and

a similarity measure.• LPSA: addition and cosine.

20

LPSA solutions for the problems with the classic 'problem space’ approach

1. The problem with the ‘generation’ of the problem space

2. The utility of the state space representation for tasks with inner dynamics is reduced because in most CPS environments it is not possible to undo the actions, and prepare a different strategy:

LPSA proposes a mechanism to generate automatically the problem space

21

• The classic problem solving theory used mainly verbal protocols as data. However, TALK ALOUD INTERFERES PERFORMANCE IN COMPLEX DYNAMIC TASKS (Dickson, McLennan & Omodei, 2000)

• Independence (or very short-term dependences) of actions/states is assumed in some of the methods for representing performance. That is, the features that represent performance are local

LPSA solutions for the problems with the classic 'problem space’ approach

LPSA does not assume independence or short dependences between states/actions. Indeed, it uses the dependences of all of them simultaneously to derive the problem space. The features that represent performance are global

LPSA uses log files and human judgments as data, but not concurrent verbal protocols

Theoretical surroundings of Latent Problem Solving

Analysis

23

Mental representations

Perceptual symbols

Propositions

Rules and theories Similarity based (varying in the amount of structure

represented)

Continuous features

Set theoretic models

24

Anderson (1978)

1. Encoding processes

2. Processes of internal transformation

3. Decoding processes

LPSA applied to model human judgments

26

Main equivalence:

Docs

Words

Participants’ trials

Actions :

Move_4_Copter_11_4_11_9_Forest_

1 Move_4_Copter_11_4_11_9_Forest_2 Move_2_Truck_4_11_17_7_Clearing_3 Drop_Water_4_Copter_11_9_Forest___4 Move_3_Copter_8_6_10_11_Forest_5 Move_1_Truck_4_14_18_10_Forest_6 Drop_Water_3_Copter_10_11_Forest___7 Move_4_Copter_11_9_21_8_Dam_8 Move_3_Copter_10_11_12_14_Dam_9 Control_Fire_2_Truck_17_7_Clearing___10 Control_Fire_1_Truck_18_10_Forest___11 Move_4_Copter_21_8_12_10_Clearing_

27

Firechief corpus

• Data from the experiments described in experiments 1 and 2 in Quesada et al. (2000), and Canas et al. (2003).

• Total: 3441 trials, 75.575 different actions

• The first 300 dimensions where used

28

log files containing series of actions

actio

ns

57000 actions 3400 log files

Tria

l 1Tr

ial 2

Tria

l 3

Action 1Action 2

29

Three examples of performance

• 8 first actions in a trial

1 2

3

RELATED

NON RELATED

301 2 3 4 5 6 7 80 10 119 12 13 14 15 16 17 18 19 20 21 22 23 24

12

34

56

78

010

119

1213

1415

CONTROL FIRE

CONTROL FIRE

1

311 2 3 4 5 6 7 80 10 119 12 13 14 15 16 17 18 19 20 21 22 23 24

12

34

56

78

010

119

1213

1415

DROP WATER

CONTROL FIRE

CONTROL FIRE

2

321 2 3 4 5 6 7 80 10 119 12 13 14 15 16 17 18 19 20 21 22 23 24

12

34

56

78

010

119

1213

1415

CONTROL FIRE

CONTROL FIRE

3

33

comparisons

method Example 1 to

Example 2 Example 1 to Example 3

Example 2 to Example 3

Subjective similarity high low Low

LSA Cosine 0.7219 0.0567 0.0711

Exact matching 0.125 0 0

Transitions between actions 0.971 1 0.971

34

Possible way of comparison: Exact matching of actions

• Exact matching: count the number of common actions in two files. The higher this number, the more similar they are

35

Example 1 Example 2 Move_2_truck_4_11_13_3_forest move_2_truck_4_11_12_15_forest

Move_1_truck_4_14_16_14_forest move_1_truck_4_14_13_5_forest Move_3_copter_8_6_11_12_forest move_4_copter_11_4_11_9_forest

move_4_copter_11_4_11_9_forest drop_water_4_copter_11_9_forest control_fire_2_truck_13_3_forest move_4_copter_11_9_13_8_forest

control_fire_1_truck_16_14_forest control_fire_2_truck_12_15_forest move_2_truck_13_3_17_7_clearing move_2_truck_12_15_13_14_forest move_1_truck_16_14_20_12_forest control_fire_2_truck_13_14_forest

Example 1 Example 3

move_2_truck_4_11_13_3_forest move_2_truck_4_11_2_2_pasture

move_1_truck_4_14_16_14_forest move_1_truck_4_14_0_5_forest move_3_copter_8_6_11_12_forest move_4_copter_8_6_8_4_clearing

move_4_copter_11_4_11_9_forest move_3_copter_8_6_8_10_clearing control_fire_2_truck_13_3_forest control_fire_2_truck_2_2_pasture control_fire_1_truck_16_14_forest control_fire_1_truck_0_5_forest

move_2_truck_13_3_17_7_clearing move_4_copter_8_4_4_2_forest move_1_truck_16_14_20_12_forest move_3_copter_8_10_2_3_clearing

36

comparisons

method Example 1 to







37

Possible way of comparison: Transitions between actions

• count the number transitions between actions in two files. Create matrices, and correlate them

38





Example 1 Example 3





39





Example 1 Example 3





40

Example 1 Example 2 move_2_truck_4_11_13_3_forest move_2_truck_4_11_12_15_forest

move_1_truck_4_14_16_14_forest move_1_truck_4_14_13_5_forest move_3_copter_8_6_11_12_forest move_4_copter_11_4_11_9_forest



Example 1 Example 3





41

Example 1 Example 2

Move_2_truck_4_11_13_3_forest move_2_truck_4_11_12_15_forest




Example 1 Example 3





42

(a) (b) Example 1 Example 2 drop move control drop Move Control drop 0 0 0 drop 0 1 0

move 0 4 1 move 1 2 2 control 0 1 1 control 0 1 0

(c) (c)

Example 3 drop move control drop 0 0 0

move 0 4 1

control 0 1 1

Possible way of comparison: Transitions between actions

43

comparisons

method Example 1 to







44

comparisons

method Example 1 to






LSA Cosine 0.7219 0.0567 0.0711

•LSA has correctly inferred that the remaining actions, although different, are functionally related

•Exact matching is not sensitive to similarity differences (exigent criterion).

Since Transitions between actions is blind to most of the information in the logs, it fails because declares as similar

performances that are not

LPSA - Human Judgments correlation

46

Human Judgment correlation • if LSA captures similarity between complex

problem solving performances in a meaningful way, any person with experience on the task could be used as a validation

• To test our assertions about LSA, we recruited 15 persons and exposed them to the same amount of practice as our experimental participants, so they could learn the constraints of the task.

47

Human Judgment correlation • Replay trials, with different similarities

• People watched a randomly ordered series of trials, in a different order for each participant, which were selected as a function of the LSA cosines (pairs A, B, C, D, E, F, G with cosines 0.75, 0.90, 0.53, 0.60, 0.12 and 0.06 respectively)

0.06

G

0.12

F

0.53

D

0.60

E

0.75

A

0.90

B

48

Human Judgment correlation

One of the pairs was presented twice to measure test-retest reliability. That is, for example, pair G was exactly the same as pair A for one participant, the same as pair F for another participant, etc. Filling out a form that presented all the possible pairings of ‘stimuli pairs’ were presented

49

Human Judgment correlation

FULL-SCREEN

REPLAY OF THE TRIAL SELECTED, 8

TIMES FASTER THAN NORMAL SPEED

50

Human Judgment correlation: Results

51

Conclusions• Applied: LSA is an automatic way of generating

a problem space and compare slices of performance in complex tasks. It scales up very well and does not depend on a-priori task analyses

• Theoretical: LSA proposes that problem spaces are metric spaces that are derived from experience. Actions or States that are functionally related are represented in similar regions of the space. In this sense Problem solving is unified with theories of object recognition and semantics.

LPSA as a theory of expertise in problem solving

53

• Ebbinghaus approach: manipulating previous knowledge by eliminating it. Random assignment of participants to groups.

• Chase and Simon approach (expert – novice), manipulating previous knowledge by pre - selecting participants (no random assignment of participants to groups)

• Move complexity to the lab, and manipulate previous knowledge (exactly = amount of practice and experience for all participants)

54

• Ebbinghaus approach: manipulating previous knowledge by eliminating it. Random assignment of participants to groups.


• Move complexity to the lab, and manipulate previous knowledge (exactly = amount of practice and experience for all participants)

55

Move complexity to the lab

• To simulate expertise environments in labs, we need tasks more complex than the standard ones:– More representative– Long learning curve– Interesting enough to keep the motivation for

a long period of time

56

The DURESS Microworld

• Goals: – To keep each of the reservoir temperatures

(T1 and T2) at a prescribed temperature ( e.g., 40 C and 20 C, respectively)

– To satisfy the current mass (water) output demand ( 5 liters by second and 7 liters by second, respectively)

57

58

59

60

61

62

63

DURESS

• Christoffersen Hunter, & Vicente (1996, 1997, 1998) 6-month long longitudinal experiment using Duress II. 225 trials, with different goals values. Every participant received exactly the same kind of trials.

• However, analysis mostly qualitative. Not without a good reason…

64

Example DURESS II protocolVAR1 VAR2 VAR3 VAR4 VAR5 VAR6 VAR7 VAR8 VAR9 VAR10 NEWVAR11NEWVAR12NEWVAR13NEWVAR14NEWVAR15

1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 02 3 0 0 0 0 0 0 0 0 0 0 0 0 0 03 6 0.273689 0 0.000138 0 0 0 0.000188 0.073746 0 7.38E-05 0 0 0 0.0001744 9 1.418777 0 0.000712 0 0 0.00261 0.001449 0.385738 0 0.000388 0 0 0.00261 0.0013595 12 2.496949 0 0.001255 0 0 0.020982 0.004418 0.696833 0 0.000701 0 0 0.020982 0.0042326 15 3.336244 0 0.001677 0 0 0.03198 0.008833 0.942067 0 0.000947 0 0 0.03198 0.0085737 18 3.936109 0 0.001977 0 0 0.06 0.014306 1.11746 0 0.001122 0 0 0.06 0.0139988 21 4.312084 0 0.002166 0.000842 0 0.081715 0.022185 1.227147 0 0.001234 0.000157 0 0.081715 0.0207159 24 4.586229 0.013575 0.002303 0.001652 8.05E-06 0.108101 0.032371 1.307358 0 0.001314 0.000394 0 0.108101 0.028693

10 27 4.743035 0.324127 0.002383 0.002292 0.000216 0.137773 0.045039 1.35428 0.146906 0.001361 0.000581 0.00017 0.133886 0.0379911 30 4.842909 0.514431 0.002434 0.00282 0.000378 0.162776 0.059069 1.383688 0.491208 0.00139 0.000736 0.000591 0.152776 0.0464812 33 4.905937 0.734883 0.002465 0.003252 0.000584 0.189602 0.074055 1.401696 0.671362 0.001408 0.000862 0.000847 0.173068 0.05414713 36 4.944792 0.909096 0.002484 0.003607 0.000772 0.209861 0.089675 1.412798 0.789265 0.001419 0.000966 0.001042 0.18493 0.06133914 39 4.969547 1.003641 0.002496 0.003904 0.000904 0.239094 0.105762 1.419871 0.872862 0.001426 0.001053 0.001201 0.199547 0.06819915 42 4.979459 0.948248 0.002503 0.00414 0.000895 0.258917 0.122595 1.422703 0.913472 0.00143 0.001122 0.001307 0.209459 0.07477316 45 4.989136 0.913023 0.002506 0.004334 0.000901 0.287408 0.140073 1.425468 0.936597 0.001432 0.001178 0.001386 0.219136 0.08121817 48 4.765001 0.890729 0.002393 0.004494 0.000917 0.308542 0.157957 1.492649 0.978125 0.0015 0.001225 0.001492 0.23 0.08753218 51 3.649077 0.88084 0.001834 0.004622 0.000944 0.328321 0.175024 1.811692 1.003555 0.001821 0.001263 0.001562 0.248321 0.09480219 54 2.691963 0.877833 0.001352 0.004727 0.000983 0.344334 0.190718 1.965049 1.017191 0.001974 0.001293 0.001601 0.263001 0.10330420 57 1.908286 0.872994 0.000959 0.004814 0.001029 0.35 0.205373 1.927847 1.025149 0.001937 0.001319 0.001628 0.288022 0.11198421 60 1.386679 0.87 0.000697 0.004886 0.001082 0.353689 0.219138 1.784063 1.029625 0.001793 0.00134 0.001652 0.303689 0.12011622 63 1.077631 0.87 0.000542 0.004944 0.00114 0.36 0.232322 1.631448 1.031909 0.001639 0.001357 0.001677 0.313366 0.12741423 66 0.897433 0.87 0.00045 0.004993 0.001199 0.36 0.24515 1.504779 1.031062 0.001512 0.001371 0.0017 0.33 0.1339724 69 0.795081 0.87 0.000399 0.005032 0.001261 0.36 0.257705 1.414729 1.01644 0.001421 0.001382 0.001703 0.33082 0.13992325 72 0.738594 0.87 0.00037 0.005064 0.001324 0.36 0.270064 1.353304 1.008371 0.001359 0.001392 0.001718 0.340703 0.1454926 75 0.768093 0.87 0.000385 0.00509 0.001387 0.36 0.282277 1.295564 1.004906 0.001302 0.001399 0.001738 0.35 0.15070827 78 0.94002 0.87 0.000473 0.005111 0.001447 0.36 0.294581 1.216285 1.002201 0.001222 0.001405 0.001763 0.357298 0.15542228 81 0.964371 0.87 0.000484 0.005129 0.001506 0.36 0.306915 1.190239 1.001429 0.001196 0.001411 0.001789 0.36 0.15980729 84 0.881871 0.87 0.000443 0.005143 0.001564 0.36 0.319035 1.172993 1.0002 0.001179 0.001415 0.001815 0.36 0.16398330 87 0.812034 0.87 0.000408 0.005155 0.001623 0.36 0.330913 1.135694 1 0.001141 0.001418 0.001841 0.368281 0.16788731 90 0.820305 0.87 0.000413 0.005164 0.001682 0.36 0.342611 1.080892 1 0.001086 0.001421 0.001868 0.37 0.17140832 93 0.825 0.87 0.000415 0.005172 0.00174 0.36 0.35417 1.038963 1 0.001044 0.001423 0.001896 0.37 0.1745633 96 0.842689 0.87 0.000424 0.005178 0.001797 0.36 0.365602 1.005498 1 0.00101 0.001425 0.001924 0.37 0.177434 99 0.85 0.87 0.000428 0.004716 0.001851 0.36 0.376412 0.984395 1 0.000989 0.001427 0.001952 0.37 0.1799435 102 0.854809 0.87 0.000429 0.004098 0.001895 0.36 0.385152 0.97192 1 0.000976 0.001263 0.001975 0.37 0.18178336 105 0.850205 0.87 0.000428 0.003578 0.001931 0.35041 0.392332 0.964579 1 0.000969 0.001124 0.00199 0.37 0.1827437 108 0.873514 0.87 0.000439 0.003179 0.001958 0.35 0.397659 0.976187 1 0.000981 0.001018 0.001996 0.37 0.18297238 111 0.909544 0.87 0.000458 0.002842 0.001979 0.357386 0.402016 1.000421 1 0.001005 0.000976 0.001996 0.37 0.18291439 114 0.947416 0.87 0.000476 0.00256 0.001994 0.36 0.405565 1.024947 1 0.00103 0.000955 0.001994 0.37 0.18284640 117 0.98365 0.87 0.000494 0.002336 0.002005 0.36 0.408281 1.049267 1 0.001055 0.000938 0.00199 0.37 0.18283541 120 1.01317 0.87 0.000509 0.002146 0.002011 0.36 0.410506 1.070121 1 0.001075 0.000924 0.001983 0.37 0.18288242 123 1.02949 0.87 0.000516 0.001994 0.002015 0.36 0.412195 1.081089 1 0.001086 0.000913 0.001976 0.37 0.18298743 126 1.03 0.87 0.000518 0.001729 0.002014 0.36 0.413206 1.082857 1 0.001088 0.000924 0.001969 0.37 0.18317644 129 1.03 0.87 0.000518 0.00141 0.00201 0.36 0.413484 1.082857 1 0.001088 0.000933 0.001962 0.378123 0.18343945 132 1.017387 0.87 0.000512 0.001155 0.002002 0.36 0.41281 1.074449 1 0.00108 0.000941 0.001957 0.38 0.18376946 135 1.005 0.87 0.000506 0.000946 0.001991 0.36 0.411474 1.059373 1 0.001064 0.000948 0.001953 0.38 0.1840947 138 1.004015 0.87 0.000506 0.000773 0.001977 0.36 0.409614 1.040043 1 0.001045 0.000953 0.001951 0.38 0.18436748 141 1.0008 0.87 0.000504 0.000631 0.001962 0.36 0.407348 1.026229 1 0.001031 0.000957 0.001949 0.38 0.18459549 144 1.00075 0.87 0.000503 0.000519 0.001945 0.36 0.404689 1.016964 1 0.001022 0.000961 0.001949 0.38 0.1847850 147 0.993036 0.87 0.0005 0.000424 0.001927 0.363928 0.401805 1.011174 1 0.001017 0.000964 0.001949 0.38 0.18494751 150 0.99 0.87 0.000497 0.000347 0.001909 0.37 0.398712 1.008056 1 0.001013 0.000966 0.00195 0.38 0.18510452 153 0.988359 0.87 0.000496 0.000284 0.001889 0.37 0.395462 1.006674 1 0.001011 0.000968 0.001951 0.38 0.18525553 156 0.985 0.87 0.000495 0.000232 0.00187 0.37 0.392096 1.005291 1 0.00101 0.00097 0.001951 0.38 0.18540254 159 0.985 0.87 0.000495 0.00019 0.00185 0.37 0.388648 1.004286 1 0.001009 0.000971 0.001952 0.38 0.18554655 162 0.985 0.87 0.000494 0.000155 0.00183 0.37 0.385145 1.003955 1 0.001008 0.000972 0.001953 0.38 0.18568856 165 0.985 0.87 0.000494 0.000127 0.00181 0.37 0.381607 1.002857 1 0.001008 0.000973 0.001954 0.38 0.18582757 168 0.985 0.87 0.000494 0.000105 0.001789 0.37 0.378052 1.002857 1 0.001008 0.000973 0.001955 0.38 0.18596458 171 0.985 0.87 0.000494 0.000184 0.00177 0.37 0.374636 1.002857 1 0.001008 0.000974 0.001957 0.38 0.18609859 174 0.985 0.87 0.000494 0.000466 0.001754 0.37 0.371877 1.002857 1 0.001008 0.000974 0.001957 0.38 0.1862360 177 0.985 0.87 0.000494 0.00082 0.001742 0.37 0.370086 1.002857 1 0.001008 0.000975 0.001958 0.38 0.186358

34 variables, governed by mass and energy conservation laws

65

Main equivalence:

Docs

Words

Participants’ trials

states

TR1_TR2_MO1_MO2_ (…)

40 _ 20 _ 15 _ 7_ (…)

35_10_15_6_ (…) 35_10_15_6_ (…) 36_12_15_6_ (…) 36_12_15_6_ (…) 36_13_15_6_ (…) 38_15_15_7_ (…) 38_15_15_7_ (…) 39_18_15_7_ (…) 40_18_15_7_ (…) 40_20_15_7_ (…) 40_20_15_7_ (…)

66

log files containing series of StatesSt

ates

57000 States 1151 log files

Tria

l 1Tr

ial 2

Tria

l 3

State 1State 2

67

Long Term Working Memory (LTWM)

Ericsson and Kintsch (1995)

EPAM IV

(e.g., Gobet, Richman,

Staszewski and Simon,

1997)

Constraint Attunement Hypothesis

(CAH)Vicente and

Wang (1998)

Current theories of expertise

68

Long Term Working Memory (LTWM)

Ericsson and Kintsch (1995)

EPAM IV

(e.g., Gobet, Richman,

Staszewski and Simon,

1997)

Constraint Attunement Hypothesis

(CAH)Vicente and

Wang (1998)

PROCESS THEORIESPRODUCT THEORY

Current theories of expertise

69

• Ebbinghauss approach: manipulating previous knowledge by constancy (=0). Random assignment of participants to groups.


• Move complexity to the lab, and manipulating previous knowledge by constancy (= Exact amount of practice and experience for all participants).

70

LTWM (Ericsson and Kintsch, 1995)

• STM accounts for working memory in unfamiliar activities but does not appear to provide sufficient storage capacity for working memory in skilled complex activities (p.220)

• LTWM is acquired in particular domains to meet specific demands imposed by a given activity on storage and retrieval. LTWM is task specific.

71


• Intense practice in a domain creates retrieval structures: associations between the current context and some parts of LTM that can be retrieved almost immediately without effort (example: SF and digits).

• LTWM permits rapid and reliable reinstantiation of a context after interruption without a decrease in performance.

72


• LTWM theory proposes that LTWM is generated dynamically by the cues that are present in short term memory.

• During text comprehension, where the average human adult is an expert, retrieval structures are retrieving propositions from LTM and merging them with the ones derived from text.

73

CAH (Vicente and Wang, 1998)• Contrary to what process theories maintain,

Constrain Attunement Hypothesis (CAH) does not commit to a particular psychological mechanism to explain the phenomenon of expertise.

1. How should one represent the constrains that the environment (i.e., the problem domain) places on expertise?

2. Under what conditions will there be an expertise advantage?

3. What factors determine how large the advantage can be?

74

CAH (Vicente and Wang, 1998)

• Describing the constraints in the environment is the task of an expertise theory.

from Shanteau (2001)Higest level of performance Lowest level of performance

Aided decisions competent restricted Random

Wheather forecasters chess masters parole officers polygrahersastronomers livestock judges psychiatrists managerstest pilots grain inspectors student admissions stock forecastersinsurance analysts photo interpreters intelligence analists parole officersPhysicists soil judges court judges

75

CAH (Vicente and Wang, 1998): the Abstraction Hierarchy

PURPOSE

STRATEGIES

TACTICS

FUNCTIONS

PLAYERS

Win the game

Score at least 2 runs in this inning

Advance all by one base

Alternative tactics to achieve strategy above

Hit Run Run

Batter 1st base runner

2nd base runner

76

FUNCTIONAL

ABSTRACT

GENERALIZED

PHYSICAL

Overall system goals (how much water each reservoir is outputting,

and at which temperature)

conservation of mass and energy for each reservoir (how much mass & energy is entering and

leaving the reservoir).

Settings of valves, pumps, and heaters

Flows and storage of heat 'FA','FA1','FA2','HTR1‘…

'D1','D2','T1','T2'

'MI1', 'MO1','EI1', 'EO1', 'M1', 'E1',…

'PA','PB','VA','VA1','VA2‘,…

Continuum of abstraction, means- ends relationship between levels


77

FUNCTIONAL

ABSTRACT

GENERALIZED

PHYSICAL

Overall system goals (how much water each reservoir is outputting,

and at which temperature)

conservation of mass and energy for each reservoir (how much mass & energy is entering and

leaving the reservoir).

Settings of valves, pumps, and heaters

Flows and storage of heat 'FA','FA1','FA2','HTR1‘…

'D1','D2','T1','T2'

'MI1', 'MO1','EI1', 'EO1', 'M1', 'E1',…

'PA','PB','VA','VA1','VA2‘,…


78

• LTWM claims that the magnitude of expertise effects is “related to the level of attained skill and to the amount of relevant prior experience”

• CAH argues that this claim is incomplete. Expertise effects in memory recall are also determined by the amount of structure in the domain (and by active attunement to that structure)

• LPSA is sensible both to ‘relevant previous practice’ and to ‘amount of structure in the domain’

LTWM vs. CAH

Design and predictions

80

3/4 1/4

?

81

3/4 1/4

?

82

83

Predictions• Only huge amounts of experience with the system

would enable the actor (human or model) to make accurate predictions of the last quarter of the trial

• Sparse practice should clearly lead to poor prediction

• Only structured environments should show the expertise advantage. Following CAH, the expert (human or model) should not do well in a completely unstructured environment

Results

85

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Functional Abstract Generalized Physical

Aver

age

cosi

ne

10 nearest neighbors10 random trials

Average cosine between the fourth quarter of a target trial and the fourth quarter of the 10 nearest Neighbors

When the three first quarters are used to retrieve the neighbors

Three years of experience with DURESS

86

Average cosine between the fourth quarter of a target trial and the fourth quarter of the 10 nearest Neighbors

When the three first quarters are used to retrieve the neighbors

Six months of experience with DURESS

-0.2

0

0.2

0.4

0.6

0.8

1

Functional Abstract generalized physical

Aver

age

cosi

ne


87

Average cosine between the fourth quarter of a target trial and the fourth quarter of the 10 nearest NeighborsWhen the three first quarters are used to retrieve the neighbors

Three year of experience in a DURESS with no constraints (random states)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Functional Abstract generalized physical

Aver

age

cosi

ne


Conclusions

89

conclusions• In LTWM’s original formulation the retrieval

structures were under-specified. In LPSA, the basic mechanisms postulated are defined computationally.

• In CAH’s original formulation, the representation of the environmental constraints (its most central assertion) where under-specified too. LPSA proposes an automatic mechanism to represent the statistical regularities of the environment.

90

conclusions• LPSA can explain both LTWM and CAH main

assertions – LTWM claims that the magnitude of expertise effects

is related to the level of attained skill and to the amount of relevant prior experience

– CAH claims that expertise effects in memory recall are also determined by the amount of structure in the domain (and by active attunement to that structure)

• Better yet, LPSA proposes both processes and representational structures

91

conclusions• What does this mean for theorizing about

problem solving?– As in LTWM for text comprehension, we propose that

in expert problem solving the current context automatically and effortless retrieve past knowledge, and adapt it to the current situation.

– This retrieval is specific to the domain of expertise, and requires a long period of practice. Short period will not do.

– This retrieval is only possible in domains that show constrains that the expert can use (attune).

92

conclusions• GENERALITY: the fact that the same

mechanism, with the very same underlying assumptions, can be used for language and Problem Solving is interesting per-se: In LTWM, the retrieval structures for chess are different compared to the ones proposed for text comprehension; In CAH, two AH for two different tasks are different too; In LPSA, any space for any task is a vector space.

Automatic Landing Technique Assessment using Latent Problem

Solving Analysis (LPSA)

94

The problem• There is currently no methodology to automatically assess landing

technique in a commercial aircraft or a flying simulator. Instructors are a significant cost for training and evaluation of pilots, and the use of instructors also incorporates a subjective component that may vary from pilot to pilot.

• The advantages of automatic landing technique evaluation are many: (1) Reduced cost of the evaluation. (2) Increased objectivity in the evaluation. (3) Decrease the influence of the instructor. (4) Perfect Test-retest reliability. (5) It is always available and can be triggered by the trainee at will. (6) The model can rate as many landings as time enables, etc.

95

A solution: Latent Problem Solving Analysis (LPSA)

• Latent Problem Solving Analysis (LPSA, Quesada, Kintsch and Gomez, 2002) is based on Latent Semantic Analysis (LSA, Landauer and Dumais, 1997) . Instead of using word occurrence statistics and huge samples of text, LPSA uses a representative amount of activity in controlling dynamic systems (actions or states).

• Like words, states and actions appear in particular contexts but not in others. Some states and actions are interchangeable, being ‘functional synonyms’. Given the right algorithms and sufficient amounts of logged trials, a problem space can be derived in a similar way as semantic spaces are.

• In this application of LPSA to landing technique evaluation, we assume that an expert uses her past knowledge to emit landing ratings by comparing the current situation to the past ones, and generates an expanded representation of the environment by composing the past situations that are most similar to the current one.

96

97

Complex, dynamic tasks are intractable when considered as a whole:

98

Complex, dynamic tasks are intractable when considered as a whole:

• We need to perform complexity reduction, in a mostly automatic way– The triangulation technique– Dimensionality reduction (LPSA)

99

The triangulation technique:

C: Modeling of the expert with restricted information:Accessible

B: Selection (part of the theory)

Expert with complete access to any variable in the system

Expert with partial access to the variables (having to make a selection)

MODEL

A: Modeling of the whole situation: Inaccessible

100

Complexity reduction (I): variable selection using differently informed experts

101

Criteria used by the experts

Levels

Flare Initiation altitude

Too high Correct Too low

Thrust Reduction Too fast Correct Too slow

Pitch Angle All the way too high

Partly too high

Correct Partly too low

All the way too low

Overall Landing Score

1 2 3 4 5

102

• A state is a string of text consisting of the values of each variable (reduced information expert’s) joined by underscores, to make it a single token, like:

“time tag_vertical acceleration_Radio altitude_Thurst”

• A landing is a collection of these states. The variables were sampled ten times per second, and the landing time was 15 seconds approximately, so each landing contained about 150 states

Complexity reduction (II) : Using SVD, the problem space is a vector space

103

Complexity reduction (II) : Using SVD, the problem space is a vector space

104

Results

106

Results

107

Results• The two landing raters’ agreement is not too high; however, it

is similar to other experts’ agreement, such as Clinical Psychologists (0.40), Stockbrokers (<0.32), Polygraphers (0.32) and Livestock Judges (0.50). Their agreement is lower than the ones reported for Weather Forecasters (0.95), Pathologists (0.55), Auditors (0.76) and Grain Inspectors (0.60) (Shanteau. 2001).

• The correlation between the model, and the reduced information expert is about the same as the correlation between the two humans (0.48 vs. 0.46). Note that the ceiling for the model is the correlation between two humans doing the task: a model that correlates with one human better than the between-human correlation is under suspicion. The correlation for the complete information expert was .39, even though the model was not trained to mimic him.

108

Results

0.36

0.52 0.510.46

0.34

0.73

0.25 0.25

0.390.46

00.10.20.30.40.50.60.70.80.9

1

flare initiationaltitude

thrust reduction pitch angle ov. Landingperformance

averageagreement

Poly

chor

ic c

orre

latio

n

agreements human - human agreements human reduced information - LPSAhuman complete information - LPSA

Flare initiation altitude

Thurst reduction

Pitch angle

109

ResultsNote that the only criterion where the model correlates with any of the

experts more than they correlate to each other is thrust reduction. Thrust reduction seems to be a very difficult feature to judge, since the agreement between human experts is the lowest (0.27) and also it is the one in which the reduced information expert obtains the lowest test-retest reliability (0.538, see Table 1 4 in page 119).

All the polychoric correlations between the reduced information expert and the model were significant (p = .002), so were the correlations between complete information expert and model.

The equivalent model without dimensionality reduction (400 dimensions, 5 neighbors, no weighting, no timestamp) produced correlation values of 0.37, 0.08, 0.57 and 0.50 for the above used criteria respectively.

110

Results: no-constraints corpus

-0.12

0.02

-0.01 -0.04

0.12

0.45

0.03 0.060.17

-0.06

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

flare initiationaltitude

thrustreduction

pitch angle ov. Landingperformance

averageagreement

Poly

chor

ic c

orre

latio

n

agreements human - human agreements human reduced information - LPSAhuman complete information - LPSA

Flare initiation altitude

Thurst reduction

Pitchangle

111

ConclusionsPreviously LPSA has been proved as a powerful theory to model behavior

in complex, dynamic problem-solving tasks, and has been proposed as a theory of expertise, see Quesada (unpublished). However, this is the first time that LPSA is used to develop technology that can be used in industrial applications.

In previous work, we have presented an experience-based approach to problem solving. Problem solving is viewed as the extraction of useful representations from a corpus of situations. The creation of the representations is a primarily bottom-up, unsupervised process. It is proposed that the problem space can be viewed as a vector space. People use their past knowledge to perform complex, dynamic tasks by comparing the current situation to past ones, and generate an expanded representation of the environment by composing the past situations that are most similar to the current one. In complex dynamic situations, this intuitive, pattern-based system can have a very important role.

112

ConclusionsIt is possible to construct systems that grade landing technique

automatically as well as humans, if we consider that the limit of performance for such a model is the human-human agreement. The correlation human-human was low (0.46) but within the range of some other areas reported (Shanteau, 2001). In a large-scale application of the model (for a training and evaluation department, for example), we can imagine that 500 pilots need to be evaluated. In that situation, only a small proportion of randomly sampled landings (that can be kept from previous sessions) must be evaluated by humans; the rest is performed by the system. Since the model has different landing criteria, it could emit recommendations such as: ‘In this landing, you initiated the flare too high, and reduced the thrust too late. Keep that in your mind for the next one’.

113

ConclusionsA direct consequence of the availability of a system like LPSA for the

development of psychological theory is that some experiments that were prohibitive before could now be planned within the budget. Since instructors are a sparse resource, an experimenter may decide that she cannot afford to run a particular, very promising experiment, because of the expenses associated with performance assessment. With an automatic and reliable method to perform the evaluation, more complex experiments could be feasible.

Documents

Complex problem solving (CPS) definition