S519: Evaluation of Information Systems Analyzing data: Causation Ch5

S519: Evaluation of Information Systems

Analyzing data:

Causation

Ch5

Step5: Analyzing data

Dealing with the causation issue, basically be able to answers following questions: How certain does the client need us to say that the

evaluand „caused“ a certain change? What are the basic principles for inferring causation? What types of evidence do we have available to help

us identify or rule out possible causal links? How should we decide what blend of evidence will

generate the level of certainty needed most cost-effectively?

Certainty about causation (D-ch5)

Each decision-making context requires a different level of certainty

Quantitative or qualitative analysis All-quantitative or all-qualitative

Sample choosing Sample size

Mix of them

Inferring causation: basic principles Two basic principles:

Look for evidence for and against the suspected main cause (i.e., evaluand)

Look for evidence for and against any important alternative causes (i.e., rival explanations)

Too many evidences or causes, which are the primary causes All based on the level of certainty you need for your evaluation Stepwise process

Put yourself in the hardest critics, gather enough evidence to support your explanation Repeat it until all remaining alternative explanations are ruled out.

Critical multiplism: triangleThe harder people attack, the more solid your answers need to be.

Inferring causation: 8 strategies

1: ask observers Two possibilities

Ask actual or potential impactees Ask indirect impactees (i.e., co-worker, parents...)

Design your interview questions to include causation questions E.g., how much has your knowledge increased as a result of

attending this program? – get primary cause E.g., did anything else besides the program increase your

knowledge in this area over the same period of time? – get other causes

E.g., please describe anything else that has happened to you or someone you know as a result of participating in this program? – get the causes which people know or believe were caused by the program.


1: ask observers Causation-rich questions tend to be leading (direct the

respondent to answer in a particular way). Be careful about the wording when designing interview questions

The causation question is not just whether the program produced the effect but also what other factors enabled or inhibited the effect.

Individual might not be a reliable witness to answer the causation question, other evidence will be required to make justifiable causal inferences.


1: ask observers Methods

Questionnaires to identify the targeted groups (people who experienced substantial changes) Using open-end to get more opinions

In-depth interview with the targeted groups To find out causation.


2. Check whether the content of the evaluand matches the outcome Alcoholics treatment program alcoholics avoid

relapses Check whether the strategies which alcoholics

use to avoid relapses after the program, are the same as the strategies taught in the program


3. Look for other telltale patterns that suggest one cause or another Modus operandi method – look for evidence --

detective metaphor to describe the way in which potential causal explanations are identified and tested.

A silly example Evidences: a naked man, dead; in the middle of the desert;

personal belongings near by; half match at hand Cause of his death

Inferring causation: 8 strategies 4. Check whether the timing of outcomes makes sense

Common sense: an outcome should appear only at the same time as or after whatever

caused it – a considerable delay. A further downstream the outcomes, the longer they should take to

appear Using timing to confirm or disconfirm causal links:

Is the outcome happened before the evaluation? Or Other downstream outcomes too early?

Is the timing of the outcomes logical to possible causes? Do the further downstream outcomes in the logic model occur out of

sequence?

More on Lipsey, M. W. (1989). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.

Inferring causation: 8 strategies 4. Check whether the timing of outcomes makes sense Example – a community health education on diet and

exercise Fairly immediate knowledge and skill gain: during or immediately

after the intervention A short delay (days or weeks) before the knowledge and skills

are transformed into changing behavior A moderate delay (weeks or months) before we see changes in

individual health indicators (weight, cholesterol, blood pressure, etc.)

A long delay (months or years) to see changes on improvement on diabetes and heart diseases

Exercise

Take grantsmanship workshop as one example List the timeline potential outcomes (fairly immediate, a short

delay, a moderate delay, a long delay) Using timing strategies to confirm or disconfirm the cause links,

state one page for how and why: One month after the workshop, 3 proposals got grants One year after the workshop, 3 proposals got grants Three months after the workshop, some people write good proposals,

but some are not.

Think about your own solution Form a group and discuss


5. Check whether the „dose“ is related logically to the „response“. The dose-response idea

The more dose of drug, the better response later on If more A (dose), then better B (response) Compare the less dose with more dose (not overdose)

to confirm or disconfirm the cause links E.g., for performance evaluation project, if we found that

performance had been improved dramatically in the unit where the system has been poorly implemented, this system is not the cause of the performance improvement.


6. Make comparisons with a „control“ or „comparison“ group Divide the participants into different groups

control group (receive the evaluand) vs. Comparison group (receive no evaluand)

Sampling should be done carefully to make sure no systematic differences between groups Sample size randomization


7. Control statistically for extraneous variables When using control and comparison groups, try to

exclude external variables and make two groups no systematic differences: Statistical methods

Regression analysis Try to identify other potential systematic differences

E.g., math improvement for students, how to sample students and think about other potential existing difference. Is the random sampling enough?


8. Identify and check the underlying causal mechanism(s) Try to look for an underlying mechanism to make

the case for causation more or less convincing Cigarette smoking lung cancer

Correlation studies Carcinogenic in cigarette causes cancer

Normally coming from literature.

Put them together

Do we need all the evidences we collect from 8 strategies?

How to select them? Put yourself in the shoes of a tough critic, identify the

most potential threatening rival explanation, then chose the types of evidence that will most quickly and cost-effectively confirm or dispel that rival explanation.

Go to next less tough rival explanation, ... Continue, until you have amassed a body of evidence

to provide you enough certainty to draw causal inferences

Exercise

Grantsmanship workshop (p57) Grantsmanship workshop strengthen local

communities For (evidences) Against (evidences) Other alternative causes (i.e., rival explanations)

Using strategies to confirm or disconfirm these evidences or causes

Putting them together Form a group to discuss

Documents

S519: Evaluation of Information Systems Analyzing data: Causation Ch5