BY TEMTIM ASSEFAOCTOBER, 2013

Quantitative Research Method (Research Design)

Overview of Quantitative Research

Uses deductive method of knowledge acquisition

Intends to falsifies an existing theory Tries to generate generalizable knowledge

Accepts objectivity of knowledge Uses a standard measurement instruments The researcher is independent in the process

of knowledge constructionIntends with prediction and controlling

Research Process

Problem

Conceptual FW, Hypothesis

Develop Instrument

Research Design Data

Collection & Analysis

Data Interpretation

Step 1: Problems/Questions

A question/problem occurs or is posed to the researcher for which that researcher has no and can find no answer(s).

Example – usabilityThe software development industry has invested in the development of a variety of Usability Inspection Methods (UIMs) to address Web usability issues (Matera et al., 2006). UIMs are procedures in which inspectors examine usability-related aspects of a user interface (Rocha and Baranauska, 2003). However, in (Rivero et al., 2013), we identified that there is a lack of UIMs that are able to find usability problems in early stages of the development process.

Step 2: Literature, Theory, Logic

Available literature/theory is reviewed to determine if there is already a solution to the problem.

Existing solutions do not always explain new conditions.

The existing solution might require some revision or even be discarded.

Develops a hypothesis as a tentative solution to the problem

Stone (1956) defines hypothesis as a tentative explanation about as the relationship between two or more variables

Empirical to theoretical mapping

Conceptual Framework - User Involvement in system development

Participation

Conflict Conflict resolution

Influence

Types of Quantitative Research Design

A research Design is nothing more than a plan for conducting a research in such a way as to answer a research questions.

A number of questions arise to select the research design – who, what shall be the focus of the study? What attributes of individuals, organizations, systems, etc, In what contexts observations will be made (organizations, laboratories, etc), what methods employed to analyze the data?

Cont’d …

Researchers are constrained with people, time, money, etc to select the best the research design

A prime consideration is designing cost effective research

Selecting a research design Quantitative research method design includes Survey /field study field experiment, laboratory experiment, simulation, and case study

Decide form of data collection

QuestionnairesInterviewsObservation

Questioners

Most frequently used measurement instrument in organization studies

Includes sets of questions to which the subject responds

It is used to measure: Attitude Opinions Demographic characteristics of the subjects

Questionnaires design

Questionnaires make use of lists and rating scalesBehaviors and attitudes are complex and cannot

be easily evaluated and quantifiedCheck list is a list of behaviors, characteristics

or other entities that a researcher is investigatingEither the researcher or participants simply

check(s) items from the listWhat are features of user friendly software

1. Graphical interface2. Clear navigation direction3. Immediate feedback4. Other specify ____________

Questionnaire …

A rating scale is more useful when a behavior, attitude, or other phenomena of interest needs to be evaluated on a continuum scale

It is designed with the following scales 1. “inadequate” to “excellent”, 2. “never” to “always” or 3. “strongly disapprove” to “strongly

approve”

Guideline for Questionnaire

1. Keep it short• What do I do with the information?• Is it absolutely essential to have this information

to solve part of the research problem2. Use simple, clear, unambiguous language3. Check for unwarranted assumptions implicit in

your questions How many cigarette do you smoke each day?

Good to add a choice 25 ___ 25-16 ____ 15-5 ___ <5 ____ None ____

4. Word your questions in ways that don’t give clues about preferred or more desirable responses

What strategies have you used to try to quit smoking?

Leads him to list strategies he did not try

Guide line …

Check consistency – that leads to give contradictory answer for two questions

Determine in advance how you will code the response

Keep the respondents task simpleProvide clear instructionsMake the questionnaire attractive and professional

looking Conduct a pilot test

Give for half a dozen friends to see they have difficulty understanding any items

Scrutinize the almost final product carefully to make sure it address your needs

Advantage of Questionnaire

Easily to collect data from large number of people, including those who live thousands of miles away

Can be administered by unskilled data collectorsPresents a uniform stimulus to all subjectsCan be collected online or through mailResearcher does not have contact with the

respondentsThe anonymity of the respondents helps

respondent to provide more truthful information than they would be in personal interviews, especially on sensitive and controversial issue

Draw back of questionnaire

Majority of people may not return, if it is mailed the return rate is not more than 50%

Returned responses may not be a true representative of the originally selected samples

Cannot be used with illiterate or persons who cannot read

Most questioners are inflexible, respondents answer by choosing only from the given lists

Interviews

Interviews are meetings in which the interviewers directs questions at the interviewee and records the obtained responses

It is direct face to face interactionsCan be undertaken through telephoneHelp to captured more detailed information

than questionersAllow to include the respondents views

Design of Interview

Can be structured and unstructured Unstructured interview is designed with open

ended questions The researchers asks and the responds reply

to the questions The same questions may not be asked for

different respondents Examples

Can you list down the benefit of the new software for your organization?

What are the main drawbacks of the new software? What do you suggest to improve software to solve

the current problems?

Structured interviews

The researcher asks standard set of questions and nothing moreIn semi structured interview, the researcher may follow the standard questions with

one or more individually tailored questions to get clarification or probe a persons reasoning

In structured interview, you can have checklist of responses Example

What are the benefits of LAN in the organizationa. Facilitates communications among employeesb. Improves information accessc. Improves relationship with customers d. Other specify

What kind of the support the management provides to the success of the Intranet system

Advantages of Interview

Interview is formal and emotionally neutralHelp to develop hypothesis at early stage of

the research than testing hypothesis It has a distinct advantage of establishing

rapport with potential participants and gain cooperation

Has higher response rateFlexible to include additional views and

opinionsHas better validity, the researcher can

further ask probing questions on some biased response

Disadvantage

It is generally expensive than other data collection methods

The researcher is the main interviewer otherwise training of interviewer is very expensive and time taking

The interviewer presents the questions in different manners that lead to different responses – affects the validity and reliability of the data

Characteristics of the respondents (age, sex, ethnicity, etc) may affect the value of measured variables

Respondents may not free on some controversial issues

Data Analysis and Interpretation

Does the observed data supports the previously postulated research hypothesis/relationships between variables

There are different data analysis (statistical ) methods for data analysis

Commonly mentioned as descriptive and inferential statistical methods

Descriptive methods include: Frequency, mean, median, correlation, Charts and graphs, etc

•

Data analysis …

Inferential methods include: Parametric test: t-test, ANOVA, linear regression, Non-parametric test: Chai square test, Man Whitney test

Benefits

Theoretical propositions can be tested in an objective fashionEnable generalizationsEasy to use (although may be an illusion)Established and accepted research method across multiple fieldsAided by statistical toolsImmensely helped by advancement computingStrong in external validity if sampling errors are minimized.

Field Experiment

Field Experiment

A field experiment is an experimental research method which is performed outside the laboratory in the natural settings

It follows all steps of scientific process1) selection and determination of a problem,

2) selection of participants and measuring instruments,

3) selection of a research plan,

4) execution of the plan,

5) analysis of data and

6) formation of conclusions

Features of FE

It has three unique features1. the research takes place in

natural setting

2. the experimenter manipulates one or more independent variables while exerting much control over confounding variables; and

3. the effect of the manipulations on one or more dependent variables is systematically observed.

FE Design

There three type of designs

Controlling Nuisance in Field Experiment

Solomon Four Groups in True ExperimentExperimental group 1: R O X O Control group 1: R O O Experimental group 2: R X O Control group 2: R O

Removing statistically the effect of suspected confounding variables

Example – Interface Design and Computer use

System use is low among employees. One reason is poor interface design. You divide the employees into two groups (Group 1 & 2)Redesign the Interface and let Group 1 work on the new

system - Treatment GroupLet Group 2 work on the old Interface– Controlled

GroupAfter six or some months after Interface design,

measure, the computer use.If computer use is increased for Group 1, you can

conclude that Interface design and computer use has association

Data collection and Analysis

The following methods are used to collect data Questionnaire Structured interview Computer log files

Data analysis Descriptive statistical methods Inferential statistical methods

Advantages of FE

Conducted in natural settings and has higher experimental validity

Help to clearly identify antecedents of observed effects in casual relationships

Used for the development of theory as well as for the solution of applied problems;

The logic of FE can be applied in the analysis of many naturally occurring changes

Help for testing of broad hypotheses dealing with complex social process in lifelike situations.

Disadvantage

The methods used to control confounding variables in FE are not sufficient

Manipulation of variables cause legal and ethical problemsDifficult to control dynamically changing environment

during the course of the experimentDifficult to precisely measure dependant variables in the

field settingsExpensive to conduct compared to lab experimentNeeds highly skilled person to design and conduct

Role of FE in computer Science

It has many applications in IS To develop theory To test hypothesis To evaluate IS tools and techniques

Practical Examples

Fukada et al (n.d.) used field experiment to study the effect

of Road Facility Management Support System on asset

management of the public infrastructure.

Mayur et al (2000) made study to compare and suggest the

best training method for companies between instructor

based training and computer based training methods.

Chen et al (2007) made a field experiment study to identify

effects of different types of social information on

contributions to an online communities.

Review Questions

Explain the different quantitative research design methods

What is the strength and weakness of each methodWhat are the different survey research methods What is the main data analysis methodswhat problem can you address with quantitative

research methods What are the procedures in quantitative Research How do you use Quantitative research in computer

science?