Customer Expectations & WTP For Improvements in Service from Electricity DNOs Pilot Debrief February 2008

Customer Expectations & WTP For Improvements in Service

from Electricity DNOs

Pilot Debrief

February 2008

Agenda

• Fieldwork Methodology

• Interviewer Feedback

• Stated Preference Methodology

• Stated Preference Findings

• Summary of Recommended Changes

Fieldwork Methodology

• Objectives of pilot were to test:– the recruitment process – the survey hit rate– the clarity and flow of the questionnaire– the appropriateness of the language used– the accuracy of all routings– ease of use of the show material– design and understanding of the stated preference exercises– the interview duration

• Approach:– Businesses: 40 CATI phone-post/email-phone (41 achieved)– Residential: 100 CAPI in-home interviews (99 achieved)

• Quotas:– Business:

• 10 EPN (11 achieved), 10 SPN (10 achieved), 20 LPN (20 achieved)• Mix of size (20 small; 15 medium; 6 large) & sector• Minimum 10% with outage experience (39% achieved)

– Residential:• 25 EPN (25 achieved), 25 SPN (25 achieved), 50 LPN (49 achieved)• Mix of age & SEG

General Feedback: Interview Duration

31.3

18.2

35.3

15.2

0

10

20

30

40

50

25 minutes orless

>25-30minutes

>30-45minutes

45 mins+

% R

esp

on

de

nts

4.9

21.9

29.3

43.9

0

10

20

30

40

50

25 minutes orless

>25-30minutes

>30-45minutes

45 mins+

% R

esp

on

de

nts

DOMESTICMean = 32 minutes

BUSINESSMean = 43 minutes

General Feedback: Comments on Domestic Questionnaire (non SP)

• Problems with clarity & flow:– Question 34: ‘did you find each of the levels of

service described easy to follow?’… if you said ‘no’, Question 35 then said ‘which levels weren’t clear to you?’; seem like separate issues

– Need showcard for income question

• Problems with routing:– Q22b; if they say don’t know should skip

subsequent question

• General comments on what would make it easier– Concern that people will think we are trying to

sell them services for an electricity company. Also, many people less aware of Ofgem than of Ofwat or the FSA, so they were very cautious. Maybe a more elaborate explanation of the survey might help.

• Interview duration:– Longer than budgeted for in most instances;

cannot be done within current fee (due both to time required for each and impact of length on hit rates) and current incentive (due to length)

• Other:– Some issues with defining rural versus urban –

we will use a postcode dbase to do so in the mainstage

– Add all screening questions (eg age, SEG) to main questionnaire

– Add question about number of children 5+

General Feedback: Comments on Business Questionnaire (non SP)

• Problems with clarity & flow:– Some people assumed it was a sales call

despite the current intro. Need to further emphasize that Ofgem are a regulatory body and that the research is looking at improvements that could be made to the services provided to their company

– RQ13 – need a “don’t know code– Q23: some said “both”; add this code?– Q26: needs a don’t know or can’t

remember– Q40: needs re-phrasing so it doesn’t

suggest anything will be sent; needs to separate leaflet and email; needs to be multicode

• Problems with routing:– none

• Interview duration:– Longer than budgeted for in most

instances; cannot be done within current fee (due both to time required for each and impact of length on hit rates)

• Other:– Some issues with defining rural versus

urban – again, we will use a postcode dbase to do so in the mainstage

Stated Preference Methodology

• Objective: to obtain estimates of customers’ willingness-to-pay for improved service levels (or willingness-to-accept for reduced service levels)

• No revealed preference evidence, so need to rely on stated preferences– Customers are not offered different service levels and prices in real world

• Two possible stated preference approaches:– Contingent Valuation questions (CV) – ask respondents directly questions

about valuations

– Discrete choice experiments (SP DCE) – elicit valuations through tradeoffs• Based on hypothetical choices• Essential that price is one of the attributes

Example Choice: Experiment 1 (Domestic)

Choice 3

Average number of power cuts longer than 3 mins in normal weather conditions

Average duration of power cut

Average number of power cuts shorter than 3 mins in normal weather conditions

Information provided during power cuts

Annual Electricity Bill

Choice (mark "X" in preferred option)

Cost£200

(no change)£198

(£2 decrease)£205

(£5 increase)

Provision of InformationTelephone information line

Telephone information lineplus call backs to provide

information updatesTelephone information line

Duration of Loss of Power100 mins on average

115 mins on average(more than now)

80 mins on average(less than now)

Which electricity distribution service would you choose?

Frequency of Power Cuts (more than 3 minutes)4 in 5 years

7 in 5 years(more than now)


As Now Alternative 1 Alternative 2

Frequency of Short Interruption (less than 3 minutes)5 in 5 years


5 in 5 years

CV vs SP DCE

Contingent Valuation (CV)Discrete Choice Experiments

(DCE)

May present cognitive problems

DCE does not explicitly ask about money values, so may

be easier for people to understand

Typically collect one response from each individual

More efficient - typically collect multiple responses

from individuals

Argued that CV should be chosen when WTP for the good or service in total is

required

Consumer surplus methods can be used to estimate the

value of a total good from DCE data

Attributes Tested for Domestic Customers

All Distributors except LPN LPN

Experiment 1

Frequency of power cuts over 3 mins Frequency of power cuts over 3 minsAverage duration of power cuts over 3

minsAverage duration of power cuts over 3

mins

Number of short power interruptions Number of short power interruptions

Provision of Information

Experiment 2

Restoration of supply (time) Restoration of supply (time)

Compensation for restoration of supply Compensation for restoration of supply

Compensation for multiple interruptions Compensation for multiple interruptions

Planned interruptions - notice Planned interruptions - notice

Experiment 3

Undergrounding Provision of InformationNetwork resilience to major storms Network resilience to flooding

Network resilience to flooding Reduction in carbon emissionsReduction in carbon emissions

Cost also included in all experiments

Attributes Tested for Business Customers

Cost also included in all experiments

All Distributors except LPN LPN

Experiment 1

Frequency of power cuts over 3 mins Frequency of power cuts over 3 mins

Average duration of power cuts over 3 mins

Average duration of power cuts over 3 mins

Number of short power interruptions Number of short power interruptions

Provision of Information

Experiment 2

Restoration of supply (time) Restoration of supply (time)

Compensation for restoration of supply Compensation for restoration of supply

Making and keeping appointments Making and keeping appointmentsPlanned interruptions - notice Planned interruptions - notice

Experiment 3

Network resilience to major storms Network resilience to floodingNetwork resilience to flooding Reduction in carbon emissions

Reduction in carbon emissions Energy efficiency advice

Energy efficiency advice Provision of Information

Stated Preference Methodology

• Potential problem with separation of attributes across experiments is that results may over-estimate total wtp

• Therefore we also include a higher-level experiment, including all attributes to test total wtp– We also include a series of CV questions to elicit total wtp for best service

• The higher-level experiment is simplified so that ‘group’ of attributes are varied simultaneously

• Two levels tested– Lowest service level– Highest service level

• Still complicated

Example higher-level choice: 6 choices

Choice 1

Frequency of Power Cuts over 3 mins Average number of power cuts longer than 3 mins in normal weather conditions

Duration of Loss of Power Average duration of power cut

Frequency of Power Cuts less than 3 mins Average number of power cuts shorter than 3 mins in normal weather conditions

Provision of Information Information provided during power cuts

Restoration of Supply (Normal Weather) Restoration of electricity supply as a result of problems not related to weather

Compensation for Failure to Restore Supply (Normal Weather) Compensation if your electricity is not restored in this time

Compensation for Multiple Interruptions Number of interruptions lasting 3 or more hours in a year which will entitles you to compensation payment

Planned Interruption to Electricity Supply Amount of notice that customers must be given before planned interruptions to electricity supply

Undergrounding An ongoing commitment to underground overhead lines for amenity reasons

Network resilience to major storms Number of customers affected by major storms

Network resilience to flooding Expenditure on flood defences

Reduction of carbon emissions Replacement of mobile generation equipment and vehicles with those using less polluting vehicles

Cost Annual Electricity Bill

Choice (mark "X" in preferred option)

Neither

Which electricity distribution service would you choose?

5% of overhead linesper annum


Guaranteed within 18 hours

Alternative 1

None

Replace 10% per year with those using less polluting fuels

80 mins on average(less than now)

£50 plus £25 for every additional 12 hour period

2 days notice


Telephone information lineTelephone information lineplus call backs to provide

information updates

Compensation after5 interruptions

Compensation after5 interruptions

120 mins on average(more than now)

£207(£7 increase)

Guaranteed within 18 hours

Increase by 15%

£196(£4 decrease)

144000 customerson average in a year(20% less than now)

216000 customerson average in a year(20% more than now)

Continue usage of current equipment and vehicles

Current expenditure

Alternative 2

1 in 5 years(less than now)

3 in 5 years(less than now)

£50 plus £25 for every additional 12 hour period

2 days notice

Stated Preference Methodology – Analysis

• Analysis based on principles of utility maximisation

• ‘Utility’ assigned to each alternative, made up of:– Attributes of the alternative– Characteristics of the respondent, e.g. income, etc.

• Assumption that error distribution is extreme value (iid) leads to logit model

• Maximum likelihood techniques used to obtain values of unknown parameters– Find the βs that optimise explanation of the model

• Model outputs– Coefficient estimates (and t-statistics)– Model fit statistics

Utility Specification

inni XXXU ...221

Random error reflecting influence

of parameters not included, differences

between individuals/responses

Parameters to be estimated

Characteristics of

alternative/individual

Logit Model Formulation

Assumption that random terms () are independent and identically distributed across alternatives (IID) and that they

vary with an Extreme-Value distribution

Jj

Uj

Ui

i e

eP

Stated Preference Methodology – Analysis

• Coefficients used to:– Determine the relative importance of attributes– Determine the monetary value for attributes– Specify utility functions for prediction models

• Monetary values for attributes identified in lower-level games

• May be adjusted by values for groups of attributes identified by higher-level game or by CV question– This is a judgement call

Stated Preference – Did it Work?

• Trading behaviour

• Background Questions on respondent understanding

• Model Results

Most domestic respondents were willing to choose new service options in at least one of the choices• Domestic – London

• Domestic – Other DNOs

Trading Exp 1 Exp 2 Exp 3 Exp 4 Always Current Option 3 2 2 n/a

Always Alternative 1 0 0 1 0


Neither/No Choice 0 0 0 2

Trading among Alternatives 46 47 46 47

Total Observation 49 49 49 49

Trading Exp 1 Exp 2 Exp 3 Exp 4 Always Current Option 4 3 3 n/a






Most business respondents were willing to choose new service options in at least one of the choices• Business – London

• Business – Other DNOs

Trading Exp 1 Exp 2 Exp 3 Exp 4

Always Current Option 4 5 4 n/a






Trading Exp 1 Exp 2 Exp 3 Exp 4

Always Current Option 0 0 0 n/a Always Alternative 1 0 0 0 0 Always Alternative 2 0 0 0 0 Neither/No Choice 0 0 0 1

Trading among Alternatives 21 21 21 20 Total Observation 21 21 21 21

Examining choices at different cost levels implies that many domestic customers were willing to pay 15% increases in DUOS for service improvements

Experiment 1 Experiment 2 Experiment 3

Other DNOs

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%

Cost Level of Alt A(% adjustment to DUOS)

% C

ho

osi

ng

Alt

ern

ativ

e A

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%

Cost Level of Alt B(% adjustment to DUOS)

% C

ho

osi

ng

Alt

ern

ativ

e B

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%


% C

ho

osi

ng

Alt

ern

ativ

e A

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%


% C

ho

osi

ng

Alt

ern

ativ

e B

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%


% C

ho

osi

ng

Alt

ern

ativ

e A

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%


% C

ho

osi

ng

Alt

ern

ativ

e B

A wider range of costs was investigated in the higher-level experiment with large numbers of domestic users choosing alternatives with large cost increases

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15% 20% 25% 30%


% C

ho

osi

ng

Alt

ern

ativ

e A

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15% 20% 25% 30%


% C

ho

osi

ng

Alt

ern

ativ

e B

Examining choices at different cost levels implies that many business customers were willing to pay 15% increases in DUOS for service improvements

Experiment 1 Experiment 2 Experiment 3

Other DNOs

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%

Cost Level of Alt A(as % of Annual Bill)

% C

ho

osi

ng

Alt

ern

ativ

e A

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%

Cost Level of Alt B(as % of Annual Bill)

% C

ho

osi

ng

Alt

ern

ativ

e B

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%


% C

ho

osi

ng

Alt

ern

ativ

e A

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%


% C

ho

osi

ng

Alt

ern

ativ

e B

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%


% C

ho

osi

ng

Alt

ern

ativ

e A

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15%


% C

ho

osi

ng

Alt

ern

ativ

e B

A wider range of costs was investigated in the higher-level experiment with large numbers of business users choosing alternatives with large cost increases

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15% 20% 25% 30%


% C

ho

osi

ng

Alt

ern

ativ

e A

0%

20%

40%

60%

80%

100%

-15% -10% -5% 0% 5% 10% 15% 20% 25% 30%


% C

ho

osi

ng

Alt

ern

ativ

e B

Most respondents understood the experiments

• ‘Were you able to make the comparisons in the choices we presented to you’

– Domestic• 48 of 49 respondent in London reported they were able to make the

comparisons• 47 out of 50 respondents in other DNOs reported they were able to make the

comparisons

– Business• 19 of 20 respondents in London reported they were able to make the

comparisons• 20 of 21 respondents in other DNOs reported they were able to make the

comparisons

A sizeable proportion of domestic customers felt that the levels of service in the choices were unrealistic, and the experiments were difficult

• ‘Were the levels of service we have been asking about in the choices realistic?’

– London, 11 of 49 domestic respondents felt the service levels were not realistic or too difficult• A 1 in 10 year electricity cut is not realistic because we are having electricity cut 6 times in a

year• I don’t think all the changes will be possible within the costs stated• It is seriously insane to raise the bills again and again• No electricity company would cut emissions, as they are not efficient enough to replace the

current system• There were too many options and everything was too similar• I don’t believe the whole concept, it’s not clear enough• They are all the same• Too much complicated

– Other, 8 of 50 domestic respondents felt the service levels were not realistic or too difficult• Because they are not achievable, ie the cost of putting all these cables underground isn’t

going to be economically viable• The last thing I want is for my bill to go up• They are talking about increasing expenditure by certain percentages, but the original

expenditure is not listed… so 15% increase of what?• The comparisons are not obvious, got to really read in order to give an answer• The different choices or packages didn’t always seem logical.• Too much to absorb all at one for the average consumer who is having enough problems

paying their bills

Most business respondents felt that the levels of service in the choices were realistic

• ‘Were the levels of service we have been asking about in the choices realistic?

– London, 3 of 20 respondents felt the service levels were not realistic• 7 power cuts in 10 years is outrageous• Because we had 3 power cuts in the last 2 years of long duration• They do not employ enough people, so realistically it is not going to happen

– Other, 4 of 21 respondents felt the service levels were not realistic• Because some reductions are not able to be guaranteed. There are elements

outside your control, i.e. the power cut we experienced was due to accidental cutting through power cables.

• Some options offered more for less….doesn’t make sense.• Typical of today’s climate, people want all these things, but are not prepared

to pay for them. Realistically, these improvements would probably cost too much.

• Some of the improvements would be very difficult to achieve.

Model results are encouraging

• Domestic Customer Models– Results are very encouraging– Cost coefficient is significantly negative– Most other coefficients have intuitive signs, many are significant– Resulting wtp values look reasonable

• Business Customer Models– Many statistically insignificant terms, probably a result of small sample sizes

and heterogeneity of sample, e.g. in terms of size of business– Expect model to be improved with larger sample size in main survey– Most important variables:

• Power cuts• Restoration of supply• Reduction of carbon emissions

Interviewer Feedback on SP

• “Very complex”

• “Too much to consider”

• “Consider reducing and rotating over 2 interviews to make each survey shorter”

• “Maybe cut down on the sets of choices; if possible, 6 instead of 8. Choice sets are quite long”

• “Most respondents found the exercise difficult or complex at first but then adapted to it. It is a bit difficult to keep them interested in the survey till the end”

• “The interview should be shortened if possible”

• “It’s a lot of reading”

Conclusions

• Subject area is abstract and the choices are complex, but – despite some concerns on the part of interviewers – respondents seem to have been able to undertake the SP experiments

• Some, mostly minor, revisions required for main survey– Main area for review is cost levels

• Mainstage interviews:– 2,100 domestic face-to-face interviews (150 interviews conducted in each of the

14 DNO territories)– 1050 CATI business interviews (75 per DNO territory)

Recommended Changes

• Review cost levels– Pilot cost levels:

– Fewer reductions and more increases for Experiments 2 and 3, e.g. 2 reductions, 6 increases and base

– Recommend that we incorporate larger cost increases

• Review wording of attributes and levels to simplify as much as is possible (may not be much scope for this)– Include ‘better than now’ and ‘worse than now’ descriptors, where appropriate

• Review ‘Network resilience to flooding’ definition– Complaint that it was difficult to make comparisons when base level is not defined

• Reduce number of choices in lower-level experiments from 8 per experiment to 6 per experiment

Levels 1 2 3 4 5 6 7 8 9Exp 1 - 3 -15% -10% -5% -2% 0% 2% 5% 10% 15%

-10% -4% 0% 2% 6% 10% 16% 22% 26%-12% -6% 0% 3% 7% 12% 18% 24% 28%-15% -8% 0% 4% 8% 14% 20% 25% 30%

Higher Level

Other Recommendations

• Minor Text Changes– Compensation for Multiple Interruptions: minor error in wording ‘which will entitles you…’,

plus (£x) in text to be removed– Network Resilience to Major Storms: include a ‘,’ separator in the numbers, e.g. 180,000

rather than 180000

• Minor Changes to Questions/Routing– Slightly reword Q34 and/or Q35 (domestic) / Q46 and/or Q47 (business)– Amend routing for Q22(domestic) / Q33 (business) “don’t knows”– Use postcode dbase to define rural & urban– Add age & SEG data from screener to main questionnaire (domestic only)– Add question about number of children 5+ (domestic only)– RQ13 (business) – need a “don’t know code– Q18 (domestic)/Q23 (business): some said “both”; add this code?– Q20 (domestic)/Q26 (business): needs a don’t know or can’t remember– Q40 (business only): needs re-phrasing so it doesn’t suggest anything will be sent; needs

to separate leaflet and email; needs to be multicode

• Expand explanation of survey & of Ofgem

• Reduce the length of the questionnaire or increase fees & incentives

Documents

Customer Expectations & WTP For Improvements in Service from Electricity DNOs Pilot Debrief February 2008