Embed Size (px)
Citation preview
Process and Methodology to Measure the
Impact and Value of FTTSA Certification
Proposal from SAP AG to Fair Trade and Tourism South Africa
1.0 Introduction
Fair Trade in Tourism South Africa (FTTSA) certifies tourism products in tourist destinations
making them accessible to fair-minded independent travelers. FTTSA also formally approves
tour operators looking to improve their sustainability performance and sell packaged travel to
sustainability-savvy consumers. Packages that meet the FTT standard are certified and labeled,
creating a unique selling point in the market and signaling to customers that travel is organized
in a manner that is consistent with the principles of Fair-trade and beneficial to workers,
communities, small-scale businesses and the environment.
In October 2012, four consultants from SAP AG, the world’s leading enterprise applications
software company, were asked to help FTTSA determine a way to measure the impact and
demonstrate the value of certification (or “approval”, in the case of tour operators) on
sustainable tourism. FTTSA wanted to specifically confirm how changes in business
performance of a certified customer (i.e. certified businesses or approved tour operator) were
linked to certification (as opposed to, for example, being attributed to other market factors
such as a boom in the economy, special event, etc.)
The SAP consultants participated in this assignment as part of SAP’s Corporate Social Sabbatical
program….a 4 week pilot program designed to provide in-kind services to non-profit
organizations in South Africa. The team included:
Vatsala Kakroo (from SAP India) [email protected]
Frank Kochendoerfer (from SAP America) [email protected]
Alex Prus (from SAP AG in Germany) [email protected]
Julie Scanlon (from SAP America) [email protected]
The deliverables of the project included:
• A process/methodology to measure the impact of certification
• Sample impact metrics and questions
• A high level process to demonstrate the value of certification and membership (i.e.
value of FTTSA business development efforts, value of FTTSA label)
2.0 Approach:
The team took a multi-faceted approach to the work including:
One on one interviews with key internal and external resources familiar with
sustainable tourism
One on one interviews with SAP subject matter experts and other external consultants
familiar with impact measurement
Desktop research both within and outside the sustainable tourism industry
The following chart illustrates the interview and desktop research conducted:
3.0 Guiding Principles
In order to make sure the final recommendation was implementable by FTTSA, the team made
sure the recommendation was:
• Lean i.e. simple to implement & cost effective
• Realistic
• Comprehensive
• Scalable as FTTSA grows
• Leverage-able by alliance partners & certification bodies
• Flexible to accommodate future builds/enhancements
4.0 Measuring the IMPACT of Certification
4.1 Metrics
In order to properly collect impact information, FTTSA must confirm the metrics they would
like/are required to report to various stakeholders. FTTSA must also construct corresponding
questions to acquire this impact data from the businesses/tour operators. The two charts
below represent initial thinking on this topic. The team leveraged The Fair Trade standard to
develop the metrics and standards. They are therefore categorized as follows:
Financial
Workforce
Social
Environmental
Note: The charts assume an accommodation business. The metrics and questions will need to
be customized for other types of tourism businesses as well for tour-operators. Also, this is still
in draft form.
Metric (Benchmark Data) Questions for business (first year audit) (or source)
Financial (YOY change attributed to certification)
1. Last 12 months revenue
2. Occupancy rate
3. # tour operators booking rooms with business (outbound, inbound)
4. # electronic distribution system operators booking rooms with business
5. Average Achieved rates
Financial:
1. What was your revenue for the past 12 months?
2. What was your occupancy rate for the past 12 months?
3. How many tour operators have booked rooms with you during the last 12 months (outbound, inbound)?
4. How many electronic distribution system operators booked rooms with you during the past 12 months?
5. What was your average achieved rate for the past 12 months??
Workforce (YOY change attributed to certification)
1. Total number of direct employees/workforce
2. Total # indirect jobs
3. Total # people benefitting from direct employment
4. Total # people benefitting indirectly
5. Total number of local residents employed
6. Total number of women employed
7. Total number of disabled employed
8. Total number of historically disadvantaged employed
9. Percentage of employee retention
Workforce
1. Confirm # of direct employees for the past 12 months
2. Data comes from multiplier formula ( x 1.5)
3. Data comes from household multiplier formula
4. Data comes from household multiplier formula
5. Confirm # of local employees during the past 12 months
6. Confirm # of female employees during the past 12 months
7. Confirm # of disabled people employed during the past 12 months?
8. Confirm # of historically disadvantaged people employed
9. What is your employee retention for the past 12 months?
4.2 Process Overview
The team reviewed a number of options to measure impact including options which were
embedded in the (soon to be launched) Flo-Cert certification / approval process, and options
outside the new Flo-Cert certification / approval processes. (FTTSA has contracted with Flo-Cert
to conduct certification/approval on their behalf.) A total number of four options were
reviewed and a full SWOT analysis was conducted on each one. (See appendix for details.)
In the end, it was determined that the best approach for impact measurement would be to
leverage the Flo-Cert certification / approval processes. The drivers of this decision related to
the fact that this option would pose the least burden, would be the least disruptive and would
be simplest for customers.
4.3 High Level Approach
The diagram below outlines the recommended process for measuring impact at a high level.
Metric (Benchmark Data) Questions for business (for First Year Audit)
Social Investment (YOY change linked to cert.)
1. # of local vendors/providers in supply chain
2. Money spent on local vendors
3. Money invested in community projects
4. # man days on community service
5. # people directly benefiting from social investment
6. # people indirectly benefitting from social investments
Social Investment1. How many local vendors/providers do you work with?
2. How much money did you with local vendors?
3. How much did you invest in community projects in the last 12 months?
4. How many man days did you contribute to community service?
5. How many people benefited directly from your social investment?
6. Data comes from household multiplier formula
Environmental (YOY change attributed to cert.)
1. Amount of money invested in improving environmental management systems
2. Amount of money invested in Biodiversity conservation projects
3. Cost of water consumption
4. Cost of energy consumption
Environmental
1. How much money did you invest o improve environmental management systems this year?
2. How much money did you invest in biodiversity conservation projects this year?
3. What was the cost of water consumption this year?
4. What was the cost of energy consumption this year?
Tourist Info (Businesses collect info at Check in)
1. # of guests that heard of property via FTTSA
2. # guests who have heard of FTTSA
3. List of ways tourists heard about FTTSA
Tourist Questions (Businesses ask questions and provide info during audit
1. How did you hear about us (i.e. hotel)? (multiple choice)
2. Have you ever heard of Fair Trade in Tourism (FTTSA)?
3. If yes, where did you hear about it? (multiple choice)
Step 1: Collect – In step 1 Flo-Cert collects impact information from businesses and tour
operators (separately) during the standard Flo-Cert certification (for businesses) and
approval (for tour operators) processes. It also includes collection of tourist data via the
businesses and tour operators. More information on this in section 4.4.3.
Once the information is collected, Flo-Cert provides it to FTTSA for analysis. (There may be
some sensitivity regarding the data that can be released to FTTSA. This issue will be
resolved by FTTSA. At a minimum, FTTSA must be able to view the impact-specific data.)
Step 2: Analyze - In step 2, FTTSA sorts and analyzes the data that has been provided by
Flo-Cert according to their requirements.
Step 3: Report - In step 3, FTTSA prepares and disseminates reports to various
stakeholders including e.g., industry bodies, government, businesses, tour operators,
donors or academics.
4.4 Detailed Process
The following chart provides a more granular look at the impact measurement process for a
business (as opposed to a tour operator). (Note: The tour operator impact measurement
process is not documented here but it is assumed that it will be similar to the process for
businesses).
The top half of this chart shows the current version of the new Flo-Cert process and the
bottom half illustrates the recommended builds to measure the impact of certification.
4.4.1 New Flo-Cert Process
Following is an overview of the draft Flo-Cert certification process.
1st Year Audit
Online Application: Businesses that are interested in certification apply online via a
two step process which involves submitting basic business info as well as answering
detailed questions on business operations. The businesses are required to answer
the online questions as well as submit back up information.
Validation: The next step in the process involves Flo-Cert validating the submitted
information (i.e., answers to questions and back up information)
Onsite Audit: The final step in the First Year Audit process involves an onsite audit
conducted by Flo-Cert. Flo-Cert goes through a checklist of criteria on the premises
of the business, interviews various owners, managers and employees and then
makes a go/no go decision with regard to certification. (If additional information is
required, Flo-Cert follows up with businesses before making a final decision.) Flo-
Cert communicates decisions to both businesses and FTTSA.
One and Two Years Post Certification
One and two years post certification, Flo-Cert requests updated licensing and legal
documentation. The businesses must submit these documents electronically to
ensure their certification status remains active.
Reassessment
35 months post certification, Flo-Cert conducts a new onsite audit to determine if a
business should remain certified. This process is similar to the First Year Onsite
Audit. Following the audit, Flo-Cert makes a go/no go decision and communicates
results to businesses and FTTSA.
4.4.2 Recommendation to Build Impact Measurement into the new Flo-Cert Process
It is recommended that the following steps be added to the new Flo-Cert process.
1st Year Audit: Flo-Cert additionally collects initial impact data
In order to determine how certification is impacting business performance, it is
critical to obtain baseline data to benchmark against in subsequent years. It is
recommended that during the Year One onsite audit, Flo-Cert collect this initial
baseline impact data. (A list of questions that enable the collection of this data are
included in section 4.1.) It is anticipated that the work effort associated with
collection of initial impact data will be minimal as only 20 or so questions will be
added to the 300+ criteria already be checked. We propose that submission of this
information, as well as corresponding backup data be a requirement for
certification. Flo-Cert would therefore collect the information and validate it while
onsite.
One and Two Years Post Certification - Flo-Cert adds a survey to check impact data
It is recommended that Flo-Cert send an online survey to businesses during their
annual check in both one and two years post certification. This means that in
addition to businesses sending in licensing and legal documents, they would also be
required to complete this online survey. This survey would include the exact same
20 or so impact questions asked in the initial onsite audit. Back up information must
also be provided by the businesses to validate their responses. The goal of collecting
this information is to compare business performance (or impact) year over year.
At this point, it is also critical to hone in on the business performance change due to
certification (assuming there is a change). To do this, we recommend employing a
method used by a renowned impact measurement company called the ROI Institute
based in the United States. The specific ROI method to be used will be confirmed at
a later date by FTTSA. It is assumed, however, that the method will involve
interviews with key business experts as well as application of proven formulas.
More on the Institute and the methodology in section 4.5.
Flo-Cert will validate the data and follow up with businesses as appropriate.
Reassessment
During the re-assessment phase, it is recommended that Flo-Cert again ask the same
impact questions as in the first year onsite audit as well as the ROI questions from
the online survey (one and two years post certification) . Details on the ROI
methodology are in section 4.5.
4.4.3 Capturing Tourist Data
It is also of interest to FTTSA to capture tourist data. Specifically, FTTSA is interested in
knowing if tourists have booked a particular tourism business due to FTTSA certification
and if indeed, the tourist is aware of FTTSA. The following outlines our recommendation
to capture this info. Again, we have used an example for a business (as opposed to a tour
operator).
Businesses collect tourist info at the point of check in
Ideally, the businesses would add questions to a paper or e-based “check-in” form or
create a new form specifically for this purpose
Businesses would tabulate the results in a very simple table (to be provided by
FTTSA) on an ongoing basis
The businesses would be required to provide the table during certification (starting
one year post certification and continuing annually)
Monthly incentives for tour establishment administrators could be included to
encourage the marketing of FTTSA
Note: There is also an opportunity to market FTTSA at the point of check in by distributing
FTTSA marketing brochures that could include e.g., QR codes linking tourists to the FTTSA
website or other social media sites.
The process for tour operators would be similar but instead of asking questions at check
in, the tour operators would ask the questions via online surveys. The data would be
captured electronically by the tour operators and shared with Flo-Cert as a required part
of the approval process (starting one year post approval and continuing annually)
4.5 The ROI Institute and Process
The ROI Institute is an industry leader in measurement and evaluation with 30 years
experience working with Fortune 500 companies and government agencies in 50 different
countries. The ROI methodology employs proven processes to determine business
impact, return on investment and intangible measures (among other things). This proven
methodology was developed in the 1970’s. The team at ROI has authored 50+ books on
this topic and is considered a pioneer in impact measurement. Clients include: The
United Nations, The International Monetary Fund, the World Bank, SAP America, and
many more.
4.5.1 The ROI Method of Measurement and Evaluation
The ROI Institute makes use of many different methods and formulas to understand and
capture ROI such as:
• Control groups
• Trend line analysis
• Forecasting models
• Participant estimates
• Manager estimates
• Senior management estimates
• Experts input
• Customer input
Based on the SAP team’s understanding of ROI techniques and also based on the
Institute’s track record, we are confident FTTSA can utilize ROI’s methods to directly tie
business performance changes to certification. One method being considered works very
nicely with the proposed new impact measurement process described in section 4. It
leverages the senior manager estimate technique. In a nutshell, the technique works as
follows:
1. Collect baseline data
2. Collect updated data 12 months later and then calculate the delta or change
3. Isolate the effects driving the change
Determine the factors influencing the change
Ask questions to estimate the percent of influence each factor has on the change
Confirm confidence of this estimate
This technique is illustrated in the following example. Note: This scenario assumes Flo-
Cert is working with a hotel during the certification process.
1. Collect Baseline Data – Flo-Cert collects initial data for benchmarking during the Year
one Audit e.g.,
Question for business - “What was your revenue for the last 12 months?”
Answer – “800,000 rand”
2. Collect updated data (12 months later) and the delta – Flo-Cert sends the online
survey to businesses to capture updated impact data and the delta e.g.,
Question for business - “What was your annual revenue for the last 12 months?”
Answer – “1,000,000 rand”
Delta = 200,000 rand (year over year change)
3. Isolate the effects due to certification - Flo-Cert asks follow up question to business via
the same online survey as noted above
Question 1 for business - “What factors contributed to the increase in revenue?”
Possible answers - “The economy, World Cup, certification, etc.”
Question 2 for business “Of those factors, what percent of the increase did
certification drive?”
Answer – “20% or 40,000 rand”
Question 3 for business - “What confidence, expressed as a percentage, can you
put in your estimate?”
Answer – “50%”
Then Multiply 40,000 x 50% = 20,000 rand (result due to certification)
During the reassessment (onsite audit phase), Flo-Cert has the opportunity to ask multiple
business experts (as opposed to just one during the online survey phase) the ROI
questions to get a more thorough calculation of the impact due to certification.
To reiterate, the method above is one of many techniques the ROI Institute uses to
measure impact. It is recommended that FTTSA work with the ROI Institute to determine
the best approach for their requirements.
5.0 Demonstrating the VALUE of certification (i.e. the value of FTTSA business development
efforts, the value of FTTSA label)
In addition to business performance data, it is recommended that FTTSA also capture and
articulate the value they provide post certification. There are two ways to approach
this….inside out and outside in.
The inside out approach allows FTTSA to articulate the value of their business development and
marketing services. In this approach, the FTTSA business development leader continues regular
tracking and monitoring of business development/marketing activities e.g.
• # Media mentions
• Trade show marketing activities
• # Postcards and where distributed
• # media tours and # locations visited
• Blog statistics
• Web statistics
The business development leader then quantifies the monetary value of this work using
industry standard formulas (to be determined by FTTSA) and communicates the results to
customers and donors via newsletters, quarterly updates, phone calls, website news, customer
video testimonials, etc.
The outside in approach, involves understanding and then communicating the value of the
label. Specifically, the goals are to: determine the market awareness of the FTTSA Label within
South Africa; and to determine how (in this case, non-approved) Tour Operators are using the
FTTSA label (and if it has resulted in increased business).
Determining market awareness of the FTTSA label could be complex, expensive and time
consuming. In the interest of time and cost effectiveness, we therefore recommend the
following for FTTSA:
Create a Facebook or Linked in survey or campaign
o Make it interesting - use incentives
o Put links on key websites (e.g. SA Tourism) to drive people to the Facebook or Linked
In page
o Include a survey on the page(s) with questions such as:
Have you ever heard of Fair Trade in Tourism (FTTSA)?
If yes, where did you hear about it?
To determine the (non-approved) Tour Operators use of the FTTSA label we recommend
FTTSA resources:
Conduct desktop research i.e. look into the tour operators website to see if the label is
profiled
Follow with phone interviews to ask how many FTTSA certified products the tour
operators have sold and why
6.0 Critical Success Factors
In order to ensure a flawless implementation, the following critical success factors have been
identified:
1. Adequate funding from donors to cover the additional costs incurred for Flo-Cert work
and for ROI training
2. Agreement with Flo-Cert to take on the additional impact measurement work during
certification and to learn and implement (as required) the ROI methodology
3. Commitment of time from FTTSA to set up and drive the process
4. Clear, concise, and quantifiable impact questions
5. A robust tool to help FTTSA in data collection and management (see section 8 for more
details)
.
7.0 Next Steps
Detailed next steps are identified in the Project Plan Excel document (separate attachment).
8.0 Certification Tool Requirements
FTTSA is in the process of developing a new tool to capture certification data. Once the tool is
developed, Flo-Cert will use it to collect and manage certification information. The following
additional requirements have been identified to enable impact and value measurement data
collection and analysis.
General Tool Features:
The tool must embed the impact analysis questions within the certification criteria
checklist and within the appropriate sections
ROI questions should not appear in the first year audit as they are not needed until one
year, two years and three years’ post certification (third year is reassessment)
The tool must be able to identify the impact questions separately from the other
certification questions to allow for easy reporting.
Survey functionality is required to collect and update baseline impact data
The tool must have the ability to pre-populate cells with previous year data, pulled from
the customer database.
The tool must have the ability to automatically calculate values, such as the delta of
year-over-year, ROI, or industry formulas that enable computation of certain factors
(e.g. indirect jobs)
The certification tool must provide basic querying and reporting capabilities
The tool must be able to collect and aggregate not only customer data but also
marketing and business development data e.g., number of media mentions, blogs, etc
into a master database
If possible, automated reminder emails regarding deadlines, next certification steps, re-
assessment etc. could be added
Authorized Users:
Flo-Cert, businesses, tour operators, and FTTSA
Example 1: Auto population and calculation capability
Example 2: Auto population and calculation capabilities with sample ROI questions:
Appendix
Impact Measurement Process Options and SWOT Analysis
• The SAP team examined 4 options related to impact measurement:
1. Within the certification/approval process (with follow ups every three years)
2. Within the certification/approval process (with follow ups annually)
3. Outside the certification/approval process (online only with annual follow ups)
4. Outside the certification/approval process (case study method with annual
follow ups)
• After conducting a SWOT analysis on all the options, we recommend Option 2 as the
best option for FTTSA
• Note: the following information focuses on certified businesses. The process for impact
measurement for approved tourism operators will be similar but will require
customization to the approval process
----------------------------------------------------------------------------------------------------------------------------
