WORKING PAPER

The Sandbox Handbook v10A guide to testing and growing EdTech ideas

Date Last updated July 2021

Authors

DOI

Asad RahmanAlice CarterDaniel PlautMiranda DixonTaiye SalamiLaurel Schmitt

105281zenodo5120788

GlobalEdTechHubemspedtechhuborgCreative Commons Attribution 40 International httpscreativecommonsorglicensesby40

EdTech Hub

About this documentRecommendedcitation

Rahman A Carter A Plaut D Dixon M Salami TSchmitt L (2021) The Sandbox Handbook v10 A guideto testing and growing EdTech ideas [Working Paper33] EdTech Hub httpsdoiorg105281zenodo5120788Available at httpsdocsedtechhuborglib772S7Z4RAvailable under Creative Commons Attribution 40Internationalhttpscreativecommonsorglicensesby40

Licence Creative Commons Attribution 40 International

httpscreativecommonsorglicensesby40

Youthinspmdashthinspdear readersthinspmdashthinspare free to share (copy andredistribute the material in any medium or format) andadapt (remix transform and build upon the material) forany purpose even commercially You must giveappropriate credit provide a link to the license andindicate if changes were made You may do so in anyreasonable manner but not in any way that suggests thelicensor endorses you or your use

Notes EdTech Hub is supported by UK aid and the World Bankhowever the views expressed in this document do notnecessarily reflect the views of the UK Government or theWorld Bank

About EdTech Hub sandboxesA sandbox fast-tracks promising EdTech interventions by providing funding tools andaccess to evidence It provides a space for partners to test and grow ideas in conditionsof uncertainty We break sandboxes up into short sprints learning and iterating as we goEach sprint informs changes and new ideas for the next For more information pleasevisit httpsedtechhuborginnovation

The Sandbox Handbook v10 2

EdTech Hub

HelloIf yoursquore reading this odds are you are interested in testing and growing EdTechideas You might be a funder or policymaker interested in how to select andshape EdTech interventions that maximise value for money Or you might be atechnologist an NGO or a grassroots organisation seeking to grow an EdTechidea to bring about large-scale change Or you might be a researcher wanting toexpand your toolkit for gathering data that informs decision making If any ofthose sound familiar this handbook is for you

This guide is based on EdTech Hubrsquos sandbox offer An EdTech Hub sandboxfast-tracks promising EdTech interventions by providing funding tools andaccess to evidence It provides a space for partners to test and grow ideas inconditions of uncertainty In 2020ndash21 we worked with partners in five countries totest EdTech in response to Covid Read about what we did and what we learnt(Rahman et al 2021) The handbook explains the processes involved in oursandboxes to provide a model for your own journey implementing EdTech

Rather than detailing specific applications the handbook outlines the keyprocesses and principles of a sandbox so that anyone can apply the method andways of working to their work Treat it as handrails rather than a manual and mixmatch add to and amend to bring your own magic to the work It is structuredchronologically from light-bulb moment to implementation You can use thehandbook as an end-to-end manual or dip into whichever parts meet you whereyou are

Table 1 What yoursquoll find in the handbook and why itrsquos useful

Section What yoursquoll find How itrsquos relevant to you

Section 1Mobilisingaround theproblem

Who we start the sandbox with

Selecting a sandbox

Understanding our problem and context

Bringing all the most relevant people

As a funder or policymaker understandhow to identify the best EdTech and set itup for success

As a technologist NGO or grassrootsorganisation bring together the bestpeople to solve the right problem

The Sandbox Handbook v10 3

EdTech Hub

together

Articulate a vision for impact

Section 2Setting out aplan to solve theproblem

Sandbox onboarding

Moving from idea to hypothesis

Auditing the hypothesis in the educationsystem using the 6 Prsquos

Mapping risks using the pre-mortem

Going from hypothesis to action plan

Understanding the people behind thedata

As a funder or policymaker make sureimplementation maximises its chances ofachieving impact

As a technologist NGO or grassrootsorganisation come up with a robust andthought-through plan for how to deployEdTech or sense-check the plans youalready have

As a technologist NGO or grassrootsorganisation make sure any techinterventions consider everythingholistically

As a researcher systematically plan allfactors in an EdTech implementationbefore you measure it

Section 3Iterativeimplementation

Reflecting on our work

Iterating our work

Producing evidence outputs from whatwersquove learnt

Additional tools to scale impact

Harnessing collective intelligence

As a funder or policymaker make surean EdTech intervention is proceedingmindfully and adapting to maximiseeffectiveness and impact

As a technologist NGO or grassrootsorganisation make sure yoursquore adaptingthe EdTech implementation based onconstant feedback loops

As a researcher package up evidence asit emerges from an EdTechimplementation to make sure itinfluences decision making

As you read yoursquoll see links to all of our artefacts (128221) and tools (128736) and greyboxes that talk about the mindsets and mechanisms that unlock progress in asandbox With that said remember our process is adaptable to the uniquesuperpowers of your team and context of the sandbox

You should have everything to get started but of course we are here foranything else you might need Please get in touch with asadedtechhuborg if

The Sandbox Handbook v10 4

EdTech Hub

you have any questions or wish to discuss how to apply the handbook to yourwork

Finally Irsquod like to remind you that our ambition is for EdTech Hub to work initerative agile ways just like our sandboxes Please feel free to leave suggestionsor reflections as comments in this document

See you out there

The EdTech Hub Sandbox Team

PS If you wish to dive further into the theory behind our approach here issome helpful reading

128278 The Promise of Lean Experimentation (Murray Peter amp Ma Steve 2015)

128278 Lean Impact (Chang amp Ries 2018)

128278 Pretotype It (Savoia Alberto 2011)

128278 The Lean Data Initiative (Acumen 2019)

128278What is Agile Methodology (Edell 2019)

The Sandbox Handbook v10 5

EdTech Hub

1 Mobilising around the problemThis phase is dedicated to understanding the challenges and opportunitiesavailable as you seek to make an impact through an EdTech intervention Itallows for brainstorming and landscaping of available resources and relevantevidence and helps our partners to articulate their vision for impact including atheory of change (TOC)

11 Who we start the work with

One of our principles for a sandbox is to give a range of actors a seat at the tableA large part of the sandbox is identifying and working with differentorganisations however at the start of a sandbox we typically begin in one of twoplaces For your work think of these as possible minimum viable starting pointsfor moving forward

Government as the lsquoproblem ownerrsquoThe EdTech sector remains more focused on products than the problems theyare trying to address Across the Hubrsquos technical assistance work we noticepartners talking about technologies rather than the problems these might beused to address

Thatrsquos why we recommend starting with the government as the lsquoproblemownerrsquo People within governments are often time-pressed uninformedcustomers of EdTech navigating a fragmented market We can support them towork effectively with the private sector and local businesses who can addresstheir challenges ensuring that wersquoll always be focused on scaling impact notscaling a product

Growing a promising EdTech interventionThere are many EdTech organisations and products that are not scaling As weexplore in our draft Position Paper (Simpson et al 2021) reasons for this includea tendency to focus on the product rather than the problem a reliance on donorfunding to reach the most marginalised and a lack of accessible evidence onlsquowhat worksrsquo

The Sandbox Handbook v10 6

EdTech Hub

Thatrsquos why we recommend starting with organisations or ideas that haveevidence of impact and the potential to scale to help them grow Even when webegin with the EdTech product we still ensure the government is involved andwe are focused on scaling impact not scaling a product

Key artefact128221 EdTech and Covid Insights from our Sandbox Portfolio(Rahman et al 2021) distils 12 key insights (against the 6 Ps framework) from ourCovid Sandbox portfolio It details factors we believe are crucial in growing andscaling EdTech interventions

12 Selecting a sandbox

Criteria for selecting sandboxes

Sandbox criteria have varied depending on EdTech Hub strategy and what wersquovelearnt about what makes a good sandbox This section gives an overview ofcriteria used in the past and our current best thinking on criteria today

In 2020ndash21 EdTech Hub selected five sandboxes in response to Covid Criteria(with more weight for the three in bold) for these sandboxes were

Complements the portfoliorsquos learning objectives

Value add of a sandbox to the overall initiative

Offers clear evidence of impact particularly in pedagogical approachand content

Strong buy-in from a lsquoplacersquo with a dedicated focal point

Access to lsquoreal-lifersquo users (ie schools students teachers communities)

Strengthens education systems in the long-term (ie not just during Covid)

Clear path to rapid scale

Our second portfolio of sandboxes has three lsquomust haversquo selection criteria(building on the previous criteria)

The Sandbox Handbook v10 7

EdTech Hub

1 Likelihood of national scale a sandbox builds towards something thatcan be fast-tracked to national scale-up if validated across the 6 Ps As aresult of this criterion wersquore more likely to work with government iNGOsand scale-ready EdTech organisations

2 Quality and relevance of lsquoevidence inrsquo and lsquoevidence outrsquo a sandboxbuilds on existing evidence that points to impact (eg it aligns with alsquoSmart Buyrsquo (World Bank 2020) and or fills a gap where the evidence ismissing or incomplete (eg it explores situations in which positivemessaging is helpful given this is a validated intervention in broad terms)

3 Strong link to EdTech Hub strategy and focus countries a sandboxshould align with the six EdTech Hub focus countries (Kenya TanzaniaBangladesh Pakistan Sierra Leone Ghana) and our five thematic prioritiesThis means itrsquos much more likely a sandbox will take place in conjunctionwith other EdTech Hub work and complement it When building aportfolio of EdTech interventions we recommend also aligning aroundcountry and or strategy to maximise the value of whatrsquos learnt ensurefocus and avoid shiny object syndrome

13 Understanding our problem and context

Sandboxes might work on education challenges in communities that they arenot members of with partners whose understanding of the context is crucial tothe success of any EdTech intervention At the beginning of a sandbox itrsquosimportant that (as far as possible) we get on the same page about contextualchallenges active or past work and resources and existing evidence whichaddresses our problem

A number of methods can be used in this process

Rapid evidence review and learning sessionsAs a sandbox approaches a specific problem it is important to build onpre-existing knowledge This includes being aware of the existing evidence baseconsulting with experts in the relevant place or theme and becoming familiarwith frameworks or resources that are relevant to the context at hand

The Sandbox Handbook v10 8

EdTech Hub

Some of the activities which can be completed to ensure you are well aware ofexisting knowledge include

Rapid literature review search and ask partners relevant colleagues andother stakeholders for important resources read review and synthesisetakeaways and share them with your sandbox partners

Learning sessions invite relevant experts to a workshop where they sharesome of their insights on a topic or approach allowing you and partners toask questions and summarise key learnings

Insights from this process should not be simply recorded and left to rot in agoogle doc somewhere Rather they should inform design decisions andexperimentation and serve to triangulate critical beliefs and assumptionsthroughout the sandbox One way to make sure these learnings are not forgottenis to turn them into design principles mdash key guidelines which should bereferenced throughout the sandbox Similar guidelines and standards exist in theEdTech space (EdTech Tulna 2020) and can be similarly incorporated into therapid evidence review process

Key tool128736 as you begin a rapid literature review therersquos no better place to startthan our very own EdTech Hub Evidence Library where our colleagues haveaggregated relevant research in an easy-to-browse database

Key artefact128221 Our Sierra Leone Learning Session Insights (Plaut 2020)provides a summary of important takeaways from three teacher continuousprofessional development (TCPD) experts relevant to implementingtech-enabled TCPD in Sierra Leone These ultimately served as design principlesfor our conceptualisation of the TCPD model that would be tested through asandbox

Ecosystem scanUnderstanding the existing efforts to solve the problem at hand keystakeholders political environment and other contextual factors is crucial tomaking sure that efforts in running a sandbox are not happening in a vacuumbut are directly informed by the ecosystem where the intervention will takeplace

The Sandbox Handbook v10 9

EdTech Hub

The aim is to be able to understand things like

What is the current state of our lsquoproblemrsquo (how does it manifest in society)

Are there existing efforts to challenge or reverse this challenge Who istackling it and how

Who are the key players in this environment working to either resolve thisproblem or (in some cases) contributing to it

What can we learn from previous or current efforts to tackle thischallenge Can we partner with others doing this work Should we

What are the politics around this issue Are there different perspectivesthat need to be taken into account Who currently has power in thissituation

In order to answer these questions therersquos no better way than speaking withrelevant stakeholders directly Whether in person or virtually key informantinterviews (KIIs) focus group discussions (FGDs) workshops and brainstormingsessions can be conducted to ensure that sandbox leads and partners have agood understanding of the ecosystem and are able to operate within it

Having a presence in the relevant context can be a game changer when it comesto better understanding the ecosystem If working in an EdTech Hub prioritycountry collaborating closely with the EdTech Hub country lead is an importantpre-step to undertaking an ecosystem scan

Key artefact 128221 The SL Scoping Phase brief (McBurnie amp Plaut 2021) offers agood example of how insights from the landscaping process (developed throughevidence review and KIIs) can be summarised and shared with sandbox partnersensuring everyone is on the same page about where they are starting from andwho they might want to partner with

14 Bringing all the most relevant people together

Itrsquos clear that an idea on its own wonrsquot fix a problem or improve learningoutcomes We need to find all the most relevant people to unlock impact whichoften means bringing people together from across the sector includinggovernment implementers subject experts and those most impacted by the

The Sandbox Handbook v10 10

EdTech Hub

work (children and teachers) As a sandbox facilitator part of our role is creatingan inclusive environment with the psychological safety to allow for all partners tomake meaningful contributions

As the sandbox develops we recommend bringing together all parts of theeducation system Product (through technology and implementation partners)people and places (users we might want to test interventions with) pedagogicalexpertise (through the Hubrsquos expert pool) provision (through connections tofunders) and policy (through government and policymaker engagement) Weconceptualise this system as the 6 Ps and you can read more about it below

Our theoretical framework for understanding the EdTech systemthe 6 Ps

When technology is introduced into education systems it does not exist inisolation We see lsquoEdTechrsquo as only one part of a solution existing within a broadersystem of factors that need to work together to make impact at scale Werecommend considering the full breadth of the education system and what itwould take to really improve learning outcomes for the most marginalised

In order to consider how an EdTech programme might perform within aneducation system we have developed a framework that considers six key aspectsof the education ecosystem (6 Ps) with which any EdTech tool must engage andintegrate to be successful people product pedagogy policy place and provisionThe 6 Ps are visualised below

The Sandbox Handbook v10 11

EdTech Hub

Figure 1 The 6 Ps framework for understanding the education system

Understanding the needs of our partnersAs early on as possible we surface our partnersrsquo needs so that we know what rolewe need to play and which other partners and experts we might need to bring in

We want to know things like

What is their appetite for iterative ways of working

What capabilities does the team have and what are the gaps

How do they like to work and communicate

Who is the problem owner

What aspects of our sandbox are of most value

What are the priorities What do they want to test through a sandbox

The Sandbox Handbook v10 12

EdTech Hub

Finding an implementing partner(s)The role of partners will vary depending on the work They might take the lead orfill a specific role in the sandbox These partners might be NGOs charities or theprivate sector A diverse team of individuals and organisations will give thesandbox more latitude to try different things and more perspectives on theproblem and potential solutions

Things to consider

Choose the most appropriate way to bring them in mdash this might be anopen call or a closed call marketplace of ideas if you have a few knownpotential partners

Develop the criteria for selecting them based on existing evidence of thattopic and EdTech standards (see Section 13)

Look for some proven track record mdash this might be the impact of theproduct service itself or expertise in supporting implementation

Look for strong experience and knowledge of the given context andplayers

Bringing together a circle of fundersIf needed we also recommend convening a circle of potential funders to be partof a funding circle

We bring together donors private investors and other funders at the start of asandbox to give advice offering de-risked deal flow to invest in a boomingEdTech market Our circle of funders have lsquofirst in linersquo access to the pipeline ofsandboxes

In return they give entrepreneurs and implementers money-canrsquot-buy strategicadvice about what it would take for them to invest and enable them to create asustainable plan from the beginning of the sandbox

This give-get helps ensure funders are engaged early and maximises thelikelihood of financial sustainability for EdTech interventions

The Sandbox Handbook v10 13

EdTech Hub

15 Articulate a vision for impact

Some partners enter into sandboxes with a good understanding of the problemthey want to address but with little idea of how to do it EdTech Hub can help tocatalyse the decision-making process about the desired intervention andultimate impact There are many ways of doing this but one of the mostcomprehensive and participatory is facilitating the development of a theory ofchange (TOC)

A TOC is a tool used to define a programme model outlining the activities thatwill be implemented and their respective outcomes It is also a framework forunderstanding the potential impact of that programme which helps to definehow future pilot activities assessment and iteration will occur

TOCs usually begin by articulating the ultimate impact partners intend tobring about and then working backward to determine whatimplementation model will get us there

They do not have to be overly complex and if done right can begin theprocess of outlining the critical beliefs which will be discussed furtherduring the mobilisation phase and ultimately tested in implementation

A TOC can also be useful in engaging with potential partners funders andother relevant stakeholders who can review it and provide feedback on themodel Sharing a draft TOC and asking for their advice is a good way to getbuy-in and validate your assumptions

Lastly a TOC can be helpful throughout the sandbox updating the TOC asimplementation leads to iterations in the model can help make sure thatthe desired outcomes are still front-and-centre

Some key tips for facilitating a TOC workshop

1 Start with the outcomes yoursquore aiming for begin by facilitating abrainstorm in which partners list all desired outcomes of their interventionThrow everything at the wall sort prioritise Some outcomes may beshort-term effects others long-term aspirations All are welcome Aprioritised list of outcomes can guide future monitoring and evaluationefforts

The Sandbox Handbook v10 14

EdTech Hub

2 Working backwards define the work which will lead to theseoutcomes outcomes will not happen just because we wish them to Theymust be connected to the intervention being implemented List outprogramme components how they tie together and how they link todesired outcomes

3 Add in contextual barriers and facilitators what factors might hinderyour ability to implement the intervention outlined What factors mightcatalyse your impact Everything from potential partnerships to politics toinfrastructure needs can play a role in whether your intervention willactually lead to impact List as many of these as needed and prioritisethem along with your partners If you canrsquot name any it may be importantto go back to your ecosystem scan

4 Identify your assumptions and areas of uncertainty is it safe to assumethat if our intervention is implemented perfectly we are guaranteed tohave the desired impact (Probably nothellip) What assumptions about ourimplementation might need validation Where are the biggest areas ofuncertainty See our mapping of critical beliefs for further support incompleting this exercise

Key artefact 128221 the Sierra Leone draft CPD TOC (Plaut 2021) was createdalongside the Teacher Service Commission and the World Bank early on in thesandbox on continuous teacher professional development in Sierra Leone Thisdeck (which is the summary of a TOC workshop) provides a breakdown of thecore TOC components as well as a draft which was later shared with potentialpartners and key stakeholders

The Sandbox Handbook v10 15

EdTech Hub

Figure 1 The 6 Ps framework for understanding the education system

Understanding the needs of our partnersAs early on as possible we surface our partnersrsquo needs so that we know what rolewe need to play and which other partners and experts we might need to bring in

We want to know things like

What is their appetite for iterative ways of working

What capabilities does the team have and what are the gaps

How do they like to work and communicate

Who is the problem owner

What aspects of our sandbox are of most value

What are the priorities What do they want to test through a sandbox

The Sandbox Handbook v10 12

EdTech Hub

Finding an implementing partner(s)The role of partners will vary depending on the work They might take the lead orfill a specific role in the sandbox These partners might be NGOs charities or theprivate sector A diverse team of individuals and organisations will give thesandbox more latitude to try different things and more perspectives on theproblem and potential solutions

Things to consider

Choose the most appropriate way to bring them in mdash this might be anopen call or a closed call marketplace of ideas if you have a few knownpotential partners

Develop the criteria for selecting them based on existing evidence of thattopic and EdTech standards (see Section 13)

Look for some proven track record mdash this might be the impact of theproduct service itself or expertise in supporting implementation

Look for strong experience and knowledge of the given context andplayers

Bringing together a circle of fundersIf needed we also recommend convening a circle of potential funders to be partof a funding circle

We bring together donors private investors and other funders at the start of asandbox to give advice offering de-risked deal flow to invest in a boomingEdTech market Our circle of funders have lsquofirst in linersquo access to the pipeline ofsandboxes

In return they give entrepreneurs and implementers money-canrsquot-buy strategicadvice about what it would take for them to invest and enable them to create asustainable plan from the beginning of the sandbox

This give-get helps ensure funders are engaged early and maximises thelikelihood of financial sustainability for EdTech interventions

The Sandbox Handbook v10 13

EdTech Hub

15 Articulate a vision for impact

Some partners enter into sandboxes with a good understanding of the problemthey want to address but with little idea of how to do it EdTech Hub can help tocatalyse the decision-making process about the desired intervention andultimate impact There are many ways of doing this but one of the mostcomprehensive and participatory is facilitating the development of a theory ofchange (TOC)

A TOC is a tool used to define a programme model outlining the activities thatwill be implemented and their respective outcomes It is also a framework forunderstanding the potential impact of that programme which helps to definehow future pilot activities assessment and iteration will occur

TOCs usually begin by articulating the ultimate impact partners intend tobring about and then working backward to determine whatimplementation model will get us there

They do not have to be overly complex and if done right can begin theprocess of outlining the critical beliefs which will be discussed furtherduring the mobilisation phase and ultimately tested in implementation

A TOC can also be useful in engaging with potential partners funders andother relevant stakeholders who can review it and provide feedback on themodel Sharing a draft TOC and asking for their advice is a good way to getbuy-in and validate your assumptions

Lastly a TOC can be helpful throughout the sandbox updating the TOC asimplementation leads to iterations in the model can help make sure thatthe desired outcomes are still front-and-centre

Some key tips for facilitating a TOC workshop

1 Start with the outcomes yoursquore aiming for begin by facilitating abrainstorm in which partners list all desired outcomes of their interventionThrow everything at the wall sort prioritise Some outcomes may beshort-term effects others long-term aspirations All are welcome Aprioritised list of outcomes can guide future monitoring and evaluationefforts

The Sandbox Handbook v10 14

EdTech Hub

2 Working backwards define the work which will lead to theseoutcomes outcomes will not happen just because we wish them to Theymust be connected to the intervention being implemented List outprogramme components how they tie together and how they link todesired outcomes

3 Add in contextual barriers and facilitators what factors might hinderyour ability to implement the intervention outlined What factors mightcatalyse your impact Everything from potential partnerships to politics toinfrastructure needs can play a role in whether your intervention willactually lead to impact List as many of these as needed and prioritisethem along with your partners If you canrsquot name any it may be importantto go back to your ecosystem scan

4 Identify your assumptions and areas of uncertainty is it safe to assumethat if our intervention is implemented perfectly we are guaranteed tohave the desired impact (Probably nothellip) What assumptions about ourimplementation might need validation Where are the biggest areas ofuncertainty See our mapping of critical beliefs for further support incompleting this exercise

Key artefact 128221 the Sierra Leone draft CPD TOC (Plaut 2021) was createdalongside the Teacher Service Commission and the World Bank early on in thesandbox on continuous teacher professional development in Sierra Leone Thisdeck (which is the summary of a TOC workshop) provides a breakdown of thecore TOC components as well as a draft which was later shared with potentialpartners and key stakeholders

The Sandbox Handbook v10 15

EdTech Hub

Finding an implementing partner(s)The role of partners will vary depending on the work They might take the lead orfill a specific role in the sandbox These partners might be NGOs charities or theprivate sector A diverse team of individuals and organisations will give thesandbox more latitude to try different things and more perspectives on theproblem and potential solutions

Things to consider

Choose the most appropriate way to bring them in mdash this might be anopen call or a closed call marketplace of ideas if you have a few knownpotential partners

Develop the criteria for selecting them based on existing evidence of thattopic and EdTech standards (see Section 13)

Look for some proven track record mdash this might be the impact of theproduct service itself or expertise in supporting implementation

Look for strong experience and knowledge of the given context andplayers

Bringing together a circle of fundersIf needed we also recommend convening a circle of potential funders to be partof a funding circle

We bring together donors private investors and other funders at the start of asandbox to give advice offering de-risked deal flow to invest in a boomingEdTech market Our circle of funders have lsquofirst in linersquo access to the pipeline ofsandboxes

In return they give entrepreneurs and implementers money-canrsquot-buy strategicadvice about what it would take for them to invest and enable them to create asustainable plan from the beginning of the sandbox

This give-get helps ensure funders are engaged early and maximises thelikelihood of financial sustainability for EdTech interventions

The Sandbox Handbook v10 13

EdTech Hub

15 Articulate a vision for impact

Some partners enter into sandboxes with a good understanding of the problemthey want to address but with little idea of how to do it EdTech Hub can help tocatalyse the decision-making process about the desired intervention andultimate impact There are many ways of doing this but one of the mostcomprehensive and participatory is facilitating the development of a theory ofchange (TOC)

A TOC is a tool used to define a programme model outlining the activities thatwill be implemented and their respective outcomes It is also a framework forunderstanding the potential impact of that programme which helps to definehow future pilot activities assessment and iteration will occur

TOCs usually begin by articulating the ultimate impact partners intend tobring about and then working backward to determine whatimplementation model will get us there

They do not have to be overly complex and if done right can begin theprocess of outlining the critical beliefs which will be discussed furtherduring the mobilisation phase and ultimately tested in implementation

A TOC can also be useful in engaging with potential partners funders andother relevant stakeholders who can review it and provide feedback on themodel Sharing a draft TOC and asking for their advice is a good way to getbuy-in and validate your assumptions

Lastly a TOC can be helpful throughout the sandbox updating the TOC asimplementation leads to iterations in the model can help make sure thatthe desired outcomes are still front-and-centre

Some key tips for facilitating a TOC workshop

1 Start with the outcomes yoursquore aiming for begin by facilitating abrainstorm in which partners list all desired outcomes of their interventionThrow everything at the wall sort prioritise Some outcomes may beshort-term effects others long-term aspirations All are welcome Aprioritised list of outcomes can guide future monitoring and evaluationefforts

The Sandbox Handbook v10 14

EdTech Hub

2 Working backwards define the work which will lead to theseoutcomes outcomes will not happen just because we wish them to Theymust be connected to the intervention being implemented List outprogramme components how they tie together and how they link todesired outcomes

3 Add in contextual barriers and facilitators what factors might hinderyour ability to implement the intervention outlined What factors mightcatalyse your impact Everything from potential partnerships to politics toinfrastructure needs can play a role in whether your intervention willactually lead to impact List as many of these as needed and prioritisethem along with your partners If you canrsquot name any it may be importantto go back to your ecosystem scan

4 Identify your assumptions and areas of uncertainty is it safe to assumethat if our intervention is implemented perfectly we are guaranteed tohave the desired impact (Probably nothellip) What assumptions about ourimplementation might need validation Where are the biggest areas ofuncertainty See our mapping of critical beliefs for further support incompleting this exercise

Key artefact 128221 the Sierra Leone draft CPD TOC (Plaut 2021) was createdalongside the Teacher Service Commission and the World Bank early on in thesandbox on continuous teacher professional development in Sierra Leone Thisdeck (which is the summary of a TOC workshop) provides a breakdown of thecore TOC components as well as a draft which was later shared with potentialpartners and key stakeholders

The Sandbox Handbook v10 15

EdTech Hub

15 Articulate a vision for impact

Some partners enter into sandboxes with a good understanding of the problemthey want to address but with little idea of how to do it EdTech Hub can help tocatalyse the decision-making process about the desired intervention andultimate impact There are many ways of doing this but one of the mostcomprehensive and participatory is facilitating the development of a theory ofchange (TOC)

A TOC is a tool used to define a programme model outlining the activities thatwill be implemented and their respective outcomes It is also a framework forunderstanding the potential impact of that programme which helps to definehow future pilot activities assessment and iteration will occur

TOCs usually begin by articulating the ultimate impact partners intend tobring about and then working backward to determine whatimplementation model will get us there

They do not have to be overly complex and if done right can begin theprocess of outlining the critical beliefs which will be discussed furtherduring the mobilisation phase and ultimately tested in implementation

A TOC can also be useful in engaging with potential partners funders andother relevant stakeholders who can review it and provide feedback on themodel Sharing a draft TOC and asking for their advice is a good way to getbuy-in and validate your assumptions

Lastly a TOC can be helpful throughout the sandbox updating the TOC asimplementation leads to iterations in the model can help make sure thatthe desired outcomes are still front-and-centre

Some key tips for facilitating a TOC workshop

1 Start with the outcomes yoursquore aiming for begin by facilitating abrainstorm in which partners list all desired outcomes of their interventionThrow everything at the wall sort prioritise Some outcomes may beshort-term effects others long-term aspirations All are welcome Aprioritised list of outcomes can guide future monitoring and evaluationefforts

The Sandbox Handbook v10 14

EdTech Hub

2 Working backwards define the work which will lead to theseoutcomes outcomes will not happen just because we wish them to Theymust be connected to the intervention being implemented List outprogramme components how they tie together and how they link todesired outcomes

3 Add in contextual barriers and facilitators what factors might hinderyour ability to implement the intervention outlined What factors mightcatalyse your impact Everything from potential partnerships to politics toinfrastructure needs can play a role in whether your intervention willactually lead to impact List as many of these as needed and prioritisethem along with your partners If you canrsquot name any it may be importantto go back to your ecosystem scan

4 Identify your assumptions and areas of uncertainty is it safe to assumethat if our intervention is implemented perfectly we are guaranteed tohave the desired impact (Probably nothellip) What assumptions about ourimplementation might need validation Where are the biggest areas ofuncertainty See our mapping of critical beliefs for further support incompleting this exercise

Key artefact 128221 the Sierra Leone draft CPD TOC (Plaut 2021) was createdalongside the Teacher Service Commission and the World Bank early on in thesandbox on continuous teacher professional development in Sierra Leone Thisdeck (which is the summary of a TOC workshop) provides a breakdown of thecore TOC components as well as a draft which was later shared with potentialpartners and key stakeholders

The Sandbox Handbook v10 15

EdTech Hub

2 Working backwards define the work which will lead to theseoutcomes outcomes will not happen just because we wish them to Theymust be connected to the intervention being implemented List outprogramme components how they tie together and how they link todesired outcomes

3 Add in contextual barriers and facilitators what factors might hinderyour ability to implement the intervention outlined What factors mightcatalyse your impact Everything from potential partnerships to politics toinfrastructure needs can play a role in whether your intervention willactually lead to impact List as many of these as needed and prioritisethem along with your partners If you canrsquot name any it may be importantto go back to your ecosystem scan

4 Identify your assumptions and areas of uncertainty is it safe to assumethat if our intervention is implemented perfectly we are guaranteed tohave the desired impact (Probably nothellip) What assumptions about ourimplementation might need validation Where are the biggest areas ofuncertainty See our mapping of critical beliefs for further support incompleting this exercise

Key artefact 128221 the Sierra Leone draft CPD TOC (Plaut 2021) was createdalongside the Teacher Service Commission and the World Bank early on in thesandbox on continuous teacher professional development in Sierra Leone Thisdeck (which is the summary of a TOC workshop) provides a breakdown of thecore TOC components as well as a draft which was later shared with potentialpartners and key stakeholders

The Sandbox Handbook v10 15

EdTech Hub

2 Setting out a plan to solve the problem

21 Sandbox onboarding

Guidance on ways of working

Before starting the lsquoget to workrsquo sessions there are some valuable lsquoonboardingrsquoactivities and conversations to have with sandbox teams These can be done inmeetings or by email based on whatrsquos best for the sandbox We recommendhaving these conversations early to align the team which may becross-organisation or cross-geography

Itrsquos good to agree

A weekly check-in time (this can become less regular as the sandboxprogresses)

An agreed upon platform for messaging (eg WhatsApp Slack MicrosoftTeams)

An agreed set of tools you want to use to collaborate together Sandboxteams have told us that they can be overwhelmed by the number of newtools so agreeing which tools (Google Suite Miro etc) up front isimportant

Working in sprintsWe break sandboxes up into short sprints learning and iterating as we go Eachsprint informs changes and new ideas for the next As such sprints are thebuilding blocks of implementation within a sandbox

A sprint is a fixed period of time within which we do low-cost quick activities tolearn about whether our idea will work Typically a sprint lasts between 6 and12weeks with around two weeks between sprints to ensure space to reflect andplan the next one Below yoursquoll learn a lot more about how we plan execute andreflect on sprints

Key tool128736 Welcome to the Sandbox mdash sprints (Slide 9) (Simpson et al 2021b)provides a more detailed introduction to what a sprint is

The Sandbox Handbook v10 16

EdTech Hub

22 lsquoGet to workrsquo sessions

After we have chosen to work with a sandbox the lsquoget to workrsquo sessions helpmove from idea to action plan Note these sessions donrsquot all have to be done inperson mdash some can be done as lsquohomeworkrsquo Sessions can also be combined intoone longer session

There might also be introductory meetings before the lsquoget to workrsquo session forthe sandbox team to get to know each other

lsquoGet to workrsquo Session 1 From idea to hypothesisKey artefact128221 Welcome to the Sandbox (Simpson et al 2021) provides anintroduction to how the sandbox works in practice It outlines the sandboxjourney and key concepts like the 6 Ps and sprints

Key tool128736 Welcome to the Sandbox mdash the idea (Slide 3) (Simpson et al 2021) isa good first exercise to help the team refine the idea into a sandbox hypothesis

The hypothesis forms the boundary (or scope) for what we will test in thesandbox

The Sandbox Handbook v10 17

EdTech Hub

DescriptionEncourage the group to think through thehypothesis using this structure if a particularEdTech intervention was implemented thena particular outcome would happen so thatwe achieve some positive impact

Write this down on the slide and ensure thegroup is aligned around it Note down widerthoughts around the hypothesis in theNOTES and encourage the group to typeinto the slide too

This tool is a good opportunity to involve awider group in particular funders or otherpeople who can make the sandbox lsquohappenrsquoand be sustainable Inviting them tocomment on the hypothesis unlocks newthinking and helps the team understandwhat it might take for key players to supportus

If a TOC has been developed during thescoping phase it can serve as a usefulreference throughout the mobilisation phasebut especially in developing a hypothesis andcritical beliefs

Mechanisms and mindsets

Bring ambition Be ambitious on behalf of thesandbox Particularly in the lsquoso thathelliprsquo part of thehypothesis We want to make a dent in theglobal learning crisis

Test the comfort with pivoting In order toachieve the lsquoso thathelliprsquo we should be willing topivot the EdTech intervention This iteration isan important part of the sandbox philosophyso establish how comfortable the team is withpivoting and identify any lsquonon-negotiablesrsquoearly

Get used to Google Suite collaboration Usethe exercise as a way to encourage the team towrite into the slide in the session This is animportant part of our ways of working You mayneed to nudge them to ensure they join thesession from a laptop

Encourage empathy with and genuinecuriosity in the end user of the interventionAs in the lsquounderstanding the people behind thedatarsquo session the fundamental idea is toremind the sandbox participants that lsquothey arenot their userrsquo mdash are they making assumptionsabout the usersrsquo needs context and behaviourbased on their own experience not the users

lsquoGet to workrsquo Session 2 Auditing the hypothesis in theeducation system using the 6 Ps

Key tool128736 Project Pack mdash Sandbox Plan (tab lsquo6Ps Auditrsquo) helps lay thegroundwork for what the sandbox should focus on

DescriptionUsing the 6Ps Audit Tool in the Project Pack thissession is about taking the time to both

1 Identify the areas of most uncertainty Itdirectly feeds into the 3rd lsquoGet to Workrsquo session

2 Fully understand all elements of the EdTech

Mechanisms and mindsets

This is not an evaluation Remindthem that this session isnrsquot about usevaluating them or passingjudgement Instead itrsquos a way for us toflesh out the detail mdash what they havealready done what they already know

The Sandbox Handbook v10 18

EdTech Hub

intervention whether it is an idea or real-lifething This understanding then helps theEdTech Hub team be of greater value

Crucially using the 6Ps framework ensures that theteam focuses on all elements of the education systemthat are needed to make the intervention successfuland doesnrsquot avoid the difficult parts

Prior to the session send the 6 Ps framework to thesandbox team to give them a chance to read itMention that it is for reading and not to score (yet)

3-step process

1 Quickly go through each lsquoPrsquo and ask all whichlevel they believe the EdTech intervention iscurrently at

2 Take time to discuss each P individuallyrecognising that there will be overlap betweenthem Ask the team to share what theyrsquove doneso far and what the biggest uncertainty is andencourage them to think beyond whatrsquosimmediately possible Begin with the Ps whichhad the greatest divergence of answers orwhich scored lowest (ie 1 or 2)

Good questions to ask

Where is the uncertainty around the PWhat are you least sure about

What did you do to get to level 234(takes focus away from lsquoassumedknowledgersquo or other assumptions)

3 Take notes of everything that is discussed inthe notes tab with the ultimate aim ofagreeing a final level for the interventionThese notes will be invaluable to refer back toduring the sprints phase of the sandbox

and where there are areas still toexplore

Make it fun Exercises like holdingfingers up to the camera (in Step 1) orencouraging the team to tell storiesabout the lsquoPsrsquo help thecross-organisational team feel like one

Give people time to warm up Bystarting with some time to read andgive a quick-fire ranking you givepeople the time to get comfortablewith the 6Ps and what they meanYoursquoll also be able to spot where thereare big discrepancies in opinion andyou might want to start the discussionwith those to uncover the reasonsbehind the difference of opinion

Exact wording not always applicableAs the 6 Ps Audit Tool is applied acrossall sandboxes sometimes the wordingwill not be relevant to a particularsandbox In that case focus less on theboxes and more on the key questionsAnd feel free to improvise on the levelsif needs be

Overlap between categoriesSometimes lsquoPsrsquo overlap For exampleplace and product or people andplace or pedagogy and people Donrsquotsweat mdash itrsquos the conversation thatrsquosimportant

The Sandbox Handbook v10 19

EdTech Hub

lsquoGet to workrsquo Session 3 Mapping risks using thepre-mortem

Key tool128736 Welcome to the Sandbox mdash the pre-mortem (Slides 11ndash12) (Simpsonet al 2021) is an optional tool to support teams in arriving at critical beliefs Allmembers of a team imagine the future end of the sandbox and share what hasgone really well (utopia) and what has gone really badly (dystopia)

It is a validated well-known exercise (Atlassian 2020) Statements from thepre-mortem can then be turned into critical beliefs

Description

1 Introduce the exercise this exercise will helpus visualise goals by imagining what successand failure look like and what we believeneeds to be true to achieve success and avoidfailure

2 Imagine utopia Ask the team to close theireyes and imagine the end of the sandbox Itrsquosbeen a roaring success Why Ask eachperson to speak about this Write what theysay in the lsquoUtopiarsquo column of the tool

3 Imagine dystopia Ask the team to close theireyes and imagine the end of the sandbox Itrsquosbeen a terrible failure Why Again ask eachperson to speak about this and write whatthey say in the lsquoDystopiarsquo column of the toolThis might be quicker as many of the themeswill have emerged from the lsquoUtopiarsquo columnalready

4 Sense-check against the 6 Ps Check if anyof the 6 Ps are notably absent from the utopia dystopia scenarios Challenge the team if so

5 Give an example of turning a pre-mortemstatement into a critical belief Within eachstatement unpick the belief that means wewill achieve it (utopia) or avoid it (dystopia)For example a dystopia statement might belsquochildrenrsquos excitement tapered off after twoweeksrsquo The critical belief might read children

Mechanisms and mindsets

Exercise deep democracy Differentpeople pick up on different thingsbased on their vantage point Thatrsquoswhy deep democracy mdash making sureevery voice is heard mdash is particularlyimportant in a pre-mortem

One useful tactic is to begin withpeople who might be reluctant tospeak up and end with senior decisionmakers or stakeholders (who cansummarise and distil what they haveheard)

Visualise the future Think as if thefuture has occurred Researchsuggests that imagining that an eventhas already occurred increases theability to correctly identify reasons forfuture outcomes by 30

One useful tactic is to pausedeliberately and get participants toclose their eyes really imaginingthemselves to be in the future Anotheris to use the past tense as if you weredoing a post-mortem For example asklsquowhat went wellrsquo rather than lsquowhat willgo wellrsquo and encourage people to saylsquoteachers knew how to use the productrsquorather than lsquoteachers will know how touse the productrsquo

The Sandbox Handbook v10 20

EdTech Hub

lsquoGet to workrsquo Session 3 Mapping risks using thepre-mortem

Key tool128736 Welcome to the Sandbox mdash the pre-mortem (Slides 11ndash12) (Simpsonet al 2021) is an optional tool to support teams in arriving at critical beliefs Allmembers of a team imagine the future end of the sandbox and share what hasgone really well (utopia) and what has gone really badly (dystopia)

It is a validated well-known exercise (Atlassian 2020) Statements from thepre-mortem can then be turned into critical beliefs

Description

1 Introduce the exercise this exercise will helpus visualise goals by imagining what successand failure look like and what we believeneeds to be true to achieve success and avoidfailure

2 Imagine utopia Ask the team to close theireyes and imagine the end of the sandbox Itrsquosbeen a roaring success Why Ask eachperson to speak about this Write what theysay in the lsquoUtopiarsquo column of the tool

3 Imagine dystopia Ask the team to close theireyes and imagine the end of the sandbox Itrsquosbeen a terrible failure Why Again ask eachperson to speak about this and write whatthey say in the lsquoDystopiarsquo column of the toolThis might be quicker as many of the themeswill have emerged from the lsquoUtopiarsquo columnalready

4 Sense-check against the 6 Ps Check if anyof the 6 Ps are notably absent from the utopia dystopia scenarios Challenge the team if so

5 Give an example of turning a pre-mortemstatement into a critical belief Within eachstatement unpick the belief that means wewill achieve it (utopia) or avoid it (dystopia)For example a dystopia statement might belsquochildrenrsquos excitement tapered off after twoweeksrsquo The critical belief might read children

Mechanisms and mindsets

Exercise deep democracy Differentpeople pick up on different thingsbased on their vantage point Thatrsquoswhy deep democracy mdash making sureevery voice is heard mdash is particularlyimportant in a pre-mortem

One useful tactic is to begin withpeople who might be reluctant tospeak up and end with senior decisionmakers or stakeholders (who cansummarise and distil what they haveheard)

Visualise the future Think as if thefuture has occurred Researchsuggests that imagining that an eventhas already occurred increases theability to correctly identify reasons forfuture outcomes by 30

One useful tactic is to pausedeliberately and get participants toclose their eyes really imaginingthemselves to be in the future Anotheris to use the past tense as if you weredoing a post-mortem For example asklsquowhat went wellrsquo rather than lsquowhat willgo wellrsquo and encourage people to saylsquoteachers knew how to use the productrsquorather than lsquoteachers will know how touse the productrsquo

The Sandbox Handbook v10 20

EdTech Hub

stay engaged after the initial two-week periodthroughout the curriculardquo

You can either turn each statement into a criticalbelief or encourage the team to do so after themeeting

lsquoGet to workrsquo Session 4 From hypothesis to action planKey tool128736 Project Pack mdash Sandbox Plan (tab lsquoSandbox Planrsquo) (Rahman ampCarter 2020) is a core exercise to move from the hypothesis to critical beliefs toan action plan It consists of six steps which are core in identifying andstructuring the experiments that are central to the sandbox methodologyExamples of the steps are in the project pack

Key artefact128221 Experiment Types (coming soon) Some inspiration for types ofexperiments with pros cons and examples

Description7-step method

1 Beliefs in hypothesis Ask the team towrite things they believe need to be true forthe hypothesis to work Focus on thelowest-scoring areas in the 6 Ps Audit

2 Critical beliefs Then ask the team to mark(with an X) beliefs that are both mostimportant to the hypothesis and that weknow least about

3 Plan activities Think about activities youcould do to get data on the beliefs tounderstand whether they are true or notThe activities should generate robustevidence but also be quick and low-costWrite a narrative description of the activityin the plan

4 Sequence activities Mark which activitieswe will do first (in Sprint 1) next (in Sprint 2)or later Focus on doing activities that getthe most data on the most critical beliefs

Mechanisms and mindsets

Use stories and examples Using storiescan help unlock creative thinking incoming up with low-cost quick activitiesthat generate robust evidence (Step 3)You can read more of our favourite storiesin the Experiment Types artefact

Learn by doing Focus on learningthrough doing and encourage the teamnot to overthink in going through theseven steps In parallel keep yourexplanations of the tool short andjargon-free

Bring collaboration The sandbox teamare both facilitators and collaborators Theteam value your experience inexperimentation in EdTech mdash bring ideasand strategies to the work

Ensure the team thinks through thewhole system Often people will jumpstraight to the beliefs around thetechnology Nudge them to the 6 Ps and

The Sandbox Handbook v10 21

EdTech Hub

first Use this opportunity to agree therough length of sprints (eg 3ndash6 weeks)

5 Agree evidence output Think about howthe evidence generated from doing theactivity will be captured For some activitiesthis will be in the Sprint Review document(see Section 2) Other activities might haveother standalone outputs such as video audio feedback or survey data

6 Agree minimum proof For some activitiesit might be relevant to identify a minimumproof(s) to help us lsquoknowrsquo whether a criticalbelief has been validated Other activitieswill be more exploratory and therefore aminimum proof might not be relevant

7 Decide if there is a storytelling outputWhere activities involve co-designing ordoing qualitative research with the users ofthe intervention plan how yoursquoll not justshare the data but represent theexperiences and voices of peopleauthentically and fully eg in a film audioor multimedia output

explain that thinking holistically about thesystem will make sure implementation issound and the intervention is able to growand scale

Focus on velocity not speed Sprints area structure to organise activities andensure clear points of reflection andcourse correction Itrsquos not just aboutmoving quickly (as the word sprintsuggests) but moving deliberately andwith purpose towards a scalableintervention (ie with velocity)

Exercise deep democracy If possible andappropriate exercise deep democracy Allpeople in the room have an equal voiceFor example everyone has an equalnumber of votes on critical beliefs

Finalising the sprint planTimings of sprints The rhythm of the lsquosprintrsquo phase is typically one-monthsprints with a gap of two weeks between sprints to review the sprint findings(Section 31) and iterate the next sprint (Section 32) However this can be adaptedbased on the particular sprint For example if the bundle of activities agreedtakes longer then the sprint can be longer However we recommend a sprint isno longer than three months so that there can be a feedback loop and we retaintime and or funding to iterate Even if an activity takes longer than threemonths we can simply choose to continue it in the subsequent sprint as long aswe still believe it is the most effective use of our resources

Holding future plans loosely The level of fidelity expected for the sprintactivities decreases over time for example the first three months might be quiteclearly defined and the later months might be more vague

The Sandbox Handbook v10 22

EdTech Hub

lsquoGet to workrsquo Session 5 Storytelling mdash understandingthe people behind the data

Key artefact128221 Storytelling Handbook (Dixon amp Patuck 2021) A comprehensiveintroduction to storytelling on sandboxes

There are two approaches to understanding the people behind the datadependent on how the data is being collected for that sprint

Approach 1 Qualitative research is already woven into the sprint planfrom the outset mdash get the team to think creatively about how they canshare what they find as stories (elevating the voice of people mostimpacted by the work)

Approach 2 In the case of there being a lack of qualitative data mdash host adedicated session to identify a lsquostorytellingrsquo strand to the work which canhelp fill this gap

Key tool128736 Storytelling Planning Template On Jamboard | On a GoogleDoc | Ona Google Slides Presentation (Dixon amp Patuck 2020) The template used istailored to the particular partnerrsquos context and preference

Description

A process for ensuring that sandboxes are taking the time tounderstand the context and lives of those they are designing forwith curiosity and empathy by putting their voices and livedexperience front and centre We call it storytelling to emphasisethe creative and communicative outputs of this work

1 Plan the focus Decide the lsquowhorsquo Who are the voicesthat are least heard or involved in this design process(hint it tends to be the end user) Write an explorationquestion that will help you to understand this user groupmore deeply in relation to the problem the sandbox istrying to solve using EdTech Think through the potentialrisks and power dynamics involved in this interaction

2 Plan the engagement Using a range of examples andideas help the sandbox think through the mostappropriate method for engaging and capturing thevoices of this group

Mechanisms and mindsets

Contextual understandingover testimonials ormarketing We know thatvideos or words from end userscan be great for marketing butthis isnrsquot the aim here the aimis greater understanding ofpeople

Depth over breadthEncourage your sandbox tonarrow their focus and scope ofparticipants to allow for deeperunderstanding It only takesinterviews with five people tostart to see patterns

Meet people where they areEncourage methods and toolsthat are best for participants

The Sandbox Handbook v10 23

EdTech Hub

lsquoGet to workrsquo Session 5 Storytelling mdash understandingthe people behind the data

Key artefact128221 Storytelling Handbook (Dixon amp Patuck 2021) A comprehensiveintroduction to storytelling on sandboxes

There are two approaches to understanding the people behind the datadependent on how the data is being collected for that sprint

Approach 1 Qualitative research is already woven into the sprint planfrom the outset mdash get the team to think creatively about how they canshare what they find as stories (elevating the voice of people mostimpacted by the work)

Approach 2 In the case of there being a lack of qualitative data mdash host adedicated session to identify a lsquostorytellingrsquo strand to the work which canhelp fill this gap

Key tool128736 Storytelling Planning Template On Jamboard | On a GoogleDoc | Ona Google Slides Presentation (Dixon amp Patuck 2020) The template used istailored to the particular partnerrsquos context and preference

Description

A process for ensuring that sandboxes are taking the time tounderstand the context and lives of those they are designing forwith curiosity and empathy by putting their voices and livedexperience front and centre We call it storytelling to emphasisethe creative and communicative outputs of this work

1 Plan the focus Decide the lsquowhorsquo Who are the voicesthat are least heard or involved in this design process(hint it tends to be the end user) Write an explorationquestion that will help you to understand this user groupmore deeply in relation to the problem the sandbox istrying to solve using EdTech Think through the potentialrisks and power dynamics involved in this interaction

2 Plan the engagement Using a range of examples andideas help the sandbox think through the mostappropriate method for engaging and capturing thevoices of this group

Mechanisms and mindsets

Contextual understandingover testimonials ormarketing We know thatvideos or words from end userscan be great for marketing butthis isnrsquot the aim here the aimis greater understanding ofpeople

Depth over breadthEncourage your sandbox tonarrow their focus and scope ofparticipants to allow for deeperunderstanding It only takesinterviews with five people tostart to see patterns

Meet people where they areEncourage methods and toolsthat are best for participants

The Sandbox Handbook v10 23

EdTech Hub

3 Ensure safety and inclusion Walk through howstory-gatherers will stay safe and gather consent ethicallyfrom those they are engaging with

4 Plan next steps Create the to-do list that will help thesandbox gain clarity on who else needs to be involvedand what needs to happen next

which may be less convenientfor the story collector butultimately more successful andinclusive

23 Other tools you might consider

After the lsquoget to workrsquo sessions you might identify other areas the team couldfocus on If the team is up for it you can work through these areas using toolssuch as those listed below Unlike the tools above which are developed byEdTech Hub these are open source and developed by other organisations

Navigating a complex stakeholder environment mdash try a people andconnections map (NESTA UK 2014)

Thinking big about the potential impact if the hypothesis was proven truemdash try an aspirational press release (Google 2020)

Thinking through the ethics and participation of people in their work mdash trysome exercises in this ethics toolkit (Common Good et al 2020)

Understanding users and their needs mdash if you have stakeholders in theroom who know the user compile this knowledge into an empathy map Ifthey donrsquot you may need to do some user research before doing somepersona building or journey mapping

For more inspiration we recommend IDEOorgrsquos Design Kit (IDEOorg 2015)(especially for tools in human-centred design) Atlassianrsquos Playbook (Atlassian2020) (especially for tools in lean and agile) and NESTArsquos DIY Toolkit(ltURL_CHANGED_LINKgtNESTA UK 2014ltURL_CHANGED_LINKgt) (especially fortools in social innovation)

The Sandbox Handbook v10 24

EdTech Hub

3 Iterative implementation

31 Reflecting on the work

During sprints the Sprint Review is the most important touchpoint in thesandbox journey The Sprint Review has two objectives

To reflect on what has been learnt so far and how it might affect what wedo next

To synthesise evidence in a Sprint Review document a key evidenceoutput from sandboxes for EdTech Hub

Key tool128736 Sprint Review (Rahman 2020c) is our tool for reflecting on what hastaken place in a sprint capturing the evidence and deciding what to do next

Preparation for the Sprint ReviewThere are two viable approaches to preparing for a Sprint Review

Approach 1 Share the Sprint Review tool with those who collected dataor were on the lsquofront linersquo of implementation during the sandbox Ask themto complete questions 1ndash3 (what was the plan what did we do what didwe learn) and share prior to the review This gives everyone in the team achance to get up to speed with what has happened and key data evidence learnt during the sprint Encourage the team to be concise in theSprint Review (a total of two pages is a good rule of thumb)

Approach 2 Ask those who collected data or were on the lsquofront linersquo ofimplementation during the sandbox to share any raw (or near raw) datawith the whole team prior to the review Questions 1ndash3 can be discussedand agreed during the meeting given everyone has had a chance to lookat the data or insights in advance

Which approach to choose depends largely on

The teamrsquos capacity and capabilities Some sandbox partners have thecapacity and expertise to analyse data and will do justice to the evidencein the Sprint Review template In this case approach 1 might be best In

The Sandbox Handbook v10 25

EdTech Hub

other cases others on the team might need to get stuck into the analysisof the data which can be a huge value add for the team

The Sprint If a sprint has not had too much data generated or thesandbox partner has already seen and reflected on the data then approach1 might be best If the sprint has generated a great deal of juicycontroversial data then approach 2 might be best

The Sprint Review

DescriptionThe Sprint Review is best done as a 90-minute sessionwith the sandbox team

The session is led by the sandbox lead with the CriticalFriend acting as Scribe and co-facilitator

It consists of two parts

1 Reviewing the sprint (60 mins) If the SprintReview has been completed prior to the session(approach 1 above)

Ask the sandbox team to present q1ndash3 ofthe review Ask powerful questions (seebelow for our favourites)

If the team have only shared raw (or near-raw) data(approach 2 above)

For each activity in the sprint allow eachperson involved to speak for one minuteuninterrupted focusing on q 1ndash3 in theSprint Review and especially the questionlsquoWhat did we Learnrsquo

After each person has spoken pick up onthemes and encourage further deeperreflection as a group

Powerful questions to unlock learning

What did you learn that you didnt knowbefore

What surprised you

Mechanisms and mindsets

Set the date well in advance Setting adate in advance not only increases thechances of everyone attending thislonger session but also emphasises itsimportance as a clear marker forreflection and iteration

Real not claimed evidence Whenasking lsquowhat did you learnrsquo focus ontangible behaviours (eg usage of aproduct) rather than claimed evidence(eg feedback from a survey)

Use the minimum proof as guidancerather than hard fact Minimum proofsare a great tool for understandingwhether we have validated criticalbeliefs It is ideal to come up with theseproofs before a sprint but it can also bea valuable exercise during a sprint Inthe Sprint Review link proofs to thebeliefs to understand whethersomething you believed to be true isvalid or not But reflect also on theminimum proof mdash was it correct basedon what we know now

Keep up the energy This can be a longsession mdash so do what you know worksto keep energy up A break betweenreviewing the sprint and reviewing thecritical beliefs is an option

Reflection is as important as whatrsquos

The Sandbox Handbook v10 26

EdTech Hub

How is this different from our minimumproof

How is it the same What do we have to back up this

learning How do you know What have we learnt about our user

Try to focus on unexpected things (this is why theminimum proof can be a powerful tool when planningactivities) and back up claims with evidence from egactual behaviours of users

2 Reviewing the critical beliefs (30 mins) Runthrough each of the most critical beliefs (markedwith an X) Quickly decide whether that belief

Has been validated or invalidated Remains critical or is now less critical

The minimum proofs will give us a sharperunderstanding of whether a critical belief has beenvalidated

Use this as a basis to iterate the next sprint (see Section32)

next Sometimes teams will be eager tojump straight to lsquoHow does this affectwhat we do nextrsquo (q 4 in the SprintReview) However we want to nudgeteams to reflect deeply on whatrsquoshappened so that we can make surewhat we do next is driven by the dataThatrsquos why the Sprint Review focuseson whatrsquos happened and the criticalbeliefs and then we shift to focusing oniterating the next sprint

Seeing is believing Invite all keystakeholders to the Sprint Review orconsider hosting a dedicated Showcaseto share the findings Being a part ofthe process will increase the likelihoodthat they are interested in and share oract on the insights as they emerge

32 Iterating the work

Following lsquoldquoGet to workrdquo Session 3 from hypothesis to action planrsquo we shouldhave a sequence of activities in sprints This is a good foundation but as weexecute activities and gather evidence we might take away or amend activitiesor add new ones

Iteration is a core element of the sandbox Although we set out a plan our tools andprocesses enable teams to easily change what we are doing based on new information

Iterating the planKey tool128736 Project Pack mdash Sandbox Plan (tab lsquoSandbox Managementrsquo)(Rahman amp Carter 2020) is our tool for capturing new or amended activities

The Sandbox Handbook v10 27

EdTech Hub

How is this different from our minimumproof

How is it the same What do we have to back up this

learning How do you know What have we learnt about our user

Try to focus on unexpected things (this is why theminimum proof can be a powerful tool when planningactivities) and back up claims with evidence from egactual behaviours of users

2 Reviewing the critical beliefs (30 mins) Runthrough each of the most critical beliefs (markedwith an X) Quickly decide whether that belief

Has been validated or invalidated Remains critical or is now less critical

The minimum proofs will give us a sharperunderstanding of whether a critical belief has beenvalidated

Use this as a basis to iterate the next sprint (see Section32)

next Sometimes teams will be eager tojump straight to lsquoHow does this affectwhat we do nextrsquo (q 4 in the SprintReview) However we want to nudgeteams to reflect deeply on whatrsquoshappened so that we can make surewhat we do next is driven by the dataThatrsquos why the Sprint Review focuseson whatrsquos happened and the criticalbeliefs and then we shift to focusing oniterating the next sprint

Seeing is believing Invite all keystakeholders to the Sprint Review orconsider hosting a dedicated Showcaseto share the findings Being a part ofthe process will increase the likelihoodthat they are interested in and share oract on the insights as they emerge

32 Iterating the work

Following lsquoldquoGet to workrdquo Session 3 from hypothesis to action planrsquo we shouldhave a sequence of activities in sprints This is a good foundation but as weexecute activities and gather evidence we might take away or amend activitiesor add new ones

Iteration is a core element of the sandbox Although we set out a plan our tools andprocesses enable teams to easily change what we are doing based on new information

Iterating the planKey tool128736 Project Pack mdash Sandbox Plan (tab lsquoSandbox Managementrsquo)(Rahman amp Carter 2020) is our tool for capturing new or amended activities

The Sandbox Handbook v10 27

EdTech Hub

DescriptionAfter lsquoldquoGet to workrdquo Session 3 fromhypothesis to action planrsquo copy over theactivities into this tab

At the end of each sprint reflect on thecurrent plan for the next sprint with theteam Based on what we know that we didnrsquotbefore

1 Which activities should stay the same2 Which activities should we remove or

move to later sprints3 Which activities should we add4 Which activities should we amend

(double down change dial back)

Depending on the situation it might berelevant to run some ideation here Forexample teams could draw or write as manynew ideas for new or amended activities asthey can think about and then vote orconverge on the best Remember we are notwedded to anything we might have planned

The Sprint Review document and the criticalbeliefs should form the basis for this

Following this work with the team to

1 Amend the plan for the upcomingsprint in the Sandbox Managementtab

2 Discuss any budgetary implicationsthis may have and make sure theseare reflected in the budget

All of this can be done either in a dedicatedsession or asynchronously over the GoogleSheet mdash depending on your teamrsquos preferredworking style

Mechanisms and mindsets

Everything is up for grabs One risk ofplanning all activities in advance is that a teambecomes lsquoweddedrsquo to the activities and thesequencing Emphasise that all activities arelsquoup for grabsrsquo and can be changed or removedor that new ones can be added if this issensible

Refer back to the lsquowhyrsquo behind our methodIn projects that are doing something new (ieour sandboxes) we expect not to be right 100of the time Thatrsquos why we have built in spaceto iterate activities and sprints

Donrsquot be disheartened if you didnrsquot doeverything Sometimes an activity just wonrsquotget completed (or even started) in a sprint Thisis perfectly fine mdash time and capacity is one ofthe assumptions being (in)validated across ourwork Use this to reframe what amount ofactivity is realistic within a sprint and whetherthat activity should be amended or evenremoved

Make space for being wrong Try to give spaceto discuss each activity and think throughwhether it is still worthwhile Avoid a mindsetof lsquoletrsquos just do it because we said we wouldrsquo

Build on othersrsquo ideas and reflect on how tomake them more leanrsquo Sandboxes try tomaximise learning while minimisinginvestment They also welcome new creativeideas which meet this goal For ideas that theteam comes up with build (rather than replaceor critique) and think about lsquoHow can we try toacquire the same amount of insight but morequickly or cheaplyrsquo

The Sandbox Handbook v10 28

EdTech Hub

Timing and RhythmThe Sprint Review and iteration should take place in the break between sprints

33 Producing evidence outputs from what wersquove learnt

Types of evidence output

Our sandboxes produce a suite of evidence outputs These outputs capture thedata that has been generated during the sprints and publish it for the world Thisensures that what is learnt about different EdTech interventions is a global publicgood

Depending on your priorities you might want to publish insights from your worktoo Evidence on what works and what doesnrsquot in real-time and directly fromthose implementing is of tremendous value Alternatively you may wish toproduce evidence outputs as internal to your organisation to ensure knowledgefrom sandboxes is retained Or evidence outputs might not be relevant at all

The key evidence output from the sandbox is the Sprint Review This briefdocument should tell the story of what was done and learnt and how it informsthe future As such it should be descriptive about activities and provide insightsinto the data collected

The sandbox might produce other evidence outputs depending on the workdone in each sprint We encourage as much of the learning from the sandboxesas possible to be made public and will continue to explore ways to do thiseffectively

Table 2 A guide to different evidence outputs from a sprint

Evidence output Description Example

Sprint Review The document that comes out of the SprintReview and iteration process (see Section 21and Section 22) and captures the teamrsquosreflections on the activity that has takenplace in the sprint as well as how it affectswhat we do next While it doesnrsquot need tobe a comprehensive report and include all

EdTech Interventions forDeaf Learners inPakistan - Sprint OneReview (Rahman ampSiddiqui 2020)

The Sandbox Handbook v10 29

EdTech Hub

Timing and RhythmThe Sprint Review and iteration should take place in the break between sprints

33 Producing evidence outputs from what wersquove learnt

Types of evidence output

Our sandboxes produce a suite of evidence outputs These outputs capture thedata that has been generated during the sprints and publish it for the world Thisensures that what is learnt about different EdTech interventions is a global publicgood

Depending on your priorities you might want to publish insights from your worktoo Evidence on what works and what doesnrsquot in real-time and directly fromthose implementing is of tremendous value Alternatively you may wish toproduce evidence outputs as internal to your organisation to ensure knowledgefrom sandboxes is retained Or evidence outputs might not be relevant at all

The key evidence output from the sandbox is the Sprint Review This briefdocument should tell the story of what was done and learnt and how it informsthe future As such it should be descriptive about activities and provide insightsinto the data collected

The sandbox might produce other evidence outputs depending on the workdone in each sprint We encourage as much of the learning from the sandboxesas possible to be made public and will continue to explore ways to do thiseffectively

Table 2 A guide to different evidence outputs from a sprint

Evidence output Description Example

Sprint Review The document that comes out of the SprintReview and iteration process (see Section 21and Section 22) and captures the teamrsquosreflections on the activity that has takenplace in the sprint as well as how it affectswhat we do next While it doesnrsquot need tobe a comprehensive report and include all

EdTech Interventions forDeaf Learners inPakistan - Sprint OneReview (Rahman ampSiddiqui 2020)

The Sandbox Handbook v10 29

EdTech Hub

data collected during a sprint it should be alsquogreatest hitsrsquo of the most compelling datapoints covered during a sprint (the datapoints most relevant to validating pushingback on the hypothesis or theory of change)to demonstrate how sandbox partnersarrived at new decisions and iterations ofthe model

WhatsApp AssistedLearning for Refugees inLebanon - Sprint OneReview (Tutunji et al2021)

Other evidenceoutputs

These are agreed in lsquoldquoGet to workrsquo Session 3from hypothesis to action planrsquo (see Section23) alongside each activity Other evidenceoutputs might include

Write-ups of interviews and userresearch (preferably on actual userbehaviour rather than predicted orclaimed future behaviour)

Results of quantitative datacollection (preferably on usage andengagement rather than surveys)

Any multimedia gathered fromthose implementing the intervention(eg voice notes videos)

Survey findings barriersto learning withWhatsApp in refugeecamps in Lebanon(Boujikian amp Carter2021)

Storytellingoutputs

This is the output of the work started in thelsquopeople behind the datarsquo session but thereare a few outcomes that these feed into

The lsquostoriesrsquo themselves arestandalone pieces that should beembedded with the final evidenceoutput on the EdTech Hub website

The learning insights are fed intothe ongoing sandbox work via ameeting or workshop with the team

Key tool Templates forcreating storytelling outputsand applying them to sandboxwork (Dixon amp Patuck 2020)

Key tool examples andinspiration of storytellingoutputs (coming soon)

What makes a good evidence outputSandboxes offer a unique opportunity to generate lots of tangible evidence We

The Sandbox Handbook v10 30

EdTech Hub

believe this evidence is valuable to implementers educators policy makersfunders and others working with EdTech To make sure the evidence gets takenup try to

1 Make it concise Deliver only the most relevant information quickly Useexecutive summaries and lsquokey pointsrsquo to distil the core messages

2 Make it user-centred For each bit of evidence as well as the output as awhole think about the target user Assume that someone is coming to theevidence output with no contextual understanding How can we get themup to speed

3 Make it practical For each bit of evidence think about what would enablea target user to put it into action straightaway

The Sprint Review could also hyperlink out to other evidence outputs generated in thatsprint In this way the Sprint Review becomes the scaffolding for the evidence linking toother more detailed or specific outputs

Limitations of sandbox evidence outputsSandboxes by definition gather data through rapid experiments and thereforeproduce evidence that is highly contextualised and intended to take the specificsandbox partner to their next step in iterating scaling their model As a resultthere are limitations on the extent to which inferences from this data can beextrapolated to other initiatives or to the theme more broadly

Evidence outputs should aim to make clear how data might be utilised by thereader and be sure to cite its limitations Most likely Sprint Reviews should beviewed as case studies providing other practitioners an example of oneintervention learnt from rapid experiments to improve their model Data andinsights should serve as inspiration but not strict guidance to thoseimplementing similar initiatives in other contexts

Key artefact128221 our Insights Report (Rahman et al 2021) includes a sectiondefining what we mean when we talk about rigour in a sandbox and explainswell some of the limitations and benefits of our approach to data collection andanalysis

The Sandbox Handbook v10 31

EdTech Hub

34 Additional tools to help scale impact

During between and after sprints the team should do all they can to scaleimpact Whether it is crafting a human-centred user journey for people using theEdTech intervention or building strategic financial models or looking ateducation content or creating the most effective feedback surveys andquestionnaires or mobilising and connecting with partners or funders there aremany aspects to implementing EdTech that we seek to execute and developintentionally and with good practice

Just as we encourage sandboxes to we want to develop test and iterate toolsand approaches to help sandbox partners We also want to capture the best toolsothers have developed

Below is a list of tools we have seen or developed To ensure we remain focusedon the education system as a whole we have listed them against our 6 Ps

Please add comments with other tools you have built or used and we willadd them to our catalogue

Catalogue of tools to help scale impact

Table 3 A catalogue of tools to help understand and grow edtech interventions

6P What it is Key features

Provision Lean CostModel

Key tool128736 Lean Cost Model (Rahman 2020a)

A way to understand how much an EdTech interventioncosts per child per year by building up the differentelements including costs of technology and costs ofimplementation

Includes inputs for certainty of different costs whethercosts are crucial or lsquonice to haversquo and how much a cost maydecrease at scale relative to in the sandbox

People ServiceDesignBlueprint

Key tool128736 Service Design Blueprint (Carter 2021)

A way to make sure that each point in the userrsquos interactionwith a product from start to finish is considered

The Sandbox Handbook v10 32

EdTech Hub

Includes everything that happens lsquofrontstagersquo (things youruser sees) and lsquobackstagersquo (things hidden from view formyour user)

35 Harnessing collective intelligence

Sandboxes are working in emergent domains and on uncertain ideas Wersquoveseen that in these domains the knowledge experience and tools of otherdifferent sandboxes are an incredible resource for each sandbox to tap into Manysandboxes work on similar themes and overcome common barriers to scalingtheir impact If you have a portfolio of EdTech interventions you are testing werecommend embedding a process for sharing knowledge among the group AtEdTech Hub we do this via the Sandbox monthly Meetup

The Sandbox MeetupThe 75-minute Sandbox monthly Meetup is a gathering of all our sandboxpartners as well as select guests and our sandbox alumni Itrsquos open to anyonewho has been part of a sandbox and includes the following key features

1 Each Meetup is based on a clear theme that the community isexperiencing in their work Having this theme means each sandbox comesto the session as an expert feeling like they can contribute and that theMeetup is relevant and applicable to their work

2 One or two organisations give a brief presentation (~10 mins) on the chosentheme that the community is experiencing which provides the anchor forthe Meetup This puts the experience and expertise of each partner on apedestal clearly framing them as experts

3 Sandboxes tell us the part they enjoy most is engaging with the speakerand with each other Thatrsquos why each Meetup begins with lsquospeednetworkingrsquo and after the presentation has a series of breakout rooms todiscuss whatrsquos been presented

The Sandbox Handbook v10 33

EdTech Hub

4 At the end of the Meetup the insights are collated into a blog by EdTechHub Herersquos an example of the blog from our first Meetup on engagingcaregivers and communities with equity (Rahman 2020d)

More specific details are in the Meetup tool below

Key tool128736 Everything you need to know about the Sandbox Meetup (Rahman2020) A planning guide to hosting a Sandbox Meetup with everything from thetemplate agenda invite copy and space to taking notes from the Meetup itself

The Sandbox Handbook v10 34

EdTech Hub

4 At the end of the Meetup the insights are collated into a blog by EdTechHub Herersquos an example of the blog from our first Meetup on engagingcaregivers and communities with equity (Rahman 2020d)

More specific details are in the Meetup tool below

Key tool128736 Everything you need to know about the Sandbox Meetup (Rahman2020) A planning guide to hosting a Sandbox Meetup with everything from thetemplate agenda invite copy and space to taking notes from the Meetup itself

The Sandbox Handbook v10 34

EdTech Hub

Bibliography⁅bibliographystart⁆

This bibliography is available digitally in our evidence library athttpsdocsedtechhuborglib772S7Z4R

Acumen (2019) Lean Data Initiative httpsacumenorglean-data (details)

Atlassian (2020) Atlassian Team Playbook - Build strong teams with PlaysAtlassian httpswwwatlassiancomteam-playbook (details)

Banerjee A Andrabi T Grantham-McGregor S Yoshikawa H Saavedra JBanerji R Akyeampong K Dynarski S Glennerster R Muralidharan KSchmelkes S amp Piper B (2020) Cost-effective approaches to improveglobal learning What does recent evidence tell us are ldquoSmart Buysrdquo forimproving learning in Low and Middle Income Countries [TextHTML]httpswwwworldbankorgentopicteachingandlearningpublicationcost-effective-approaches-to-improve-global-learning (details)

Boujikian M amp Carter A (2021) Lack of Means or Lack of Awareness Surveyfindings barriers to learning with WhatsApp in refugee camps in Lebanon[Sandbox Evidence Output] EdTech Hubhttpsdoiorg105281zenodo4780972 Available fromhttpsdocsedtechhuborglib4UQT8WRN Available under CreativeCommons Attribution 40 International (details)

Carter A (2021) Service Design Blueprint [Internal papers] EdTech Hubhttpsdoiorg105281zenodo5147560 Available under Creative CommonsAttribution 40 International (details)

Chang A M (2018) Lean Impact How to Innovate for Radically Greater SocialGood (1 edition) Wiley (details)

Common Good Co-Op Digital Hyper Island Open Data Institute amp PLOT (2020)Ethics Kit | Methods amp tools for ethics in the design processhttpsethicskitorgindexhtml (details)

The Sandbox Handbook v10 35

EdTech Hub

Dixon M amp Patuck L (2020) Storytelling Planning Template [Internal papers]EdTech Hub httpsdoiorg105281zenodo5147549 Available under CreativeCommons Attribution 40 International (details)

Dixon M amp Patuck L (2021) The Storytelling Project Handbook EdTech Hubhttpsdrivegooglecomfiled10BOrSxq6B5yUOI-bSpJ4oBMOSjhhuiANview (details)

EdTech Tulna (2020) EdTech Tulna Standardshttpsedtechtulnaorgstandards (details)

Edell P (2019 August 28) What Is Agile Methodology A Primer on Moving FastAngelList Bloghttpsangelcoblogagile-methodology-a-primer-on-moving-fast (details)

Google (2020) Design Sprints with Googlehttpsdesignsprintkitwithgooglecommethodologyphase2-definefuture-press-release (details)

IDEOorg (2015) Design Kit Design Kit Methods for Human Centered Designhttpswwwdesignkitorgmethods (details)

McBurnie C amp Plaut D (2021) Continuous professional development in SierraLeone Scoping Phase Overview [Internal papers] EdTech Hubhttpsdoiorg105281zenodo4774318 Available fromhttpsdocsedtechhuborglibEKMQD8WQ Available under CreativeCommons Attribution 40 International (details)

Murray Peter amp Ma Steve (2015) The Promise of Lean Experimentation StanfordSocial Innovation Review (SSIR)httpsssirorgarticlesentrythe_promise_of_lean_experimentation (details)

NESTA UK (2014 February 16) Development Impact and You (DIY) Toolkithttpsdiytoolkitorg (details)

Plaut D (2020) Lessons for effective CPD - Synthesis of a Sandbox LearningSession [Internal papers] EdTech Hub httpsdoiorg105281zenodo5123352Available under Creative Commons Attribution 40 International (details)

Plaut D (2021) Sierra Leone CPD Sandbox - Draft Theory of Change [InternalPaper] EdTech Hub httpsdoiorg105281zenodo4959786 Available from

The Sandbox Handbook v10 36

EdTech Hub

httpsdocsedtechhuborglibT7AUREA5 Available under CreativeCommons Attribution 40 International (details)

Rahman A (2020a) Sandbox Lean Cost Model [Internal papers] EdTech Hubhttpsdoiorg105281zenodo5147557 Available under Creative CommonsAttribution 40 International (details)

Rahman A (2020b) Sandbox Meet-up Planning Template [Internal papers]EdTech Hub httpsdoiorg105281zenodo5147564 Available under CreativeCommons Attribution 40 International (details)

Rahman A (2020c) Sandbox Sprint Review Template [Internal papers] EdTechHub httpsdoiorg105281zenodo5147553 Available under CreativeCommons Attribution 40 International (details)

Rahman A (2020d November 23) Engaging with equity insights from our firstsandbox meetup EdTech Hub httpsdoiorg105281zenodo4707465Available fromhttpsedtechhuborg20201123engaging-with-equity-insights-from-our-first-sandbox-meetup (details)

Rahman A amp Carter A (2020) Virtual Sandbox Pack [Internal papers] EdTechHub httpsdoiorg105281zenodo5147542 Available under CreativeCommons Attribution 40 International (details)

Rahman A amp Siddiqui M (2020) EdTech Interventions for Deaf Learners inPakistan [Working Paper] (details)

Rahman A Plaut D Salami T amp Carter A (2021) EdTech and Covid-19 Insights from our Sandbox Portfolio In Collaboration with Sandbox PartnerOrganisations [Working Paper] EdTech Hubhttpsdoiorg105281zenodo4776960 Available fromhttpsdocsedtechhuborglibMXMPWAA2 Available under CreativeCommons Attribution 40 International (details)

Savoia Alberto (2011) Pretotype It Make sure you are building the right it beforeyou build it righthttpwwwpretotypingorguploads140914099067pretotype_it_2nd_pretotype_edition-2pdf (details)

The Sandbox Handbook v10 37

EdTech Hub

Simpson L Carter A amp Rahman A (2021) Hello and Welcome to the Sandbox[Internal papers] EdTech Hub httpsdoiorg105281zenodo5123366Available from httpsdocsedtechhuborglibCQQURC65 Available underCreative Commons Attribution 40 International (details)

Simpson L Carter A Rahman A amp Plaut D (2021) EdTech Hub Sandboxes in2021ndash22 [Position Paper] EdTech Hub httpsdoiorg105281zenodo4966322Available from httpsdocsedtechhuborglib8HKCSIRC Available underCreative Commons Attribution 40 International (details)

Tutunji S Boujikian M Carter A amp Atkinson G (2021) JusoorrsquosWhatsApp-Assisted Learning Programme Sprint Review 1 [Sprint Review]httpsdoiorg105281zenodo4468599 Available fromhttpsdocsedtechhuborglibAR38F6N8 (details)

⁅bibliographyend⁆

The Sandbox Handbook v10 38