ERASMUS PLUS 2015 SECTOR SKILLS ALLIANCES · Deliverable 6.2 – Evaluation and Monitoring Plan 562634-EPP-1-2015-IT-EPPKA2-SSA CARESS Project 3 of 73 4 I.T.C. “Vittorio Emanuele

ERASMUSPLUS2015SECTORSKILLSALLIANCES

AGREEMENTNo.2015–3212/001–001

PROJECTNo.562634-EPP-1-2015-IT-EPPKA2-SSA

DeliverableNumber: 6.2

TitleofDeliverable: EvaluationandMonitoringPlan

WPrelatedtotheDeliverable: 6

DisseminationLevel:(PU/PP/RE/CO)*: PU

NatureoftheDeliverable:(R/P/D/O)**: R

ActualDateofDeliverytotheCEC: 16/06/2017

PARTNERresponsiblefortheDeliverable: UVa-Education

WPstartingmonth M16

WPendingmonth M19

PartnerContributor(s): SI4LIFE,UVa-Tech,Uva-Nurses,OMNIA,VE-II,AGE,GGALERY,AGE

PartnerReviewer(s): SI4LIFE,UVa-Tech,Uva-Nurses

*DisseminationLevel:

PU=Public

CO=Confidential,onlyformembersoftheAlliance(includingCommissionServices).

PP=Restrictedtoexternalsubjectsinconfidentialmode(includingCommissionServices)

RE=RestrictedtoagroupspecifiedbytheAlliance(includingCommissionServices).

**NatureofDeliverables:

R=Report

P=Prototype

D=Demonstrator

O=Other

Deliverable6.2–EvaluationandMonitoringPlan

562634-EPP-1-2015-IT-EPPKA2-SSA CARESSProject 2of73

1 ABSTRACTThisdocumentillustratestheevaluationprocesssettledintheCARESSproject,addressingtwofacetswithintheproject:theevaluationoftheframeworkandguidelines,aswellastheevaluationoftheCaressPilotsandtheirresponsivenesstotheneedsofthejobmarket.Moreover,thedocumentdefinesandintegratesalist of criteria and indicators thatwill beused in T6.1, and in T6.3 tomake theevaluationprocessmorereliable.Throughthedocument,wewillpresentsomeexamplesofbasic instrumentsthatwehavedevelopedfortheevaluationofthequalityandtheefficacyoftheCARESSpilotsinFinland,ItalyandSpain(WP4).Pilotswill be evaluated in their threemain dimensions: blended learning, e-learning and non-formal/informallearning,throughtheuseoftheCARESSVirtualCommunityofPractice(VCP).This document also presents an “evaluation toolkit” of basic instruments that have been developed forpilots’evaluation inthethreecountries involved.Theseassessmenttoolscanbeusedbefore,duringandafter the implementationof theaforementionedpilots, tocollectvaluabledata,whichwillbehelpful forformativeandsummativeevaluationpurposes.AsthedesigndocumentsforthePilotsarenotbeingreleasedalready,weneedtotakeintoaccountthoseevaluation instrumentsareprovisional, andmay suffer someminormodifications tobetter fit thePilot’sdesign.Thepilotdesignerscanusethe“evaluationtoolkit”astheyarepresentedhere,translatethem,ordevelopsimilartoolsbasedonthese,accordingtotheirspecificneeds.

2 KEYWORDS:Qualityassessmentstrategies,evaluationandmonitoringplan,CARESSEUframework,Pilots

3 LISTOFBENEFICIARIES(PP-RE)/PARTICIPANTS(PU-CO)***

Ben.No. BeneficiaryName ShortName Country

1 Si4Life–ScienzaeImpresaInsiemeperMigliorarelaQualitàdellaVitas.c.r.l.

Si4Life Italy

2 RegioneLiguria LiguriaRegion Italy

3 Ggallerys.r.l. GGallery Italy



4 I.T.C.“VittorioEmanueleII-Ruffini” VE-II Italy

5 AGE-Platform AGE Belgium

6 OMNIA OMNIA Finland

7 FinnishNationalBoardofEducation FNBE Finland

8 Super SUPER Finland

9 AyuntamientodeValladolid AYTO Spain

10 UniversidaddeValladolid UVA Spain

11 SociedaddeGeriatríayGerontologíadeCastillayLeón SGGCYL Spain

12 AssociazionePoloTecnicoProfessionaleProfessioniVita Pro.ViHub Italy

13 AziendaRegionaleSanitariaLigure ARS Italy

14 RoyalCornwallHospitalsTrust RCHT UK

15 UnitedKingdomHomecareAssociation UKHCA UK

16 NestorPrimecareServicesLtd–AlliedHealthcare AlliedHealthcare UK

***ListofBeneficiaries

IncaseofdisseminationlevelPUorCOpleaseindicateallthepartnersinvolvedinthisDeliverable.

IncaseofdisseminationlevelPPpleaseindicatethenamesoftheothersubjecttowhomthedeliverableisdevoted

IncaseofdisseminationlevelREpleaseindicatetherestrictedgroupofpartners.

4 VERSIONHISTORYandAUTHORS

VERSION PRIMARYAUTHOR VERSIONDESCRIPTION DATECOMPLETED

1 SaraVillagráSobrino(UVa)EstefaníaArribasLeal(UVa)

Deliverablestructureanddraftcontent

26/04/2017

2 SerenaAlvino(SI4LIFE)

FirstdraftfeedbackQualityindicatorsT.6.1Criteriaandindicatorstablestructure

05/05/2017

MaríaJoséCao,MaríaJoséCastroAlija,JoséMaríaJiménez,MiriamIdrissiCao(UVa-Nurses)

FirstdraftfeedbackDescriptionofSpanishPilotcontext

26/05/2017

JuanIgnacioAsensioPérez(UVa-Tech)

FirstdraftfeedbackVCPanalyticsindicators

17/05/2017




ModifytheannexesAddPilots’contextdescriptionIntroduceTableofqualityassessmentindicators

02/06/2017

4 SerenaAlvino(SI4LIFE) Reviewseconddraft 03/06/2017

5 FrancescaFerrando(Ggallery) Trackingindicators 05/06/2017

6 BarbaraMazzarino(SI4LIFE)MartaRomagnoli(SI4LIFE)DanieleMusian(SI4LIFE)SerenaAlvino(SI4LIFE)

Task6.1DescriptionandindicatorsReviewofVCPevaluationandindicatorsFinalreview

14/06/2017


Finalintegrationandapproval 16/06/2017

5 TABLEOFCONTENTS1 ABSTRACT.....................................................................................................................................2

2 KEYWORDS:..................................................................................................................................2

3 LISTOFBENEFICIARIES(PP-RE)/PARTICIPANTS(PU-CO)***.......................................................2

4 VERSIONHISTORYandAUTHORS................................................................................................3

5 TABLEOFCONTENTS....................................................................................................................4

6 INTRODUCTION:STRUCTUREANDAIMSOFTHEDOCUMENT....................................................5

7 EVALUATIONCONTEXTANDMETHODOLOGY.............................................................................7

7.1 EvaluationoftheCARESSframeworkandWizard/guidelines(T.6.1)...............................................10

7.2 Evaluationofthepilots(T6.3)...........................................................................................................11

7.3 EvaluationtaskswithinCARESSproject:generaloverview...............................................................15

8 CONTEXTANDEVALUATIONMETHODINT6.3..........................................................................24

8.1 Descriptionofthetimelineandtheevaluationinstruments............................................................29

8.2 Whowillcollectthedata?.................................................................................................................31

8.3 DraftoftheevaluationtoolkitinT.6.3..............................................................................................33

9 QualityAssuranceandOverallProjectEvaluation.....................................................................34

10 BIBLIOGRAPHY...........................................................................................................................35

11 ANNEX........................................................................................................................................36



11.1 Annex1:Guidelinestoperformtheobservations(Draft)...............................................................36

11.2 Annex2:Guidelinestocompletetheteacher’sReflectiveDiary(Draft)........................................37

11.3 Annex3:Guidelinestoperformtheteacher’sinterview(Draft)....................................................38

11.4 Annex4:Aquestionnairetoevaluatethetrainingmaterialsbythestudents(Draft)...................39

11.5 Annex5:QuestionnairetoevaluatethetrainingmaterialsbyTeachers(Draft)............................43

11.6 Annex6:Students’initialquestionnaire(Draft)..............................................................................48

11.7 Annex7:Students’finalquestionnaire(Draft)...............................................................................49

11.8 Annex8:Teacherfinalquestionnaire(Draft)..................................................................................53

11.9 Annex9:Questionnaireevaluatingteachers’Pilotdesign(Draft)..................................................58

11.10 Annex10:GuidelinestocompletetheStudent’sReflectiveDiary(Draft)....................................65

11.11 Annex11:Student’posttrainingquestionnaire(Draft)...............................................................66

11.12 Annex12:Student’posttraininginterview(Draft).......................................................................67

11.13 Annex13:TrackingreportingCARESSe-learningplatform...........................................................68

6 INTRODUCTION:STRUCTUREANDAIMSOFTHEDOCUMENTThe CARESS Evaluation and Monitoring Plan is a document that describes how we will monitor andevaluate theCARESSpilot courses carriedoutwithin theCARESSproject, and the criteria and indicatorsthatshouldbeusedtoevaluatetheCARESSEUframeworkandguidelines.Afterwards,thosecriteriaandindicatorswillbeusetoevaluatetheresponsivenessofthetrainingprovidedwithintheCARESSprojecttotheneedsofthejobmarketanddrawdownacomprehensivefinalevaluation.Aswe can see in figure 1, whereas the ultimate purpose of Task 6.1 is to gather feedback from pilots’designerstovalidateandprovetheuseoftheCARESSEUframeworkandtheguidelinesdeliveredinT.3.1and3.2,task6.2isfocusedontheevaluationplanandmonitoringofthepilotsanditsfunctioning.Besides,weprovideadescriptionof theprocedurethatwill follow,aswellasasetof indicatorstoassesstheEUframework and the guidelines for the design of the pilots. Finally, a description of the instruments andguidelinesthatwillbeusedtoassessthedesignandtheenactmentofthepilotsisprovided,withtheaimof assessing the responsiveness of the training developed to the jobmarket needs according to home-healthcareprofessional’sperspective. According to the aforementioned ideas, a comprehensive final evaluation of the pilots, linked with theresponsiveness to jobmarketneeds, (i.e., task6.3),will alsobeevaluated following specific,measurableindicatorsdefinedinthisdocument.



Figure1.Relationshipsbetweenevaluationtasks

Animportantfirststeptowardsthataimisthedescriptionofthescopeandthemethodofdatacollectionand analysis that will follow. The evaluation plan will clarify the procedure whereby the informationgatheredfromthepilot'sdesignhavebeenusedtoassesstheEUframeworkandtheguidelinesaswellasabouthowthepilot’sexperienceshavebeendevelopedinFinland,ItalyandSpain.Asaconsequence,wewanttoknow,towhatextenttheactivitiesandtasksdevelopedinthenationalpilotsarelinkedwiththeintended learning outcomes. Moreover, we want to illuminate whether the pilots were operatingcoherentlytotheprogram’sdesign.Additionally,theprocessevaluation,alongwithoutputandshort-termoutcomeinformation,willhelptoclarifywhetherpilotsrespondtotheintendedCARESSprojectobjectives.It isworthnoting that theevaluation topicshavebeendefined taking intoaccount the listof short-termresultsand longtermoutcome indicators thatwereexplicitlymentioned in theCARESSprojectproposal.These indicators are specific for eachactivity tobeevaluated, and correspond to thedifferent aimsandgoalsregardingthefunctionalitiesoftheelementtobeassessed.Someoftheindicatorscanbetrackingbycompiling thedata requested (e.g. that the learningmaterials arewriting in two languages, thenationallanguage(Italian,FinnishandSpanish)andEnglish,whileothersmayneedthedevelopofaparticulartool.Nevertheless, all of them are linked with the quality standard for the whole project, connecting theevaluation with the assessment of the project quality. Thus, quantitative and qualitative indicators aredescribedinsection7.3.This report is structured as follows. Section 7 provides a general description of the context and theevaluation methodology guided by the CSCL-EREM evaluation framework (Jorrín-Abellán, 2013) withinCARESSproject.Besides,clarifieswhatitisthescopeoftheevaluationattendingtothedifferentscenariosandobjectives,withasectiondevotedtotheevaluationofEUframeworkandguidelinesandanothertotheevaluationofthenationalPilotsandtheposttrainingresponsivenessofthejobmarket.Likewise,weshowedageneraloverviewof theevaluationprocesswithin theCARESSproject.Also in this section,wepoint out the evaluation objectives linked to criteria and indicators according to the status of theassessmenttoolsandthetimewheretheevaluationwillbecarriedout.



Then,section7.2explainshowtheevaluationofthequalityandtheefficacyofthepilot’scourses inVETwillbeperformedinItaly,SpainandFinland.Ontheonehand,weprovidedadescriptionoftheevaluationtimelineand theevaluation instruments thatare supposed tobeused forgatheringevidenceduring thedesignandtheenactmentofthepilot’scourses.Ontheotherhand,wehighlightpartner’sresponsibilitiesandrolesofCARESSpartnerswithrespectontheevaluationtasksthatwillbecarriedout.Finally, section 11, the annexes, draws down a draft of the “evaluation toolkit”: a set of assessmentinstrumentsandguidelinesdevelopedwith theaimofhelpingCARESSpilots'designers togatherdata toassess theusefulnessof theEU frameworkand theguidelines todesignVETpathways.Moreover,aboutthepilots'qualityandefficacyandresponsivenesstojobmarketduringandposttraining.It is worth notice that the evaluation toolkit showed just a first draft of assessment instruments (i.e.,observations,questionnaires,interviews,etc.)thathastheaimofguidingevaluatorsduringtheevaluationtasks. The ideas provided in the “evaluation toolkit” could suffer modifications to better adapt theinstrumentstotherealneedsofthecontextandthe informants (i.e., teachers,students,VETexperts)oftheevaluationtasksperformed.

7 EVALUATIONCONTEXTANDMETHODOLOGYThe evaluation context and scope is divided attending the two different scenarios. First of all, theevaluationoftheusefulnessoftheEUframeworkandguidelines(theWizardtool)duringthedesignofthepilotslearningdesign,describedinsection7.1.Secondly,theevaluationofthenationalpilotsandthejobmarketresponsivenessafteraperiodofsix-monthposttraining,describedinsection7.2.Withtheaimofclarifying the scopeof theevaluation taskswithinCARESSproject, section7.3will showa tablewith theassessment objectives linked with criteria, sub criteria and indicators according to the status of theassessmentinstrumentsaswellasthetimewheretheevaluationwilltakeplace.

DuetotheinherentcomplexityofevaluationitselfandthedifficultyofhealthcarescenarioscarriedoutindifferentplacesacrossEurope, it is convenient tousea theoretical framework toguide thecollectionofevaluationdata,itsanalysisanditsinterpretation.Specifically,theevaluationofCARESSprojectfollowstheEREM model (Evaluand-oriented Responsive Evaluation Model) (Jorrín-Abellán, 2009). The EREM is aframework that relies on a responsive evaluation approach, empowering the idea of conductingevaluationsfocusedonthephenomenatobeevaluated(evaluand)ratherthaninthefieldofexpertiseoftheevaluators(Stake,2004).Itisworthnotingthatanevaluandcanbedefinedastheentity,theprocess,theactivity,ortheproductbeingevaluated.Besides,theEREMtriestobeaframeworkforhelpingintheevaluation of Computer Supported Collaborative Learning programs, innovations, learning and teachingstrategies,educationalresourcesaswellasresearchprojects.This evaluationmethod follows an interpretative research perspective that does not pursue statisticallysignificant results or generalizations. Rather, it aims at a deeper understanding of the particular



phenomenaunder study. Inour case [1] theusefulnessofCARESSEU framework tooland thewizard tohelpteachersduringthedesignofthepilotsand[2]thequalityandefficacyofthelearningcoursescarriedoutthroughthepilotsperformedinItaly,SpainandFinlandacrossCARESSproject.Conductinganevaluationthat is responsivetotheneedsofdifferenthealthcarepractitionerswillensurethat evaluative information ismeaningful andmore likely to be used for programme enhancement andimprovement.Accordingtotheaforementionedideas,wethinkthatEREMcouldbeausefulframeworkwhichcanhelpusin the creation of an evaluation design suitable to the particular necessities derived from the differentcontexts in where the pilots will take place. Furthermore, the responsive evaluation model has beenimplementedinvariouspolicysectors,includingthefieldofhealthcare:

● To the evaluation of clinical skills assessment and training (CSAT) programme at the Faculty ofMedicine(Curran,Christopher,Lemire,Collins&Barret(2003).

● Palliativecareprogramsandunits(Abma,2001).● Elderlycare(Koch,1996)andnursingcurricula(Koch,2000).● Rehabilitationprogramsforpsychiatricpatients(Wadsworth,2001).● SupportedEmploymentprogramsformentallyhandicapped(Widdershoven&Sohl,1999).

InthecontextofCARESSproject,theevaluationhastheaimofprovidingevidenceaboutthequalityandtheefficacyofpilotsandtheirresponsivenesstotheneedsofthejobmarketandabouttheusefulnessofCARESS EU tool to help practitioners during the design of VET courses. Therefore, this evaluation has athreefoldevaluandthatincludes:thequalityandefficacyofpresencecoursesinhomecare,theCARESSe-learning platform and Virtual Community of Practice acceptability and the usefulness of EU CARESSframework tool and their guidelines to support teachers during thedesignsof learningpathways inVETscenarios.



Figure:1GraphicalrepresentationofCARESSpilots’evaluationdesign

Aroundtheevaluandconcept,theframeworkstructurestheevaluationdesignintothreedifferentfacets:perspective (why the evaluand is evaluated), covering themain goal pursued andother significant openquestions;ground(wheretheevaluandisevaluated),gatheringtheinformationaboutthecontextsandtheparticipants;andmethod(howtheevaluandisevaluated),showingthedatagatheringtechniquesandthedocuments that support the conclusions. Figure 1 illustrates a general representation of the EREMframeworkincludingtheevaluand(inthecoreofthefigure)andthedifferentfacetsaroundtheevaluand.ThisgraphicalrepresentationhasbeenobtainedusingtheCSCL-EREMwebtool1.TheperspectivefacetisorganizedaroundthedefinitionofIssues.Anissuecanbeunderstoodasatension,atroublingchoice.Oneissuenaturallyarisesfromtheobjectivesofthepresentevaluation.Itisstatedasfollows:[Issue1]HowtheCARESSEU framework tooland thewizardcanhelp teachersduring thedesignof thepilots?[Issue 2] Can training courses based on competences and active methodologies be helpful to improvenurses/OSSandpracticalnursesclinicalpracticeandtransversalskillsinelderlycare?

1 EREMwebtool:https://pandora.gsic.uva.es/cscl-erem



7.1 EvaluationoftheCARESSframeworkandWizard/guidelines(T.6.1)Accordingwiththeproject,twoofthemainresultsofCARESSprojectare:

• AEuropean framework forvocationaleducationandtraining (VET) in the fieldofhomecare, thatreflects the professions and occupations actually involved in domiciliary care, thecompetencies/skills required foreachprofession,andtherelated learningoutcomes (referring toECVET);

• Guidelines supporting different stakeholders working in the field of homecare in the use theEuropean framework itself for designingVET curriculawhich take into account contextual needsandalreadyexistinginitiatives.

Framework andGuidelines are delivered inWP3 andwill evolve during the lifespan of the project. Thisprocedure of progressive refinement requires scheduling different evaluation slot following thedevelopmentoftheresults.

Thedesignofthethreenationalcurriculafortheimplementationofthepilots(WP4)isthefirsttestbedfortheFrameworkandguidelinesandcorrespondtothefirstvalidationslot (M21),aswellas thereleaseoftheirfinalversioncorrespondtothelastevaluationperformed(M32-M34).Theadditionalevaluationstepswillbescheduledintask6.1.

Asdescribed inD3.2.1and inD3.3.1 tosupport thecollaborativedevelopmentof theFrameworkand itsusability, intermsofprovidingstructured,organizedandsystematized information inaneasilyaccessiblemanner,aweb-basedsolutionhasbeenimplemented.Thischoicewillallowfor:

• asimpleanduser-friendlyaccess(throughqueries)tospecificdatacollectedduringthecontextualanalysis;

• a simple and user-friendly review of information, even in a bottom-up approach, involvingrepresentativesofstakeholdersfromdifferentcountries;

• aprogressiveintegrationofinformationthroughoutthelifespanoftheproject.

ObjectivesoftheFrameworkevaluationaretoassess:

1. Theusabilityofitsfunctionalities(asidetothesupportoftheWizard/Guidelines);2. Therepresentativenessoftheinformationcontainedintheframeworkand3. Theusefulnessinthedesignofcompensativelearningmodules.

Details about the related indicators are provided in Table 3. Specific Questionnaires and Interviews toevaluatetheefficacyandusabilityoftheFrameworkaswellasfortheGuidelines/WizardwillbedesignedinT6.1alsoincludingtheresultsofthepreliminaryevaluationoftheframeworkusabilityperformedatM14andtherelatedquestionnaire(D3.1.1).

SincetheFrameworkhasbeenimplementedasaweb-basedplatform,theguidelinessupportingdifferentstakeholdersintheuseoftheFrameworkhasbeenreleasedinT3.2asanonlineWizard.



The Wizard is a user-friendly interface allowing for a personalized "step-by-step" access to thefunctionalities of theweb-based framework. In this way different stakeholderswill be supported in theaccess,use,andnavigationoftheFrameworkandinperforming3mainactivities,asdescribedinD3.2.1byfourUseCases:

• Case0:Registration/AccessPersonalization• Case1:Frameworkconsultation(queries)• Case2:Frameworkintegration(dataentry)• Case3:InstructionalDesignofcompensativemodules

Case0andCase1hasbeen implemented in its first releaseandwillbeevaluated following the timelinethatwillbeprovidedinTask6.1.

Case 2will be implemented by the end of the second year of the project, in order to allow frameworkreview/integrationinthethirdyear.Beforeitsrelease,itwillbetestedasplannedinT6.1.FeedbacksfromuserswillbecollectedbyM32inordertopossiblyrefinefeaturesofCase2forthefinalrelease.

Case3isrelatedtothesupportwhichtheWizardcouldprovidetothedesignofnationalcurriculastartingfromtheirrealcontext,takingintoaccountcoursesandinitiativesalreadyimplementedinthecountryandspecificlocalrulesandcertifications.

Specificguidelinesfor“CARESSFramework-supported” instructionaldesignhavebeendelivered inD3.3.2intermsof“DesignPatterns”,i.e.“recurrentsolutions”whichcouldbeusedbyVETinstructionaldesignersand teachers when approaching the challenge of designing a Compensative Learning Module to fill aspecific“competencegap”thataffectsacertainHHCP(seefordetailsD3.3.2).

CARESSDesignPatterns,togetherwiththeCARESSIDTemplate(seeD3.3.2)willbeatthebaseof"Case3"andwillbeintegratedintotheWizardinitsfinalrelease.BeforeimplementingthesedesigntoolsintotheWizard,ithasbeendecidedtoevaluatethemwiththeCARESSpilotdesignersduringT3.4(PilotDesign);tothisend,boththeDesignPatternsandtheCARESS IDTemplatehavebeenusedbythepilotdesigners inT3.4toformalizeandsharethedesignofnationalpilotsFeedbacksgatheredfromthistestingbeusedtorefinetheinitialpatternsandtemplatewhichwillbeimplementedintheWizardinterface

7.2 Evaluationofthepilots(T6.3)Nationalpilotswillbedesignedandimplementedinthreecountries,targetingdifferentHCCPs.

1. SpanishPilotcontextdescription:ThedegreeofNursing, inordertoachieveallofthecompetenciesprovidedwillberequiredtocomplete240ECTS (eachECTS,byagreementof theGoverningCouncilof theUVa,willhaveanequivalenceof25hours), which will be distributed as follows (Distribution in accordance with the guidelines of the RD1393/2007):



· 60ECTSforbasictraining,84ECTSforthecompulsorysubjects· 6ECTSforelectivecourses· 84ECTSforthepracticum· 6ECTSforthecompletionofadegreedissertation

TotalcreditsECTS: 240

Typeofcourse

BasicTraining 60

Compulsory 84

Elective 6

Externalinternships 84

Degreedissertation 6

Table1.ECTScreditsinnursingcareerinSpainaccordingtoeachtypeofcourse

For 2016/2017 there are 57 universities in Spain with Faculties of Health Science, Faculties of Nursing,FacultiesofMedicineorUniversitySchools standingaDegree inNursing,with4750newopeningsbeingoffered. Nevertheless,when considering specifically the homecare assistance of the elderly, several skillgapshavebeenfound.Theseskillgapsareanobstacleandmaketheinvolvementofthenurseswiththisquestiondifficult.Theseskillgapsare:

● Socio-emotionalneeds. ● Administrativeoperationsandsanitarydocumentation. ● Communication. ● Relationswiththeworkequipment. ● Environmentandriskpreventionintheworkplace. ● TraininginEthics. ● Basicknowledgeaboutsocialandhealthservicesorganizationsandnetworks. ● Basicknowledgeoflegalframeworkandhumanrights. ● Useofinformationandmonitoringtools. ● Useofcomputersystemsforsocialparticipation. ● Useofcomputersystemsforhealthmonitoring. ● Informationtocustomersaboutreliableservicenetworks. ● Competenciestocoordinateotherprofessional’stasks. ● Competenciesformentalhealthassessment. ● Competenciesforuserempowerment. ● Supportindepressionordespaircases.



● Theneedofsupporttooperatetechnologicaldevicesforhomehealthmonitoring. ● Theneedtosupportusersintheself-managementofhis/hermentalhealth. ● Maintainsocialandfamilyrelations. ● Solvingdailyproblemsorknowingwheretoseekforhelp. ● Significantimprovementincoordinationcapacitytoplanactions. ● Meetings with health teams, and keeping in touch with other professionals who also deal with

eldercare. ● Reportsontheactivitiescarriedout. ● PersonalAssistancePlanning.

Asaconsequence,thecoursewillbeorientatedtoallthegraduatednurseswith6EQFLevelthatmayhavetheinterestinimprovingtheirknowledgeandskillsinelderlyhomecare.Forthiscourse,previouspracticalexperienceonthefieldwillnotberequired.Finally,inrelationtotheICTcompetenciesbeingfamiliarwiththeuseofanonlineplatformwillbeadvisable.

2. ItalianPilotcontextdescriptionItalianpilotwilltargetOSS(SocialHealthOperators),whichareHealthandSocialOperatorswhoperformasubsidiaryroleinnursingcare(EQF4).Atthemoment,characteristicsandlearningobjectivesofcoursesforbecomingOSSareless-structuredandless-defined.

3. FinlandPilotcontextdescriptionFinland pilot will target practical nurses (EQF4). They must command value competence, occupationalethics,toleranceandproblem-solvingskills inthefield, inadditiontonursingandcareskills;skillsgaponthesetransversalcompetenceshasbeendetectedinFinland.

According with the project, the implementation of the training courses has three different spaces orapprenticeshipareas:BlendedlearningEachVETpathwaydesignedwillbeimplementedinablendedlearningsituationinItaly,Spain,andFinland.Table2 illustrates the location inwhich thepilotswill be carriedout, the target, aswell as thepartnersinvolved.

Blendedcourse

Location Target Coordinator

Genoa(Italy) Healthsocialoperators

VittorioEmanueleII.Professionalschool.

Valladolid(Spain) Nursingstudents FacultyofNursingandMedicine.



Espoo(Finland) Practicalnurses OMNIA,multi-sectoreducationproviderandregionaldevelopmentcentre

Table2.Blendedcourseslocation,targetandCARESSpartnercoordinator

Presence learning should be based on different methodologies (such as education in the workplaceenvironment,school/workturnover,etc.)andstrategies(i.e.,casestudies,criticalincidents,problem-basedlearning,etc.).E-learningSpecific learning activities will be set up in the e-learning course, fostering learning with differentmethodologies(i.e.,casestudy,criticalincidents,roleplaying,etc.).Learning activities will be supported by the presence of both technical and didactic tutors: an emailticketingsystemwillbeimplementedfortechnicalquestions,whileeducationalissueswillbecoordinatedandmanagedwiththesupportofqualifiedexpertsfromeacheducationalsetting.Tosupporte-learningdelivery,ane-learningplatformwillbedevelopedwithMultilingualOpencontents.Informal/nonformallearning:CARESSVirtualCommunityofPracticeThe CARESS Virtual Community of Practice Platform will be integrated with the e-learning platformdeveloped, so as to facilitate the transition of students from formal to informal/non-formal learningactivities.Thegoalof theevaluation is tosupport theCARESSresearchprocess,byknowinghowthedesignof thepilots has been done, how the pilots’ training has been carried out and whether this training programanswers to the purpose of the Caress project. Thus, CARESS pilots will be conducted with the aim ofovercomingthegapsbetweenjobmarketandlearningprogramsfordifferenthealthcarepractitioners.Thecourseswill be evaluated attending its learning design, specific goals, learning outcomes, the quality ofmaterialsandactivities,aswellasteachers’andstudents’satisfaction.Tobeevaluated,eachnationalpilotwill havea referenceperson to collect the informationandmanageevaluationissues.Thispersonisintendedtobelocatedineachtrainingcentre.Themainkeyissueswheretheevaluationwillbefocused,arethosedescribedintheCARESSproject.Theaimoftheevaluationprocessistocollectdata(fromaqualitativeandquantitativepointofview)toanalysethosefactorsandcontributetothepurposeoftheproject.



7.3 EvaluationtaskswithinCARESSproject:generaloverviewInordertobettercontextualisetheevaluationwithoutlosingsightoftheobjectivesofthis,wehavemadeup a table summarising the aims, criteria, sub-criteria and their link with indicators and evaluationinstruments. Moreover, we provide some insight into the status of developing of different evaluationinstrumentswithin CARESS project. Thus, it can see how some instruments are showed as drafts in thesectionofannexesinthisdocumentmeanwhileothers’arestillunderdeveloping.Resuming the different phases of the evaluation process, that will take into consideration the differentcomponents of the CARESS project. The partners will work to evaluate

• (T6.1)theFrameworkandGuidelinesaredeliveredinWP3• (T6.3)thepilotstudies,thoughtthecollaborationwiththeWP8thatwillprovidetheindicatorsof

thequalityassuranceoftheproject• theVirtualCommunityofPractice(T6.3)thatwillbeevaluatedbothduringthepilotactivitiesand

afterthem,monitoringitsefficacyandsustainmentafterthelaunchphase.As described in D3.6 the VCP design process include several steps. To promote the sustainability of theCARESSVCPafterthelaunchperiod,theuserwillbeinvolvedinnewrolesandresponsibilities.TheuseroftheVCPcouldbe involvedassumingdifferentroles includingVCPMemberandVCPSeniorMember(VCPMemberpromotedbytheModeratorortheTechnicalAdministratortobemoderatorofaspecificgroupwithin the community). During the launch phase of the VCP, that will take place during the pilots, thestudentsinvolvedwillbeenrolledintheVCPasVCPmembers.Aftertheendofthepilots,tomaketheVCPself-sustainablethestudentswillbe invitedtoassumetheroleofVCPseniormembersandstimulatetheinvolvementandtheengagementoftheVCPmembers.Thesuccessindicatorsaredesignedalsotodetectthe efficacy of the VCP to be self-sustainable through the involvement of the VCP members in thecoordinationoftheworkofthesubject.The indicators marked in bold (see Table 3) are those collected from the short and long term CARESSoutcomeindicators.ThischartisintendedtobeaguidetothedifferentelementstobetakenintoaccountwithintheevaluationtasksinWP6.

NB:IndicatorsinboldaretheSUCCESSINICATORSoutlinedintheprojectproposal

EVALUATIONOBJECTIVES

CRITERIA SUBCRITERIA INDICATORS EVALUATIONINSTRUMENTS

STATUS TIMINGofEVALUATION

FRAMEWORK

EFFECTIVENESS(T6.1)

Supporttothedesignof compensativelearningmodules

Usefulness of the dataincludedintheframeworktosupportID

- Feedbacksfrompilotdesigners- Quality level of pilots’ IDdocuments

Annex9

Questionnaires andinterviews targeting pilotdesigners tobedefined inT.6.1

DraftD.6.2 During/after pilotsdesign

- OverallqualityevaluationoftheItalianpilot

- Overall quality evaluation ofSpanishpilot

- Overall quality evaluation ofFinnishpilot

Annexes4,5,6,7,8,10,11,

12

Draft.

D6.2

Atthebeginning/endofpilots

Usability of theframeworkfunctionalities (asidetothesupportoftheWizard)

UsabilityofFreeSearch Positive feedback fromrepresentatives of VET providersaboutclarityofusers

Annex9

Questionnaires andinterviews involving usersrepresentatives to bedefinedinT.6.1

DraftD6.2

During/after pilots’design

For final refinement(M34)

Usability of dataintegration


To bedeveloped

During/after pilots’designFor final refinement(M34)

Representativeness Representativeness of EUcountries

- Number of EU Member Statesrepresented in the framework(>75%)

- Coverage of occupations in thefield of homecare defined byESCO(>75%)

Frameworkanalysis Firstrelease(M15)

Finalrelease(M34)

Compliance with EUstandards/tools

- CompliancewithECVET- CompliancewithEQAVET

ECVET/EQAVET Expertsevaluation

To bedeveloped

Firstrelease(M15)

Finalrelease(M34)






WIZARDEFFECTIVENESS

(T6.1)

Supporttothedesignof compensativelearningmodules

Effectiveness of DesignPatterns

- Feedbackfrompilotdesigners- Quality level of pilots’ IDdocuments

Annex9

QuestionnairesandinterviewstargetingpilotdesignerstobedefinedinT.6.1

Draft.6.2 During/after pilotsdesign

Effectiveness of theCARESSIDTemplate

- Feedbackfrompilotdesigners- Quality level of pilots’ IDdocuments

Annex9QuestionnairesandinterviewstargetingpilotdesignerstobedefinedinT.6.1

Draft.6.2 During/after pilotsdesign

Effectiveness of Wizardfunctionalitiesfor ID(Case3)

- Feedback of representatives ofend-users

Interview/questionnairetargeting representativesofend-users

To bedeveloped

After the first release ofCase3


WizardUsability Usability ofRegistration/Personalization

(Case0)

- Feedbackofpilotdesigners- Positive feedback ofrepresentativesofVETprovidersaboutclarityofuse

- CompliancewithEQAVET


To bedeveloped



Usability of Free Search(Case1)




To bedeveloped



Usability of dataintegration(Case2)




To bedeveloped





Usability of Support toInstructional Design (Case3)




To bedeveloped



Representativeness Compliance with EUstandards/tools

- CompliancewithECVET ECVETExpertsevaluation To bedeveloped

Firstrelease(M15)

Finalrelease(M34)

Representativeness of EUcountries

- 4 languages: English, Italian,FinnishandSpanish

Finalrelease(M34)






QUALITYOFPILOTS–(T6.3)

PresenceLearning

Students’performance

The participation andstudents’performance.

- Achievementleveloflearningoutcomes defined accordingtoECVET

Annexes 1, 2, 3,7,8 10,11,12

Draft.6.2 During/ at the end ofpilotsenactment

Students’satisfaction



Annex7,8,10

Draft6.2 During/ at the end ofpilotsenactment

Impact of thelearningprogram

Learning outcomes ofthe pilots fit bettermarketneeds.


- Responsiveness to jobmarketneeds

Annexes8,9 Draft6.2 During/ at the end/six-months posttraining impact, ofpilotsenactment

Quality of learningmaterial

Quality of the learningdesignsandmaterials

- The quality of trainingmaterials

Annexes4,5 Draft6.2 During/ at the end ofpilotsenactment

Teachers’competencies

Teacher’s competenciesandskills.

- Competence andeffectivenessofteachers

- Cost-effectiveness- Presence ofmultiple trainingmethodologies

Annex2,3,9,10 Draft6.2 During/ at the end ofpilotsenactment

Teachers’effectiveness


Annex7,8,11,12 Draft6.2 During/ at the end ofpilots enactment andsix-months posttrainingimpact






QUALITYOFPILOTS(T6.3)

e-learning

Students’performance


- Learningoutcomesdefined ininstructional design documentsareaddressed

- Platformstatistics.- Functionalities of the platform

allow for implementing all theactivities envisaged by theinstructionaldesigndocuments

Analysis of data log filesthat will be trackedautomatically by thesystem

Annex1,2

Annex7,8,10

Draft6.2 During/ at the end ofpilotsenactment

Students’satisfaction The participation andstudents’performance.

- Positive feedbacks of usersaboutclarityandusability

Annex7,10 Draft6.2 At the end of pilotsenactment

Impact of thelearningprogram

Rateof job findingafter6months post training oftargetedstudents

- Learning outcomes of the pilotsfitbettermarketneeds.

- Career improvement/jobfinding of practitionerstargetedbythepilot.

- Responsivenesstojobmarketneeds

Annex11,12 Draft6.2 Six-months post trainingimpact

Quality of learningmaterial

Quality of the learningdesignsandmaterials.

Non-formal and InformalLearning.

- Positive feedbacks of usersabout clarity and usability oflearningmaterials

Annex4,5 Draft6.2 During/ at the end ofpilotsenactment

Teachers’competencies


Sameaspresencelearning Annex2,3,9,10 Draft6.2 During/ at the end ofpilotsenactment

Teachers’effectiveness


Sameaspresencelearning Annex7,8,11,12 Draft6.2 During/ at the end ofpilotsenactment



E-learning platformusability

The uploaded contents(documents or othermultimedia content), theuseofaccessingresources(e.g.,watchingthisorthatvideo);uploadingcontents(this or that document);the active participation incertain communities orvirtual meetings (postingmessages, joining ateleconference).

- Opencontentsavailableforfreeonthee-learningplatform- Platformstatistics


SeepossibleindicatorsinTable4.

During/ at the end ofpilotsenactment

QUALITYOFPILOTS-(T6.3)

CommunityofPractice

Members’performance

Platformstatistics

- Uploadedcontents- The use of accessing

resources- The active

participation incertain communitiesorvirtualmeetings

- NumberofpractitionersinvolvedintheVCP;- Level of activity, i.e. level ofinteractionsamongmembers- Connections with professionalassociations;- Connections with final and endusers


Annex7,10

SeepossibleindicatorsinTable4.

Draft6.2

Attheendofpilots

M34

Members’satisfaction

The effectiveness of VCPthrough the membersatisfaction

- The meeting of the community’sinitialobjectives.

- The added value provided to thetarget as a group, in terms ofnew/enhanced organization, tools,opportunities derived from thebelongingtotheVCP

Annex7,10

Draft6.2 Attheendofpilots

M34



2VCPMemberpromotedbytheModeratorortheTechnicalAdministratortobemoderatororaspecificgroupwithinthecommunity

Seniors members2activityeffectiveness

The scaffolding providedby seniors members andmoderators

Presence or not of the followingqualitycriteria:

- Senior members provide withincentives to increase theparticipation

- Senior members promotemotivating situations within theVCPtoincreaseparticipation

- Senior members provide withusefultools

- Senior members provide withnetwork situations andopportunities to increase andmotivatethe internalandexternalparticipation

- Interactions and networks arestimulatedbothatanationallevel,with the support of nationalpartners,andatinternationallevel

- Inter-professional exchanges andcontacts are being fosteredthrough theVCPparticipation andthecreationofinterest’sgroup

- The senior member promotes,motivateandmoderatethegroupshim/herhaveassigned


To bedeveloped

M34






QUALITYOFPILOTS-(T6.3)

CommunityofPractice

VCP Platformeffectiveness

The scaffolding providedby platformstructure/functionalities

Presence or not of the followingqualitycriteria:

- Theplatformprovidesuseful toolsenhancing participation andnetworking

- The platform allows for aneffective management ofinteractions and networks both ata national level at internationallevel

- The platform allows for aneffective management of inter-professional exchanges andcontacts

Attheendofpilots

M34

VCP platformusability

Positivefeedbacksofusersaboutplatformusability

- Functionalitiesoftheplatformallowfor implementing all the activitiesenvisaged by the instructionaldesigndocuments.

Questionnaire/

Interview.

To bedeveloped

During/ at the end ofpilotsenactmentandsix-monthsaftertraining

Effectiveness ofcompetencerecognitiontools

Effectiveness ofDiary/portfolio

- Achievement level of learningoutcomes defined according toECVET- Thebenefitsprovidedtoindividualssuch as learning/job/networkingoccasions derived from thebelongingtotheVCP

Annex11,12 Draft6.2 During/ at the end ofpilotsenactmentand sixmonthsaftertraining

Effectiveness of tracking/self-reporting

Achievement level of learningoutcomes defined according toECVET


To bedeveloped

During/ at the end ofpilotsenactmentandsix-monthsaftertraining

Table3.GeneraloverviewofevaluationtaskswithinCARESSproject

8 CONTEXTANDEVALUATIONMETHODINT6.3Withtheaimofassessingtheevaluand,threestudiesinauthenticscenarioswillbecarriedoutfromMay2017untilFebruary2018.ThesestudieswereformalisedusingtheEREMevaluationmodel(seesection7)andinvolvedrealendusers,likeeducatorsandstudents.Weaimtoadeeperunderstandingoftheparticularphenomenaunderstudy, inourcase,thedesignandtheenactmentofthreenationalpilotsinVETtofosterstudentsofnursing,OSSandpracticalnurse’sskillsandcompetences inelderlyhomecare.Besides, thisevaluationproposalwill showhowteachers supportlearningactivities in face to face,on-lineandnon-formal and informal courses, andallows todetect thebenefitsandlimitationstoassessthequalityandtheefficacyoftheenactedpilots.The aforementioned issue 2 (Can training courses based on competences and active methodologies behelpfulto improvenurses/OSSandpracticalnursesclinicalpracticeandtransversalskills inelderlycare?)willbeexploredthroughinformativequestionsgroupedintoseveraltopics,bothdefinedbyusatthestartoftheevaluation.Alsoemergenttopicswillbetakenintoaccountwhilegatheringandanalysingdata,ina“progressivein-focusapproach”(Stake,2010).Thesetopicscanbeillustratedbythefollowing:Topic1:InstructionaldesignoftheVETcourses:toevaluatethedesignofthecourse,weaimtoanalysetheinstructionaldesigndocumentsforformal,non-formalandinformallearningaccordingtothenationalVET pathways targeting Social Health Operators, practical nurses and registered nurses. Thus, in thisparticularway,weneedtoknowwhetherthecontentsarewell-suitedtothecourse’sobjectives.Also,wewillgatherinformationabouttheinstructionalstrategiestobeused,aswellastowhatextentthedesignofthecoursesacrosscountries isgoing to followtheEQAVET indicators.Finally,wewill collect informationaboutstudents’ impressionsaboutthedesignofthecourse, their learningexpectationsand if thedesignandthecontentsofthecoursearewellsuitedtotheneedsoftheworkmarket.Topic 2: Participation and students’ performance: we pretend to gather information regarding theenactment of the pilots. We will ask students about the course contents, activities, methodologiesemployedaswellastheirperceptionsabouttheexperienceandsatisfactionwiththecourse.Furthermore,wewillfocusonunderstandingifthecoursesperformedhavecontributedinsomewaytoovercometheirlearninggaps.Wealsowillaskaboutthecourseutilitytocovertheneedsperceivedintheirtraining/job.Ontheotherhand,wewillgatherdataabouttheteachers’perceptionsofthepilotcarriedout.Topic3:Teacher’scompetenciesandskills,wherewepretendtoputemphasisinevaluatingspecificissuessuchastheadequateknowledgeonthesubjectshowedbyteachers/tutors,theclarityoftheexplanations,theirlevelofpreparationofatopic,theirinteractionswithstudentsregardingquestionsandissuesraisedduringthetraining,etc.Topic4:Qualityandusefulnessoftrainingmaterials:Qualityandusefulnessoftrainingmaterials:thatwillbe used in the training courses from the participants and the teacher's point of view in the learningexperiences.



Topic5:NonformalandinformallearningthroughCARESSVCP:avirtualcommunityinhealthcarerefersto a “group of people (and the social structure that they collectively create) that is founded ontelecommunication with the purposes of collectively conducting activities related to health care andeducation” (Demiris,2006,p.179). Suchactivitiescanbesupportedbyaplatformaimedatallowhealthprofessionalstodiscussonelderlycareissuesandproblems,sharingdocumentsaswellasconsultingwithexpertsandsustainingrelationshipsbeyondface-to-faceevents.CARESSVCPplatformwill supportnon-formal and informal learning recognitionaccording to threemaintypesofevaluationtoolsandfunctionalities(seeD.3.6):a)Usertrackingfunctionalities:wewillgather informationcomingfromtheanalysisofdata logfilesthatwillbetrackedautomaticallybythesystem.Thisway,weobtaineddataforevaluationpurposesfromthefinalusersoftheCARESSe-learningplatformandVCPusers.Itisworthnoticingthatdatacouldbeexportedinstandardformatswiththeaimofmakingitaccessibletothepilots’teacherstohelpthemintheidentificationofcompetencerecognition.Apotentialtrackingdatacouldbetheuploadedcontents(documentsorothermultimediacontent),theuseofaccessingresources(e.g.,watchingthisorthatvideo)andtheactiveparticipationinparticularcommunitiesorvirtualmeetings(e.g.,postingmessages,joiningateleconference).Table 4 illustrates some potential tracking data that could be obtained from the use of the e-learningplatformaswellasfromtheVCP.Furthermore,annex13providesadetaileddescriptionaboutthetrackingindicatorsthatcanbeobtainedfromtheCARESSe-learningplatform.

Trackingindicators VirtualsettingVirtual Community ofPractice

e-learningplatform

Uploadedcontents(documents,multimedia,etc.) x Accessingresources(numberofvisits,lastaccessed)

x x

Postingmessages x xJoiningateleconference x Activitysubmissions xNºofEventsubscriptions x Enrolledstudents x xBadgesachieved x Create/Joinagroup x Inviteuserstoagroup x Nºofsearchings x Nºofinvitationstojoinagroup x Nºofcreateddiaries x Nºuploadedcontents(documents,multimedia,etc.)inthegroupmoderated

x

Accessingresources(numberofvisits,lastaccessed)tothegroupmoderated

x



Nºofusersthatjoinedagroup x Nºofteleconferencecoordinatedinagroup x NºMessagesPostedinagroup x NºActivitysubmittedinagroup x NºEventsubscriptedinagroup x

Table4.TrackingdatafromVCPandCARESSe-learningplatform

b)VCPexperiencediary/portfolio: is anorganised collectionofmaterials that presents and verifies skillsandknowledgeacquiredthroughexperience.AccordingtoCedefop,2015,“Itisofparticularrelevanceforvalidatingnon-formalandinformallearningbecauseitallowstheindividualcandidatetocontributeactivelytothecollectionofevidenceandalsooffersamixofapproachesstrengtheningtheoverallvalidityoftheprocess”(p.61).Aportfoliomightincludedocumentssuchasabstracts,performanceappraisals,referencefromcurrentandpastemployers,etc. Inthecontextofthisevaluation,anexperiencediary/portfoliowillbeused toallowstudentsdescribehis/her thoughtsandexperienceswithin theVCP (e.g. lessons learnt,discussions,createdcontacts,activityreflections,etc.)AproposalofthistoolwillbepresentedinD.5.1.2“IntegrationofVCPplatformwithe-learningplatform”.Thistoolwillbeusedforevaluationpurposeswiththeaimofcollectingdatafromfinalusersabouttheirpersonalexperiencesrelatedtoelderlycareinanon-formalandinformalcontextaswellastoassesstheircompetenceacquisition.TheVCPexperiencediarywillallowforapersonaldescriptionoftheuserexperienceasamemberoftheVCP.TheVCPmembercouldfill-inperiodicallyadiarywherehe/shecoulddescribehis/herownexpertiseintheVCP(e.g.,activities,discussions,lessonlearnt,createdcontacts,etc.).Theexperiencediarycouldbeprovidedwith questions or suggestions to encourage the textual description of the personal experienceandthenewknowledgeachieved.Thechronologicalversioningoftheresultingdiaryshouldbetrackedandpresentedinaformattoallowtobeanalysedforevaluationpurposes.Besides,itispossibletocreatesometemplatesintheVCPthatmaymakeiteasiertodescribetheuser’spersonalexperience.Also,thesoftwarethatwewillusefortheplatformallowstheissuingof“badges”totheparticipantswhencarrying out specific tasks in the platform. This can be used for generating an “activity report” of theparticipantthatcanbecomplementedwithinformationprovidedbytheparticipantthroughtheirreflectivediary, information that couldbe labelled tohelp furtheranalysis. In addition, the categories let theuserfilter the entries depending on different categories. These categories could describe just personalexperiences or could be aimed at competencies recognition. The use of a blog-like activity (where theparticipants would provide links to, e.g., the artefacts created during their interaction with the VCPplatform)couldhelptoanalyseaparticularsuccessindicatorfortheVCP.c)VCPusers’acceptanceandusage:forevaluationpurposes,wewillusetheTechnologyAcceptanceModel(TAM)(Davis,1989)toassessabouttheusersCARESSVCPperceivedusefulnessaswellastheirperceivedease of use. The Technology Acceptance Model was developed to predict and explain, informationtechnology acceptance and usage. Themain topics to be look at according with thismodel are systemquality, service quality, and perceived usefulness, perceived ease of use and user satisfaction with thesystem.



Moreover, we will use the “Think Aloud Protocol (TAP)” as a research strategy commonly used inpsychologicalandeducationalresearch(Lewis,1982;Charters,2003).TAPprotocolisdevotedtoinvolvingparticipantsthinkingaloudmeanwhiletheyareperformingasetofspecifiedtasks.Participantsareaskedtosaywhatevercomesintotheirmindastheycompletethetask.Thismightincludewhattheyarelookingat, thinking, doing, and feeling. This gives observers’ insight into the participant's cognitive processes(rather than only their final product), to make thought processes as explicit as possible during taskperformance. To provide evidence of the user's acceptance and usage of the CARESS VCP partnersresponsible for the evaluation issues will need to design a process, which takes into account someconcerns,byselectingasuitabletask.Accordingtotheformalresearchprotocol,allverbalizationswillbetranscribedandthenanalysed.It is worth notice that mix method will be used as a strategy for collecting information from differentsourcesanddifferentmoments.The finalaim is tohaveamorecompletepictureof thePilotsworkandtheir strengthsandweaknessesasaneducationaltoolformanagingtheCARESSprojectgoalofovercometheskillmismatchinhealthcareprofessionals.Figure 3. Connections between data gathering techniques, evaluation happenings and topics.

According to the aforementioned ideas, the EREM highly encourages to use mixed data gatheringtechniques and multiple informants in order to obtain different perspectives about the evaluand, thusenriching theevaluationprocess (Green,Benjamin&Goodyear, 2001). Specifically, in this evaluationwe



will combinequantitativeandqualitativedatacoming from4 typesof informantsandsources (teachers,students, CARESS Virtual Community of Practice and CARESS e-Learning platform), using different datagathering techniques such as interviews, questionnaires, log file analysis, observations and reflectivediaries,aswecanseeinFigure3.Then,evidencecollectedfromdifferentsourcesand informantswillbeanalysedfollowingatriangulationapproach (Guba, 1981), in order to reach the global conclusions of this evaluation, increasing thetrustworthinessofthefindings.Likewise,wewillperformananalysisofthecontentofthedocumentsprovidedbythepilots’instructionaldesigners, the transcriptions of the interviews, the reflective diaries, the observations of the pilots’enactment and the open questions coming from the questionnaires. A quantitative analysis will beperformedusingdatacomingfromtheLikertquestionnairesaswellasfromthelogfilesobtainedfromtheCARESSVCP,asitisillustratedinFigure4.

Figure4.GraphicrepresentationofthedatagatheringandanalysistechniquesflowduringtheevaluationofCARESSpilotsinVET.

Thesupportingtechnologiesthatwillbeusedduringfacts(e.g.teacherdesignsandenactmentsoftheVETpilots)willincludeNVIVOqualitativeanalysissoftware(QSR,2008),MSExcelforquantitativeanalysisandGoogleformsforweb-basedquestionnaires.

8.1 DescriptionofthetimelineandtheevaluationinstrumentsBeforewedescribetheevaluationmethodologythatwilltakeplace,wethinkitisimportanttogobrieflyovertheevaluationphasesandprovidedageneraloverviewoftheassessment instrumentsthatwillbeused within the evaluation of CARESS pilots in VET. Furthermore, in this section, we identify a set ofevaluationresponsibilitiesthatmightbeassumedbyCARESSpartnerswithdeclaredeffortinWP6.AsitisillustratedinFigure3,theCARESSevaluationofVETpilotswillbedividedintotwomainmoments.Afirstevaluationhappeningcalled[H1]:Design,wherewewilllookatteachers’coursedesigns,thatwillinformtheDesignofthePilots.Thisphaseispriortothepilots’launch.Asecondevaluationhappeningcalled[H2]:Enactment,whilethePilotsarealreadycarriedout.Inturn,itisdividedinthreemainmoments:

a) The beginning of the Pilot. The evaluation team will collect a student questionnaire, a courseobservationandateacherinterview

b) Midtermof the Pilot. During this period, a classroomobservationwill take place for each Pilotcourse.Also, a TAP (ThinkingAloudProtocol) questionnaire regarding theuseof theE-learningplatformandtheVCPwillbefilledbytheevaluators.

c) FinalofthePilot.Wewillcollectthetrainingmaterials,teacher’sfinalreflectionsdiary,students’and teachers’ final questionnaire, and a final teacher interview. Additionally, we will collect astudentdiaryregardingtheusetheVCP.

Figure 5 illustrates a third evaluation happening [H3]: Post training, in which we will collect data toevaluate the impact after six-month training. This way, we will perform a student's interview and aquestionnaire with the aim of assess the impact of the pilots training considering the professionals’satisfaction,jobimprovementsandcareersteps.

Figure5.Connectionsbetweendatagatheringtechniquesandtopicsduringpost-trainingevaluationphase.



Through all those moments, the teachers and students’ log files will be collected to evaluate the e-learning module and the VCP use. Furthermore, statistics will also derive from the online communitythroughwhichteacherswillbeabletocommunicateandinteract,inordertohaveadescriptiveaccountofparticipants’contributions.Table5summarizesthedatagatheringtechniques,informants,topicsandtimeswhentheinformationwillbecollectedaccordingtoFigure3and4.

DATAGATHERINGTECHNIQUES

INFORMANTS TOPICS HAPPENINGS

[Diary1]

Teachers Design H2:EnactmentFinal

[Diary2] Students VCP H2:EnactmentFinal

[Art1][Art2]

Design H1:Design

QualityofTrainingMaterials H2:EnactmentFinal

[Quest1] Students Design H2:EnactmentBeginning

[Quest2][Quest3]

StudentsTeachers

Participationandstudent’sperformanceTeachers’competencyandskillsQualityoflearningmaterials

H2:EnactmentFinal

[Quest4] StudentsTeachers

VCP H2:EnactmentFinal

[Quest5] Students Posttrainingimpact H3:Sixmonthsposttraining

[Ob1][Ob2]

TeachersStudents

Participationandstudents’performanceTeachers’competencyandskills

H2: Enactment Beginning andMidterm

[Int1][Int2]

Teachers

DesignQualityofLearningmaterialsParticipationandstudent’sperformanceTeachers’competencyandskills

H2: Enactment Beginning andFinal



[Int3] Students Posttrainingimpact H3:Post-training

[LogFiles] TeachersStudents

VCPe-learningplatform

H2: Enactment Beginning,Midtermandfinal

[TAP] StudentsTeachers

VCP H2:EnactmentMidterm

Table1.Evaluationinstruments,informantsandhappenings

8.2 Whowillcollectthedata?Therewillbeonepersonresponsibleforeachpresencelearningcourse(VittorioEmanueleII,OMNIAandthe Nurses School of UVA) in charge of collecting the data required using the evaluation instrumentsprovided in this document, according toa time schedule theevaluation coordinatorswill beproviding.The e-learning log files will be collected by GGallery, along with the VCP’s feeding. One responsiblepartnerfromGGalleryandSI4Lifewillbethereferencecontactforevaluationpurposesofthee-learningevaluationandtheVCPs.Eachpersoninchargeoftheevaluationineachcountrywillberesponsibleofprovidingthedatacollectedbytheevaluationinstruments,sotheevaluationcoordinatorscananalyseandwritethereportwiththefinal results. Besides, to facilitate the analysis, we will ask the partners to send the data required inEnglish.Furthermore,membersofAGEwillmakeatleasttwoobservationsofthepilots’enactmentandtheywilltransfertheresultstothecoordinatoroftheevaluationworkpackage.All the instruments showed in the appendix are an initial proposal. The primary structure will bemaintained,althoughtheinstrumentsmayberefinedoradaptedwiththehelpoftheinvolvedpartners’feedback,andoncethecontext’scharacteristicswheretheevaluationwillbetakenwillbebetterknown.Table 6 illustrates the responsible partners according to the different evaluation tasks identifiedwithinthisworkpackage.



Evaluationtarget Responsiblepartners Tasks

Presence learning:designandenactmentofCARESSVETpilots.

One person responsible foreach presence learningcourse:

·-VE-II·-OMNIA·-NursesSchoolatUVa

-Collaboration in the definition and refinement of thedatagatheringinstrumentsavailableintheappendixofthisreport.-Translate the evaluation instruments into thecorrespondinglanguages(Spanish,ItalianandFinnish)-Completetheevaluationtoolkit-Sendthe information inEnglishtothecoordinatorsofthisWP.

On responsible person byAGEplatform

-Performat leastoneobservationof theenactmentofeachpilot-Sendthe information inEnglishtothecoordinatorsofthisWP.

One responsible person byOMNIA

-Identifyappropriateindicatorstoevaluatethetrainingmaterials from the pilots according to EQAVETframework.

e-learningcourses One responsible person byOMNIA

-Identifyappropriateindicatorstoevaluatethetrainingmaterials from the pilots according to EQAVETframework

One responsible person byGGalleryOne responsible person bySI4LIFE

-Identify useful indicators coming from the analyticsobtainedfromtheuseofthee-learningplatform.-Report the data coming from the e-learning platformanalyticstothecoordinatorofthisWP.

Virtual Community ofPractice(VCP)

One responsible person byGGallery.OneresponsiblebyUVa-Tech.One responsible person bySI4LIFE.

-Identifyuseful indicatorscoming fromtheanalyticsofthe VCP to assess the final user’s performance withinthisvirtualspace.-Report the data coming from the analysis performedbythecoordinatorofthisWP.

Table2.Evaluationresponsibilities



8.3 DraftoftheevaluationtoolkitinT.6.3The resources available for the evaluation will be the instructional designs and the trainingmaterials.Also,wewilllookatthefinaleducationalplatform,whereweexpecttocollectdataabouttheeducationaland pedagogic process, along with the pilots’ designs. The teacher’s actions and reflections, and thestudents’actionswillprovidedataandmaterialtoevaluatemeaningfulinputs.Themainelementsdesignedfortheevaluationkitare:

•Guidelinestoperformtheobservations•Student’sandteacher’squestionnaires•Guidelinestoperformthestudentsandteachers’interviews•Guidelinestocompletethestudent'sandteacher’sdiaryreflection•Questionnairestoevaluatethetrainingmaterials

Thoseelementswillbeapplyingtoin-personcourses,thee-learningplatformandthevirtualcommunitiesofpractice.Thefollowinginstrumentsaredesignedtobeusedfordatacollectionpurposes.Thetopicstobeanalysedforevaluationpurposesare:

● InstructionaldesignoftheVETcourses.Thisdesignwillbeinformedmainlythroughtheteacher’scoursedesign,theanalysisofthetrainingmaterialsandthefinalteacher'sreflectionsaboutthedesign and the enactment of the pilot. Some aspects of this topic may be enriched by datarecorded for other topics, like the participation and students’ performance, the teachercompetenceandskillsandthequalityoftrainingmaterials.

● Theparticipationandstudents’performance.Thistopicwillbeilluminatedthrough:

○ Thestudents’previousskillsandmotivation○ Thecontentandstructureofthelearningcourses○ Theoverallstudents’satisfactionandtheimpactofthelearningprogram

Wewillgatherdatathroughstudentsandteacher’squestionnairesatthebeginningandtheendof theenactmentphase.Also,datawillbecompletedwith reflectionscollectedwhenanalysingthequalityoftrainingmaterialsandtheteachersandstudentslogfiles,regardingtheuseofthee-learning platform and the VCP. The courses’ activities and knowledge acquisition will beevaluatedthroughobservations,teacherinterviewsaswellasbytheemploymentoftheteacherreflectivediary.

● Teacher’s competencies and skills. This evaluation topic will be observed through courseobservations in the classroom, but also analysing the student’s participation and performance.Teacher interviews, final students’ questionnaire, classroom observation and teacher reflectivediarywillbethemaintoolsfordatacollectingforthisevaluationtopic.



● The quality of training materials. Directly linked with the design of the course, final students’

questionnaire,teacherinterview,questionnairesforcourseproducts’evaluationandlogfiles,willprovideuswithevidenceaboutthisissue.

● NonformalandInformalLearning.Thiswillbeevaluatedthroughtheteachers’andstudents’log

files regarding theuseof e-learningplatformand theVCP, the student's’ diary collectedat theendoftheenactmentandtheuseofaThinkAboutProtocol(TAP)withtheaimofevaluatingtheusabilityandacceptabilityoftheVCP.Regardingthistask,thefinalevaluationtoolswillbefullydevelopingoncetheinformationabouttheVCPhavebeencompletedandallthefunctionalitiesandpotentialdatatobeextractedforevaluationpurposeswouldbeanalysed.

● Post-trainingimpact.ThiswillbeevaluatedsixmonthsafterthecompletionofthePilotandwill

helpustoassesstheimpactofthetraining.Wewilluseastudentpost-trainingquestionnaireandaninterviewtogetdatatoilluminatethistopic.

9 QualityAssuranceandOverallProjectEvaluationAccording toRIGA conclusions (June, the22ndof 2015), andaspartof quality assurance systems, it isimportant to establish continuous information and feedback loops in VET systems based on learningoutcomes.ThatwouldincludetheuseofinformationonVETgraduateemployabilityandacombinationofdata on learning, labour market entry, and career. Moreover, the use of outcomes of VET should besystematically used as part of quality assurance systems. As the RIGA document states, to delivervocational curricula, would be necessary to track learners after completing their training, through theidentificationofadequatemeasures,inordertoprovide“feedbackloops”.Therehavebeenalready someexperiencesabout tracking students (e.g. thedevelopmentofGraduateTrackingSysteminCroatia,2013-2016.In2013,theCroatianMinistryofEducationbeganapilotprojectwith theaim to collectexistingdataongraduate students,employedandunemployedpersons). Thesetrackingcouldbedonecombiningadministrativedatawiththeresults,sometimeseasiesttoget,throughtargetedsurveys,feedbacksheetsorcourseevaluations,face-to-facecontactorfocusgroupinterviews.Trackingcanbecomplementedwithothermeasuressuchas interviews, focusgroupmeetings, studentfeedbacksheets,etc.Tracking Learners’ and Graduates’ Progression Paths (2012), states “Tracking should lead to theenhancementofteachingandlearning,andbelinkedtostudentservices.Relevantresultswouldalsobemade available to staff and students, prospective students, policy makers and employers” (pg. 51).Employment ratesofgraduateswithinaspecifiedperiodaftergraduationandstudent feedbackon thequalityofstudyprogrammesandfacilitiescouldprovidefeedbacktoimproveandconfirmthequalityofVETprograms.TheinformationcollectedfromthedifferenttrackingactivitiesdesignedwithintheCARESSVirtual Community of Practice could be combined and used in this “feedback loop”. Thatwill allowussystematicallymonitorthestudentslearningoutcomesanditseffectiveness.



Thismonitoringwill take place during the early period after the conclusion of the Pilots and over thetargeted students, during a period of six months. Those students will be tracked using the e-learningplatformandVCPdata,togetherwithstudents’interviewsandquestionnaires.Thefollowinganalysiswillallowustoprovidetheprojectwiththisfeedbacktocontributetotheimprovementofthequalityofthehomecareprogram.

10 BIBLIOGRAPHY

● Abma,T.A.(2001).Evaluatingpalliativecare.Facilitatingreflexivedialoguesaboutanambiguousconcept.Medicine,HealthCareandPhilosophy,4(3),261–276.

● Cedefop(2015).Europeanguidelinesforvalidatingnon-formalandinformallearning.Availableathttp://www.cedefop.europa.eu/en/publications-and-resources/publications/4054 . Last visited2017/04/27.

● Charters, E. (2003). The use of think-aloudmethods in qualitative research an introduction tothink-aloudmethods.BrockEducationJournal,12(2),68-82.

● Curran,V.,Christopher,J.,Lemire,F.,Collins,A.,&Barrett,B.(2003).Applicationofaresponsiveevaluationapproachinmedicaleducation.MedicalEducation,37(3),256-266.

● Demiris,G.(2006).Thediffusionofvirtualcommunities inhealthcare:conceptsandchallenges.Patienteducationandcounselling,62(2),178-188.

● Gaebel,M.,Hauschildt, K.,Mühleck, K. and Smidt,H. (2012) Tracking Learners’ andGraduates’Progression Paths. TRACKIT. European University Association. Available athttp://www.eua.be/Libraries/publications-homepage-list/EUA_Trackit_web.pdf?sfvrsn=2 Lastvisited2017/05/10.

● Greene, J.C.,Benjamin, L.,&Goodyear, L. (2001). Themeritsofmixingmethods inevaluation,7(1),25-44.

● Guba,E.G.(1981).Criteriaforassessingthetrustworthinessofnaturalisticinquiries.EducationalTechnologyResearchandDevelopment,29(2),75-91.

● Jorrín-Abellán,I.M.,&Stake,R.E.(2009).DoesUbiquitousLearningCallforUbiquitousFormsofFormalEvaluation?:AnEvaluandorientedResponsiveEvaluationModel.UbiquitousLearning:AnInternationalJournal,1(3).71-82.

● Koch, T. (1996). Implementation of a hermeneutic inquiry in nursing: Philosophy, rigour andrepresentation.JournalofAdvancedNursing,24,174–184.

● Koch,T.(2000).“Havingasay”:Negotiationinfourthgenerationevaluation.JournalofAdvancedNursing,31(1),117–125.

● Lewis,C.,&Mack,R. (1982). Learning tousea textprocessingsystem:Evidence from“thinkingaloud”protocols.InProceedingsofthe1982conferenceonHumanfactorsincomputingsystems(pp.387-392).ACM.

● Wadsworth,Y.(2001).“Becomingresponsive-andsomeconsequencesforevaluationasdialogueacrossdistance”.NewDirectionsforEvaluation.92.SanFrancisco:JosseyBasspp.45–58.



● Widdershoven,G.A.M.,&Sohl,C.(1999).Interpretation,actionandcommunication:Fourstoriesabout a supported employment program. In T. A. Abma, Telling tales. On narrative andevaluation.AdvancesinProgramEvaluation(6)(pp.109–130).Connecticut:JAIPress,109–130.

● RIGA conclusions, on a new set ofmedium-termdeliverables in the field ofVET for theperiod2015-2020, as a result of the review of short-term deliverables defined in the 2010 BrugesCommuniqué. Available athttp://ec.europa.eu/dgs/education_culture/repository/education/policy/vocational-policy/doc/2015-riga-conclusions_en.pdf.Lastvisited2017/05/10.

● Stake,R(2004).Standards-basedandresponsiveevaluation.ThousandOaks,CA:Sage.● Stake,R.E.(2010).QualitativeResearch:StudyingHowThingsWork.NewYork,NY:GuilfordPress

11 ANNEX11.1 Annex1:Guidelinestoperformtheobservations(Draft)

Date:___________________Location:_________________________

Instructor(s)evaluated:________________________________________

Course:_____________________________________________________

Class:________Numberofstudentspresent:_____________________

Observer:___________________________________________________

Instructions: This is an open-ended tool. The observer should take notes regarding the activity that isimplementedduring the lesson. Someof the areasonwhich theobserver should focus arementionedbelow.Feelfreetomodifytheseareasaccordingtotheprogressofthelesson/activity.(a)Abouttheteacher

1. Describetheteachingmethodthattheteacheruses.

2. Makeareferenceontheteachermainskillsthatareusedduringthelesson/activity.

3.Describetheclassroom’sorganization.

4. Observeandtakenotesregardingtheflowofthelesson/activity.

5. Describeteacher’sflexibilityindealingwithunexpectedsituations.



6. Indicateteacher’sfeedbacktothestudents.

(b)Aboutstudents

1. Howdostudentsuseandinteractwiththeteacher’spedagogicalmaterials?

2.Describetheiractivities.

3.Describetheirbehaviour.

4. Describethestudents’skillsthataredevelopedduringthelesson.

Time Data(Whathappensnow?) Interpretations/Comments

00:00

00:05

00:10

00:15

00:20

11.2 Annex 2: Guidelines to complete the teacher’s Reflective Diary(Draft)

Country:

Teacher:

Course:

Starttime:

Endtime:

Date: Namethreethingsthatimpressedyouthemostaboutthislesson:_______________________________________________________________________________________________________________________________________________________________________________________________________________



Namethreedifficultiesthatyoufaced: _______________________________________________________________________________________________________________________________________________________________________________________________________________ Whathasworkedwellduringthelesson? _______________________________________________________________________________________________________________________________________________________________________________________________________________ Additionalcomments: _______________________________________________________________________________________________________________________________________________________________________________________________________________

11.3 Annex3:Guidelinestoperformtheteacher’sinterview(Draft)Annex3A.InitialInterview. 1.Whatisyourmainmotivationregardingtheteachingofthiscourse?

2.Areyoupleasedwiththeinstructionaldesignprovided?

3.Doyouthinkitwillbeeasytoimplementtheinstructionaldesign?

4.Whatisthebiggestdifficultyyoufaceregardingthedesignofthispilotcourse?

5.Whatdoyoubelieveisthebiggestadvantageofthisthepilotcourse?

6.Haveyouanypreviousexperienceinteachinghealthcare?

7.Doyouthinkyourcompetencesandexperiencearesufficienttoteachthistrainingcourse?

8.Whatisyourmaingoalregardingtheteachingofthispilot?

9.Howfamiliarareyouwithevaluatingyourstudentsintermsoftheirlearningoutcomes?

10.Inyouropinion,whichwillbethedifficultiesthatthestudentswillfaceduringthepilotcourse?Name3

11.Whatdoyouthinkarethestrongpointsofthelearningdesign?Andtheweakpoints?

12.Isthereanythingelseyouwouldliketosharewithus?

Annex3B.FinalInterview.



1.Didyouconsiderthetrainingpilotcoursehelpyoutofillsomeneedoftraininginthefieldofhomecareeducation?

2.Wereyoupleasedwiththeinstructionaldesignprovidedduringthetrainingcourse?

3.Wasiteasytoimplementtheinstructionaldesign?

4.Whatwasthebiggestweaknessofthepilotcourseenactment?

5.Whatwasthebiggestadvantageofthepilotcourseenactment?

6.Didthestudentsseemtobehappy/excitedduringthecourse?

7.Describethereactionsofthestudents.Whatdidtheysayduringandattheendofthecourse?Whatdidtheycommenton?

8.Howwouldyouevaluateyourstudentsintermsoftheirlearningoutcomes?

9.Whichwerethedifficultiesthatthestudentsfacedduringthepilotcourse?Name3.

10.Whatdoyouthinkthatcouldgobetter?

11.Whatwouldyouchangeifyoucouldimplementthepilotcourseforasecondtime?

12.Isthereanythingelseyouwouldliketosharewithus?

11.4 Annex 4: A questionnaire to evaluate the training materials bythestudents(Draft)

Student/Evaluator:_________________________________Date:_____________________________________________Instructions: Grade the items below on a scale from 1-5. To what degree does the training materialsdemonstrate the following? Please, take in account the training materials regarding the in-persontraining,e-learningandVirtualCommunitiesofPractice.Q:Towhatextentdoyouagreewiththefollowingcriteriaregardingtheproductofthepilotcourse? Criteria Completely

disagree Partiallydisagree

Neitheragreeordisagree

Partiallyagree

Completelyagree

1 2 3 4 5

Content

Coursehaveacurrentcontent.

Coursehaveanaccuratecontent.



ContentsupportsrelevantknowledgeinsocialandhomecareinVET.

Scope(range)anddepthoftopicsareappropriatetostudent’sneeds.

Thelevelofdifficultyisappropriateforintendedaudience.

Content materials consider andintegratework-basedlearning.

Contentmaterialshelpstudentstoapplypracticalknowledgeinworkbasedlearningenvironments.

Instructionaldesign

Instructional goals and learnerobjectivesareclearlystated.

Materialsaresuitableforawiderangeoflearning/teachingstyles.

Learningmaterialspromotesstudentengagement.

Learningmaterialsfostergroupinteraction.

Learningmaterialsencouragesstudentcreativity.



Learningmaterialsallows/encouragesstudentstoworkindependently

Learningmaterialsaresuitableforintendedpurpose.

Materialsarewellorganizedandstructured.

Materialshaveunity/congruency.

Technicaltermsareconsistentlyexplained/introduced.

Adequate/appropriateassessment/evaluationtoolsareprovided.

Technicaldesign

Learningmaterialsarevisuallyattractive.

Illustrations/visualsareeffective/appropriate.

Charactersize/typefaceisappropriate.

Studentscaneasilyemploytheresource.

Thelearningmaterialsarewellexplainedandinaclearlanguage.

Learningmaterialsarepresentedinadifferentformat(i.e.videos,documents,webpages,software,openeducationalresources,ICTtools)



Inclusion/socialconsiderations

Thelearningmaterialstakeintoaccountdifferentwaysoflearningbythestudents.

Learningmaterialstakeintoaccountstudentspreviousknowledgeandlearningoutcomes.

Learningmaterialsarewellsuitedtothestudents’diversity.

Learningmaterialsestablishdifferentdifficultygradesaccordingwiththediversityofstudents.

Workoriented

Learningmaterialsopportunitiestodevelopentrepreneurialmindsets.

Learningmaterialsintegrateopportunitiesofknowinglabourmarketneedsandskills.

Learningmaterialsarerelevantforthestudent’sfuturejobsopportunities.

Educational/multimediaandmanagementaspectsrelatedtoe-learningandVCPmaterials

Thedesignisuser-friendly.

Thetextisclearandeasytoread

Thesoftware/webpages/videosarelogicallyorganized



Navigationisclearandconsistent

Thescreenisuncluttered

Navigationisuser-controlled

Littlehelpisrequiredtoworkindependently

Thegraphicsandsoundarewellused

Thegraphicsarenotoutdated.

Graphicsandaudioarenotdistractingelements

Thankyouforyourparticipation!

11.5 Annex 5: Questionnaire to evaluate the training materials byTeachers(Draft)

Teacher/Evaluator:__________________________________ Date:______________________________________________ Instructions: Grade the items below on a scale from 1-5. To what degree does the training materialsdemonstrate the following? Please, take in account the training materials regarding the in-persontraining,e-learningandVirtualCommunitiesofPractice. Q:Towhatextentdoyouagreewiththefollowingcriteriaregardingtheproductoftheworkshop?



Criteria Completelydisagree

Partiallydisagree

Neitheragreeordisagree

Partiallyagree

Completelyagree

1 2 3 4 5

Content

Coursehaveacurrentcontent.

Coursehaveanaccuratecontent.

ContentsupportsrelevantknowledgeinsocialandhomecareinVET.

Scope(range)anddepthoftopicsareappropriatetostudent’sneeds.

Thelevelofdifficultyisappropriateforintendedaudience.

Content materials consider andintegratework-basedlearning.

Contentmaterialshelpstudentstoapplypracticalknowledgeinworkbasedlearningenvironments.

Instructionaldesign

Instructionalgoalsandlearnerobjectivesareclearlystated.

Materialsaresuitableforawiderangeoflearning/teachingstyles.



Learningmaterialspromotesstudentengagement.

Learningmaterialsfostergroupinteraction.

Learningmaterialsencouragesstudentcreativity.

Learningmaterialsallows/encouragesstudentstoworkindependently.

Learningmaterialsaresuitableforintendedpurpose.

Materialsarewellorganizedandstructured.

Materialshaveunity/congruency.

Technicaltermsareconsistentlyexplained/introduced.

Adequate/appropriateassessment/evaluationtoolsareprovided.

Technicaldesign

Learningmaterialsarevisuallyattractive.

Illustrations/visualsareeffective/appropriate.

Charactersize/typefaceisappropriate.



Studentscaneasilyemploytheresource.

Thelearningmaterialsarewellexplainedandinaclearlanguage.

Learningmaterialsarepresentedinadifferentformats(i.e.videos,documents,webpages,software,openeducationalresources,ICTtools)

Inclusion/socialconsiderations

Thelearningmaterialstakeintoaccountdifferentwaysoflearningbythestudents.

Learningmaterialstakeintoaccountstudentspreviousknowledgeandlearningoutcomes.

Learningmaterialsarewellsuitedtothestudents’diversity.

Learningmaterialsestablishdifferentdifficultygradesaccordingwiththediversityofstudents.

Workoriented

Learningmaterialsopportunitiestodevelopentrepreneurialmindsets.

Learningmaterialsintegrateopportunitiesofknowinglabourmarketneedsandskills.



Learningmaterialsarerelevantforthestudent’sfuturejobsopportunities.

Educational/multimediaandmanagementaspectsrelatedtoe-learningandVCPmaterials

Thedesignisuser-friendly.

Thetextisclearandeasytoread.

Thesoftware/webpages/videosarelogicallyorganized.

Navigationisclearandconsistent.

Thescreenisuncluttered.

Navigationisuser-controlled.

Littlehelpisrequiredtoworkindependently.

Thegraphicsandsoundarewellused.

Thegraphicsarenotoutdated.

Graphicsandaudioarenotdistractingelements.




11.6 Annex6:Students’initialquestionnaire(Draft)InformantAge:_______________________________________________________Previousleveloflearning:________________________________________________Instructions:Please complete thisquestionnaireby responding to the topicsbelow. Your feedbackwillassistus inevaluating thequalityof theCARESSpilot course.This informationwillbe kept confidentialandusedonlyforthepurposeofevaluatingtheCARESSpilotcourse.Note:Beforefillingthisquestionnaire,youhavetoreadthelearningprogramorbefamiliarwithit.1.WhyhaveyoudecidedtoattendthiscoursePilot?

�Iwascurious.�Itwasmandatory. �Itcouldbeusefulformycareer. �Other(pleasespecify):________________________________________ 2.Aretheobjectivesofthetrainingactivitiescleartoyou? �Yes �No �Idon’tknow Q Towhatextentdoyouagreewiththefollowingstatements?

Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neither agree nor disagree 4= Partially agree 5=Completelyagree

3. Thecoursegoalsareclear 12345

4. Thereareknowledge,skillsandcompetencesinhomecarethatcanbeusefultoimprovemyfuturejoboptions

12345

5. I find the courseprogram interesting to improvemyactual knowledge, skillsandcompetences.

12345

6. I am able to express doubts and concerns about the skills needed in homecare.

12345



7. Iamabletoworkinteamwithotherstudenttoreachthelearningobjectivesthroughtheproposalactivities

12345

8. Ifindthelearningstrategiesmotivatingandattractive 12345

Pleasewriteinyourpersonalresponses 9.Doyouhaveapreviouslearningexperienceinhealthcare?Incaseyouansweredaffirmatively,what

kindoflearningwas?(presential,on-line,blended…)

10.Whatdoyouexpecttolearnfromthispilotcourse?


11.7 Annex7:Students’finalquestionnaire(Draft) Instructions:Pleasecompletethisevaluationbyrespondingtothetopicsbelow.YourfeedbackwillassistusinevaluatingthequalityoftheteacherPilotCourse.ThisinformationwillbekeptconfidentialandusedonlyforthepurposeofevaluatingtheCARESSpilotcourse. a)ContentandstructureofthePilot

Q Towhatextentdoyouagreewiththefollowingstatements? Pleaseusethefollowingscale: 1=Completelydisagree 3=Neitheragreenordisagree 2=Partiallydisagree4=Partiallyagree5=Completelyagree

1. Thegoalsofthecoursewerecleartoyou. 12345



2. Courseactivitieswerewellorganized. 12345

3. Courseactivitiesallowedinteractiveparticipation. 12345

4. Courseactivitiesenhancedyourpreviousknowledgeregardinghomecare.

12345

5. Courseactivitiesenhancedyourknowledgeregardingjobopportunitiesinhomecare.

12345

6. Themethodologychosentakesinaccountthestudentsdiversityandpossibledifficultiestobesolved.

12345

7. Therewasenoughtimetocompletethecourseactivities. 12345

8. Courseactivitiesweresuitableforthegoalsofthetraining. 12345

9. Coursecontentwasrelevanttoyourfuturepractice. 12345

10. Thecoursematchedyourexpectations. 12345

b)Teachers-Tutors Q Towhatextentdoyouagreewiththefollowingstatementsregardingyourtutors?

Pleaseusethefollowingscale: 1=Completelydisagree 3=Neitheragreenordisagree 2=Partiallydisagree 4=Partiallyagree 5=Completelyagree

11. Thedescriptionofthecourse’sactivitieswasfullyunderstandable

12345

12. Teacherwasprovidinguswithguidancethroughouttheprogram.

12345

13. Teacherwasprovidinguswithfeedbackthroughouttheprogram.

12345

14. Teacherhadadequateknowledgeonthesubject 12345

15. Teachercommunicatedwellwithyou 12345

16. Teacherwaswellprepared 12345

17. Teacherdevotedenoughtimetoanswertoyourquestionsandissues

12345

18. Teacherguidedyouwell,throughoutyourlearningexperience 12345

c)Students’experience



Q Towhatextentdoyouagreewiththefollowingstatements? Pleaseusethefollowingscale: 1=Completelydisagree 3=Neitheragreenordisagree 2=Partiallydisagree4=Partiallyagree5=Completelyagree

19. IfeeltheCARESSprojectgoalsareinterestingandIconsidermyselfmotivatedtoknowmoreaboutit.

12345

20. Iwouldliketoincorporatethenewknowledge,skillsandcompetenceslearnedtomypractice.

12345

21. Thelearningdesigntakesinaccountmypreviousexperiences. 12345

22. Ifeltlessonsareeasytoputinpracticeinafuturejobsituation. 12345

23. Iwasabletocollaboratewithmyclassmatesduringthetraining.

12345

24. Theknowledge,skillsandcompetencesshowedduringthetrainingwereputincontext.

12345

25. Ihadtheopportunitytoaskquestionsandconcernsabouthomehealthcarenewknowledge,skillsandcompetences.

12345

26. Iwasabletodemonstratemyunderstandingbypreparingpresentations/writingreports,etc.

12345

27. Iwasabletoanalysed,criticized,comparedandcontrasted,throughthelearningactivities.

12345

28. Iwasabletoprovideconstructivefeedbacktomyteacherandclassmates.

12345

29. Iwasabletoreceiveconstructivefeedbackfrommyteacherandclassmates.

12345

30. Ihadtheopportunitytoreflectonmyownlearning. 12345

31. Iwasabletosharelearnedknowledge,ideasandlearningmaterialsthroughtheVCP

12345

32. IwasabletomakeonlinecontactwithotherstudentsandprofessionalsthroughtheVCP

12345

33. Iwasabletoshareideas,doubtsand/orexperienceswithfinalandendusers

12345

d)Overallsatisfaction



Q Towhatextentdoyouagreewiththefollowingstatementsregardingyoursatisfaction? Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neitheragreenordisagree4=Partiallyagree 5=Completelyagree

34. Theoverallorganizationoftheprogram 12345

35. Thecontentoftheprogram 12345

36. Thecourseactivities 12345

37. Thetrainingresourcesandmaterialsgivenbytheorganizers 12345

38. Thecommunicationwithinteacherandotherstudents 12345

39. Iwaspleasedattendingthiscourse 12345

40. Iwaspleasedwiththefacilitiesprovidedduringthecourse 12345

41. Iwaspleasedwiththematerialsprovidedduringthecourse 12345

42. Iwasprovidedwithsatisfactorysupportwhileacquiringthenewlearning.

12345

43. Iwaspleasedwiththelearningmethodologyusedduringthecourse

12345

44. Ifeelthemethodologyusedwasadaptedtomylearningspreferencesandstyleoflearning

12345

45. IwaspleasedwiththeuseandpossibilitiesoftheVCP 12345

e)Impact Q Towhatextentdoyouagreewiththefollowingstatements?

Pleaseusethefollowingscale: 1=Completelydisagree 3=Neitheragreenordisagree 2=Partiallydisagree 4=Partiallyagree5=Completelyagree

46. Iwillusethelearningacquiredinhomecareinmyfuturepractice.

12345

47. Ifeelthatthisnewlearningisusefulformyfuturejobpractice 12345

48. Thisnewlearningiseasytoapplyinmyfuturejobpractice 12345

49. Theapplicationofthislearningwillimprovemyjobopportunities

12345

50. Itwouldbegoodtoextendtoothercolleaguesthelearnings 12345



acquiredinthisPilotcourse.

f)Pleasewriteinyourpersonalresponses 51.Haveyourobjectivesforthecoursebeenmet?Whatwerethese?

52.Whatdidyoulikeabouttheformatofthecourse?Nameatleastonething.

53.Whatdidn'tyoulikeorcouldbeimproved?Nameatleastonething.

54.Othercomments


11.8 Annex8:Teacherfinalquestionnaire(Draft) Instructions:Pleasecompletethisevaluationbyrespondingtothetopicsbelow.YourfeedbackwillassistusinevaluatingthequalityoftheteacherPilotCourse.ThisinformationwillbekeptconfidentialandusedonlyforthepurposeofevaluatingtheCARESSpilotcourse.

a)Satisfactionandgoalachievement

Q Towhatextentdoyouagreewiththefollowingstatements? Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neitheragreenordisagree4=Partiallyagree5=Completelyagree



1. Thestructureofthelearningactivityhavebeensatisfactory a.12345

2. Thelearningdesignmatchedthegoalsforthepilot 12345

3. Courseactivitiesallowedinteractiveparticipation 12345

4. ThelearningactivitiesandthelessonsmetthePilot’sobjectives 12345

5. Thelearningactivitiesandthelessonsfocusedonthedevelopmentandassessmentoftheskillgapsinhomecare

12345

6. Courseactivitieswereeasytofollow 12345

7. Therewasenoughtimetocompletethecourseactivities 12345

8. Courseactivitiesweresuitableforthegoalsofthetraining. 12345

9. Coursecontentwasrelevanttoyourstudents’futurepractice 12345

10. Thecoursematchedyourexpectations 12345

b)Teachingperformance Q Towhatextentdoyouagreewiththefollowingstatementsregardingyourteacher

performance? Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neitheragreenordisagree4=Partiallyagree5=Completelyagree



11. Ihadanadequateknowledgeontheskillgaps 12345

12. Thestudentscommunicatedwellwithme 12345

13. Thelessonsandactivitiesplannedwerewellprepared 12345

14. Ihaveenoughtimeduringthelessonstoanswerthestudentsquestionsandissues

12345

15. Thetimeforthelearningobjectiveswasenough 12345

c)Overallsatisfaction Q Towhatextentdoyouagreewiththefollowingstatementsregardingyoursatisfaction?

Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neitheragreenordisagree4=Partiallyagree 5=Completelyagree

16. TheoverallorganizationoftheCARESSPilot 12345

17. Thecontentofthepilot 12345

18. Thecourse’activities 12345

19. Thetrainingresourcesandmaterialsused 12345

20. Thecommunicationbetweentheteacherandstudents 12345

21. IampleasedIteachthiscourse 12345



22. Iampleasedwiththefacilitiesprovidedduringthecourse 12345

23. Iampleasedwiththeresourcesprovidedbythelearningcentreduringthecourse

12345

24. IwasprovidedwithsatisfactorysupportwhileteachingthePilot 12345

d)Impact Q Towhatextentdoyouagreewiththefollowingstatements?

Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neitheragreenordisagree4=Partiallyagree5=Completelyagree

25. Iwillusethisteachingmaterialinthefuture 12345

26. Ifeelthatthislearningonskillgapsisbeenusefulformystudents 12345

27. Thisnewlearningiseasytoapplyinmyfuturejobpractice 12345

28. Theapplicationofthislearningwillimprovemystudentsjobopportunities

12345

29. Itwouldbegoodifothercolleaguescouldteachaboutthisknowledge,skillsandcompetences.

12345

e)Pleasewriteinyourpersonalresponses



30.Haveyourobjectivesforthecoursebeenmet?Whatwerethese?

31.Whatdidyoulikeabouttheformatofthecourse?Name3things.

32.Whatdidn'tyoulikeorcouldbeimproved?Name3things.

33.Othercomments




11.9 Annex9:Questionnaireevaluatingteachers’Pilotdesign(Draft)Teacher/Evaluator:__________________________ Date:______________________________________ Centre:____________________________________ Instructions:Pleasecompletethisevaluationbyrespondingtothetopicsbelow.YourfeedbackwillassistusinevaluatingthequalityoftheteacherPilotCourse.ThisinformationwillbekeptconfidentialandusedonlyforthepurposeofevaluatingtheCARESSpilotcourse.1.Pilot’spreparation

Q TowhatextentdoyouagreewiththefollowingstatementsregardingthePilot’sPreparation?


1. Thedocumentstodesignthepilotwereclearandenoughtohelpmedesignthepilot.

12345

2. TheCARESSframeworkhelpedmeinidentifytheknowledge,skillsandcompetencestobereach.

12345

3. Iwasabletoworkinteamwithotherteacherstoreachthelearningobjectivesthroughtheproposalactivities.

12345

2.Objectives

Q TowhatextentdoyouagreewiththefollowingstatementsregardingthePilot’sObjectives? Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neitheragreenordisagree4=Partiallyagree5=Completely

agree

4. Objectivesaremadeavailableinavarietyofareasinthecourse(withineachindividuallearningunitormodule)

12345

5. Objectivesareclearlywrittenattheappropriatelevelandreflectdesiredoutcomes.

12345



6. Objectivesarewritteninmeasurableoutcomes(studentsknowwhattheyareexpectedtobeabletodo)

12345

3.Methodology Q Towhatextentdoyouagreewiththefollowingstatementsregardingthemethodology?


7. Themethodologywaschosentomatchthegoalsaccordingwithmypreviousexperienceinteachinghealthcarers.

12345

8. Themethodologytakesintoaccountthestudentsdiversityandpossibledifficultiestobesolved.

12345

9. Thechosenmethodologyusesavarietyofappropriate mechanisms(contentmodules,linkstoexternalresources,and/ormultimedia,etc.)

12345

4.Contentpresentation Q Towhatextentdoyouagreewiththefollowingstatementsaccordingtothecontent

presentation? Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neitheragreenordisagree4=Partiallyagree5=Completelyagree

10. Contentispresentedindistinctlearningunitsormodules 12345

11. Contentflowsinalogicalprogression 12345

12. Contentispresentedusingavarietyofappropriate mechanisms(contentmodules,linkstoexternalresources,and/ormultimedia,etc.)

12345

13. Contentissequencedandstructuredinamannerwhichenableslearnerstoachievethestatedgoals

12345



14. Purposeoflearningactivitiesisclearlypresented. 12345

15. Theselectedtoolforeachactivityisappropriateforeffectivedeliveryofthecontent.

12345

16. The description of the units reflects the knowledge, skills andcompetencesassociated.

12345

5.Instructionalstrategies Q Towhatextentdoyouagreewiththefollowingstatementsregardingtheinstructional

strategies? Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree 3=Neitheragreenordisagree4=Partiallyagree5=Completelyagree

17. Itisclearhowtheinstructionalstrategieswillenablestudentstoreachcourseobjectives

12345

18. Coursedesignincludesguidanceforlearnerstoworkwithcontentinmeaningfulways

12345

19. Technologiesareusedcreativelyinwaysthattranscendtraditional,teachercentredinstruction

12345

20. Learnershavetheopportunitytogiveanonymousfeedbacktotheinstructorregardingcoursedesignandcoursecontentbothduringcoursedeliveryandaftercoursecompletion

12345

21. LearningactivitiesandotheropportunitiesaredevelopedtofosterStudent-Studentcommunicationand/orcollaboration.

12345

22. LearningactivitiesandotheropportunitiesaredevelopedtofosterStudent-Instructorcommunicationand/orcollaboration.

12345



23. LearningactivitiesandotheropportunitiesaredevelopedtofosterStudent-Contentinteraction.

12345

24. ThelearningactivitiesincludedonmyteachingdesignareeasilyusedintheVCPenvironment(provisional)

12345

25. TheIDtemplateshavebeenusefulregardingthedesignofthelessonplan(provisional)

12345

26. TheIDtemplateallowedmetohaveanoverallideaofthePilotcontentandhelpmetoidentifyrelevantinformationtopurposealessonplan.(provisional)

12345

27. TheILDEtemplateshavehelpedmetobetterdesigntheteachinglessonsaccordingtothelearningoutcomes,havinginmindthegoals,knowledge,aswellastheskillsandcompetencesthatshouldbeacquiredbythestudents.(provisional)

12345

28. TheILDEtemplatescreatedcouldbeusefulforfurtherVETcourses.(provisional)

12345

6.Assessment

Q Towhatextentdoyouagreewiththefollowingstatementsregardingtheassessment?


29. Assessmentactivitieshave“facevalidity”(i.e.,theyappeartomatchthecurriculumandareexplainedusingappropriatereadinglevelandvocabulary)

12345

30. Higherorderthinkingisrequired(e.g.,analysis,problem-solving,etc.) 12345



31. Assessmentsaredesignedtomimicauthenticenvironmentstofacilitatetransfer

12345

32. Assessmentactivitiesoccurfrequentlythroughoutthedurationofthecourse,andtheinstructorprovidesmeaningfulfeedbackinatimelymanner.

12345

33. Multipletypesofassessmentsareused(researchproject,objectivetest,discussions,etc.)

12345

34. Thereareopportunitiesforstudentself-assessment,providingfeedbackthatallowsstudentstoseekadditionalhelpwhennecessary

12345

9.CompliancewithEQVET

Q TowhatextentdoyouagreewiththefollowingstatementsregardingEQVET?

Pleaseusethefollowingscale:1=Completelydisagree 3=Neitheragreenordisagree 5=Completelyagree 2=Partiallydisagree 4=Partiallyagree

35. TheteacherhasbeentrainedinthepurposeandgoalsofthisPilot(regardingdesign,teachingmethods,evaluation,etc.)

12345

36. Theparticipantshavebeenchosenaccordingtothetypeofprogramme 12345

37. Stakeholdersfromtheareahavebeeninvolvedduringtheteachingprocess. 12345

38. Learningmaterialshavebeenadapted,consideringvulnerable/disabledstudentsinthegroupclass.

12345

39. Thestudents’participationrateincludesvulnerablegroups(earlyschoolleavers,highesteducationalachievement,migrants,personswithdisabilities,lengthofunemployment)

12345



10.QualitylevelofPilot’sIDdocuments Q Towhatextentdoyouagreewiththefollowingstatementsregardingthequalitylevelof

Pilot’sIDdocuments? Pleaseusethefollowingscale: 1=Completelydisagree 3=Neitheragreenordisagree 2=Partiallydisagree 4=Partiallyagree 5=Completelyagree

40. TheCaressIDTemplatehavebeenusefulregardingtheteachingperformance.CaressIDdocumentshavebeenusefulregardingtheteachingperformance.

12345

41. TheIDdocumentshelpmereflectingaboutdifferentteachingstylesandlearningstrategiesthatcanbeusedinmylessons

12345

42. TheIDdocumentsareflexibleenoughtobeadaptedthroughtheteachingperformance.

12345

43. TheIDdocumentshelpmeinreflectaboutmethodology,skillsandtransversalcompetences.

12345

44. TheIDdocumentsreflectthemacroandmicro-design. 12345

11.UseofCaressFrameworktodesignthePilot Q TowhatextentdoyouagreewiththefollowingstatementsregardingUseofCaress

FrameworktodesignthePilot? Pleaseusethefollowingscale: 1=Completelydisagree 3=Neitheragreenordisagree 5=Completelyagree 2=Partiallydisagree 4=Partiallyagree

45. Tobecompleted 12345







12.UseofWizardinhelpingtodesignthePilot Q TowhatextentdoyouagreewiththefollowingstatementsregardingtheuseofWizardin

helpingtodesignthePilot? Pleaseusethefollowingscale: 1=Completelydisagree 3=Neitheragreenordisagree 5=Completelyagree 2=Partiallydisagree 4=Partiallyagree






13.Pleasewriteinyourpersonalresponses



55.HaveyourobjectivesforthedesignofthePilotbeenmet?Whatwerethese?

56.Whatwillyouimprove?

57.Othercomments


11.10 Annex 10: Guidelines to complete the Student’s ReflectiveDiary(Draft)

School:

Student:

Course:

Starttime:

Endtime:

Date: Namethreethingsthatimpressedyouthemostaboutthee-learning/VCPsessions._______________________________________________________________________________________________________________________________________________________________________________________________________________ Namethreedifficultiesthatyoufacedduringtheactivitiesperformed:



_______________________________________________________________________________________________________________________________________________________________________________________________________________ Whathasworkedwellduringthesession? _______________________________________________________________________________________________________________________________________________________________________________________________________________ Namesomelessonlearntduringthee-learning/VCPsession. _______________________________________________________________________________________________________________________________________________________________________________________________________________ Wereyouabletodiscusstheemergedissueswithotherstudents/professionalprofilesduringthee-learning/VCPsession(understanding“session”asafullperiodofusebetweenloginandlogout,withthedevelopofoneormoreactivitieswithintheplatform)? _______________________________________________________________________________________________________________________________________________________________________________________________________________ Wereyouabletocreatecontactswithotherstudents/professionalprofilesduringthee-learning/VCPsession? _______________________________________________________________________________________________________________________________________________________________________________________________________________ Additionalcomments: _______________________________________________________________________________________________________________________________________________________________________________________________________________


11.11 Annex11:Student’posttrainingquestionnaire(Draft)Instructions: Please complete this evaluation questionnaire by responding to the topics below. YourfeedbackwillassistusinevaluatingthequalityoftheCARESSpilotcourse.ThisinformationwillbekeptconfidentialandusedonlyforthepurposeofevaluatingtheCARESSpilotcourse.



Q Towhatextentdoyouagreewiththefollowingstatements? Pleaseusethefollowingscale: 1=Completelydisagree2=Partiallydisagree3=Neitheragreenordisagree4=Partiallyagree5=Completelyagree

1. The learning achieved through the pilot helpedme to understandtheneedsofthejobmarket.

12345

2. Thelearningachievedthroughthepilothelpedmetobetteraddressmyeffortsinlookingforajob.

12345

3. I could apply to awide range of job offers thanks to the learningachievedthroughthepilot.

12345

4. Itwas able to show and talk about the learning achieved throughthepilotduringajobinterview.

12345

5. IbelievemyjobopportunitiesarebetterafterthePilot. 12345

6. I was able to prove my new learning knowledge, skills andcompetencesaftercompletepilotlearning.

12345

7. Iwouldrecommendthiscoursetootherprofessionals. 12345

Pleasewriteinyourpersonalresponses

8.Howwasyourjob/jobsearchingexperienceafterthePilot?


11.12 Annex12:Student’posttraininginterview(Draft)



1.HowwasyourexperienceofjobsearchingafterthePilot? 2.Wereyouabletogointoanyspecificjobselectionthankstothelearningachieved? 3.Doyoubelieveyourpossibilitiesoffindingajobhasbeenincreasedthankstothisexperience? 4.DoyouthinktheVirtualCommunityofPracticehelpyoubygivingyouadviceoraddressingyour jobsearch? 5.DoyoufindtheVirtualCommunityofPracticeusefultomeetotherprofessionals,keeponprofessionalupdateorfindsupportandhelpinyourprofessionalcareer? 6.Didyourecommendthiscoursetootherfellows?Why? 7.Inyouropinion,areyoubetterpreparedfromthejobmarketsinceyoucompletedthecourse? 8.Doesthejobmarketopportunitiesreflecttheneedofthenewlearningacquired?


11.13 Annex13:TrackingreportingCARESSe-learningplatform

TRACKINGPROGRESS

The tutor can check all the work the users have been doing through several tracking methods andthereforeprovidingthemwithafeedbackonit.

HereyoucanfindsomeofthesolutionsthatMoodleofferstocheckthejobcarriedoutbyeachuser.

• LOGS

LogsinMoodleareactivityreportsandtheyareavailableatsiteandcourselevel.

Youcanseewhatpagesthestudentsaccessed,thetimeanddatetheyaccessedthem,theIPaddresstheycamefromandtheiractions(view,add,update,delete).

Actually,thelogreportpagecontainsactivelinkswhichenableyoutoaccessastudent'sprofilepageoraspecificpagethatthestudentwascheckingatacertaintime.TheIPaddresslinkprovidesanestimateofthestudent'slocation

• ACTIVITYREPORTS

Acourseactivityreportshowsthenumberofviewsforeachactivityandresourceandanyrelatedblogentries.Itcanbeviewedbymanagers,teachersandnon-editingteachers.Ifactivityreportsareenabledforacourseinthecoursesettings,eachcourseparticipantcanaccessthereportsfortheircontributions,suchasforumpostsorassignmentsubmissions,logsandstatisticsreports.

• PARTICIPATIONREPORT



Some examples of participation reports are Forum View: (View Discussion, Search, Forum, Forums,Subscribers)andForumPost:(AddDiscussion,AddPost,DeleteDiscussion,DeletePost,MoveDiscussion,PrunePost,UpdatePost).

Anextremelyuseful featureof theparticipation report is theoption to sendamessage toall studentswho have not completed a certain action. Other options with selected users are to add a note or toextendenrollment.

• STATISTICS

If statistics are enabled for the site, graphs and tables for the user activity may be generated. Thestatisticsgraphsandtablesshowhowmanyhitstherehavebeenonvariouspartsofthesiteatadifferenttime.Theydonot showthenumberofusers though.Theyareprocesseddailyat thetimeyouspecify.Youmustenablestatisticsinordertoseeall.

Furthermore,thetutorhasvariousmethodstocheckthejobcarriedoutbytheuser.

ThegraderreportcollectsitemsthathavebeengradedandcomefromthevariouspartsofMoodlethathavebeenassessedandallowsyoutoviewandchangethemaswellassortthemoutintocategoriesandcalculate totals in variousways.When you add an assessed item in aMoodle course, the grade bookautomatically creates space for the grades itwill produce and also adds the grades themselves by thetimetheyaregenerated,eitherbythesystemorbyyou.

Thegradesareinitiallydisplayedasrawmarksfromtheassessments.Therefore,theywilldependonhowtheyhadsetthoseupe.g.anessayoutof36willappearashowevermanyrawmarksthatstudentgot,notapercentage(althoughthiscanbechangedlater,seebelow).

Noticethatvariousdefaultoptionsforthegradebookaresetatasystemlevelbytheadministratorandcanbemarkedasbeingoverreadablebyyou,orfixed.Thismeansthattheoptionswillnotalwaysbesetupinthesamemannerforeveryuserwhentheyseethegradereportforthefirsttime.

• COURSEGRADESETTINGS

Coursegradesettingsdeterminehowthegradebookappearstoallparticipantsinthecourse.

Course grade settings are found in Settings > Grade Administration > Course grade settings or via thegradebookSettingstab.

ThedefaultcoursegradesettingsaresetbyanadministratorinSettings>Siteadministration>Grades.

• Graderreportpreferences

Teacherscansettheirpreferencesforthegradereportviathegradebookcalled:'Mypreferences'.Thesesettingswillapplytoallcoursesforthatteacher.Theteachermaychangethematanytime.



Thedefaultgradereportpreferencesaresetbyanadministrator inSettings>Siteadministration>Grades>Reportsettings>Gradereport.

FromMoodle 2.4.2 onwards, the default number of students per page (grade report students perpage)cansafelybeincreasedwithoutanydatalosswhengradinglargenumbersofstudents.

• Siteadministrationsettings

Defaultvaluesforallgradesettingscanbesetbyanadministrator.Theeasiestwaytodothisistologinasadminandthenbrowseacoursegradebookandfollowthe'Changedefaults'links.

The following additional grade settings can be found in Settings > Site administration > Grades >Generalsettings.Thesettingswillaffectallgradebooksusedbyallteachers.

• GradedRoles

Gradedrolesarethetypeofuserthatwillappearinthegradebook.Mostofthetime"student"istheonlyoneneeded.

• USERPROFILEREPORT

Thedefaultuserprofilereportsettinghasjustoneoption,'UserReport',unlessacustomuserreporthasbeenaddedtothesite.

Teacherscanviewtheuser reports forallorsomeselectedstudents in thecourse inSettings>Gradesadministration>Userreport.

• INCLUDEDSCALESINAGGREGATION

Scalescanbeincludedasnumbersinallaggregatedgradesacrossallgradebooksinallcourses.Changingthissettingwillforceallaggregatedgradestoberecalculated.

If this is checked by default, then all custom scaleswill, therefore, have a corresponding value that isassignedbasedonthenumberofoptionsinthescale(i.e.threevalueswillhavevaluesof1,2,3)andthiswillbeincludedinthegradeaggregation.

• Enablepublishing

Gradepublishingsettings:

GradepublishingisawayofimportingandexportinggradesviaaURLwithoutbeingloggedintoMoodle.Gradepublishing isforadministratorsonly.Security implicationsshouldbethoroughlyexaminedbeforenon-adminusersareallowedtoreleasegrades.



Grade publishingmay be enabled by checking the grade publishing box. If enabled, administrators areprovidedwithgradeexportpublishingsettingsineachcoursegradebook.

• Recovergradesdefault

Whenmanually enrolling a student in a course, there is an option called 'Recover user's old grades ifpossible',butthischeckboxiseasytomiss.Enabling'Recovergradesdefault'resultsin'Recoveruser'soldgradesifpossible'beingtickedforeverycourse.

• Unlimitedgrades

Teachers can enter grades over 100% directly in the grade book if the unlimited grades setting areenabled.

Gradescanbedistributedintogradecategories.Agradecategoryhasitsownaggregatedgrade,whichiscalculatedfromitsgradeitems.Thereisnolimittothelevelofnestingofcategories.Thatbeingsaid,eachgradeitemmaybelongtoonlyonecategory.Also,allgradeitemsandcategoriesbelongtoatleastone,permanent category: the course category. A grade calculation is a formula used to determine grades,based)onothergradeitems.NotethatthisisnotthesameasCalculatedquestiontypes.

Calculations for the grade book follow the pattern of formulas/functions in popular spreadsheetprograms.Theystartwithanequal (=)signandusecommonmathematicaloperatorsand functions. toproduceasinglenumericaloutput.Thisoutputisthenusedasthecomputedvalueforthegradeitemyouareediting.

• Scales

Scalesareawayofevaluatingorratingastudents'performance.Moodleoffersastandardsetofnumericscales.Itisalsopossibletocreatecustomscales,whichcanbeavailableonthesite.Forinstance,youcangivethestudentawordorsmallphraseasawayofratingorgiveastudentfeedback.

ScalesmaybeusedinForums,GlossariesandAssignmentsforratingand/orgradingastudent'sactivity.

• Outcomes

Outcomes are specific descriptions of what a student has demonstrated and understood at thecompletionofanactivityorcourse.Eachoutcomeisratedbysomesortofscale.

Basically, outcomes are similar to sub-components of a grade. A grade is an assessment of the overallperformanceandittmayincludetests,participation,attendance,andprojects.Outcomesassessspecificknowledgelevelsthroughaseriesofstatements,whichmaybecodedwithnumbersorletters.Thus,anoverallgradecanbegivenforacoursealongwithstatementsaboutspecificcompetenciesintheformofoutcomes.



• Gradeletters

Gradeletters,alsocalledlettergrades,aresymbolsusedtorepresentarangeofgrades.Forexample,"A"couldbeusedtorepresentgradesof80%andabove,"B"torepresentgradesbetween70and80%,"C"torepresentgradesbetween50and70%,andsoon.Alternatively,youcouldhave"Pass"forgradesabove50%and"Fail"forgradesbelow50%.

Whenatutorrealizesthatauseriseithernotveryactiveorisonthevergeofabandoningthecoursehecan take action by stimulating the user through a series of utilities on the platform itself. The Surveymodule is a course activity that provides a number of verified survey instruments, including COLLES(Constructivist On-Line Learning Environment Survey) and ATTLS (Attitudes to Thinking and LearningSurvey),whichhavebeenfoundusefulinassessingandstimulatingtheonlinelearningprocess.Teacherscanusethesetogatherdatafromtheirstudentsthatwillhelpthemlearnabouttheirclassandreflectontheirownteaching.Currently,Moodleonlyoffersspecifictypesofsurveys(futureversionswillenableyoutocreateyourown).

The available surveys have been chosen as being particularly useful to evaluate online learningenvironmentsthatuseaconstructivistpedagogy.Theyareusefulto identifycertaintrendsthatmaybespreadingamongparticipants.

• COLLES-ConstructivistOn-LineLearningEnvironmentSurvey

TheCOLLEScomprises24statementsdividedintosixscales,eachishelpfultoaddressakeyquestiononthequalityoftheon-linelearningenvironment:

• Relevance:Howrelevantison-linelearningtostudents'professionalpractices?• Reflection:Doeson-linelearningstimulatestudents'criticalreflectivethinking?• Interactivity:Towhatextentdostudentsengageon-lineinrelevanteducativedialogue?• TutorSupport:Howwelldotutorsenablestudentstoparticipateinon-linelearning?• PeerSupport:Isthesupportprovidedon-linebyfellowstudentssensitiveandencouraging?• Interpretation:Dostudentsandtutorsmakegoodsenseofeachother'son-linecommunications?

AnewknowledgeisasocialConstructivism,whichportraysthelearnerasanactiveconceptualizedwithinasocially interactivelearningenvironmentandit isactuallybasedonunderpinningthedynamicviewoflearning. Social Constructivism is an epistemology or a way of knowing where learners work togetherreflectivelytoco-buildnewunderstandings,especiallywhenitcomestothemutualinquirygroundedintheirpersonalexperience.

Essentialtothiscollaborationisthedevelopmentofthestudents'communicativecompetence,thatistheability to engage in an open and critical discourse with both teacher and peers. This discourse ischaracterized by an empathic orientation to build a reciprocal understanding and a critical attitudetowardsexaminingunderlyingassumptions.



COLLES has been designed to enable you tomonitor the extent to which you are able to exploit theinteractivecapacityoftheWorldWideWebtoengagethestudentsindynamiclearningpractices.

• ATTLS-AttitudestoThinkingandLearningSurvey

The theory of 'ways of knowing', originally from the field of gender research (Belenky et al., 1986)providesuswithasurveytooltoexaminethequalityofdiscoursewithinacollaborativeenvironment.

TheAttitudesTowardsThinkingandLearningSurvey(ATTLS)isatooldevelopedbyGalottietal.(1999)tomeasuretheextenttowhichapersonisa'connectedknower'(CK)ora'separateknower'(SK).

People with higher CK scores tend to find learning more enjoyable and are often more cooperative,congenialandmorewillingtobuildontheideasofothers,whilethosewithhigherSKscorestendtotakeamorecriticalandargumentativestancetolearning.

Studies have shown that these two learning styles are independentof eachother (Galotti et al., 1999;Galottietal.,2001).Additionally,theyareonlyareflectionoflearningattitudes,notlearningcapacitiesorintellectualpower.

Documents

ERASMUS PLUS 2015 SECTOR SKILLS ALLIANCES · Deliverable 6.2 – Evaluation and Monitoring Plan 562634-EPP-1-2015-IT-EPPKA2-SSA CARESS Project 3 of 73 4 I.T.C. “Vittorio Emanuele