External Evaluation Report - TEMPUS ECESMecesm.net/sites/default/files/FINAL - External... · European Commission Tempus Project: 544088-TEMPUS-1-2013-1-SI-TEMPUS-JPHES This project

EuropeanCommissionTempusProject:544088-TEMPUS-1-2013-1-SI-TEMPUS-JPHES

ThisprojecthasbeenfundedwithsupportfromtheEuropeanCommission.Thispublication[communication]reflectstheviewsonlyoftheauthor,andtheCommissioncannotbeheldresponsibleforanyusewhichmaybemadeoftheinformationcontainedtherein.

|PROJECTCOORDINATOR:UniversityofMaribor||Slomškovtrg15,SI-2000Maribor | ecesm.net|

|Phone:+38622207105|Fax:+38622207272|

–

ExternalEvaluationReport

ProjectNo.544088-TEMPUS-1-2013-1-SI-TEMPUS-JPHES

FundedbyTempusIVProgrammeoftheEuropeanCommission

Preparedby:

ExternalQCMexpertDanijelaMiloševićFacultyofTechnicalSciencesČačak,UniversityofKragujevac,SerbiaČačak,Serbia•15thofSeptember2016




|Phone:+38622207105|Fax:+38622207272|

–

This project has been funded with support from the European Commission. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.




|Phone:+38622207105|Fax:+38622207272|

–

TableofContentsExecutiveSummary.................................................................................................................................4

ExternalEvaluationMethodology...........................................................................................................5

ProjectDescription..................................................................................................................................6

Projectboard.........................................................................................................................................10

ExternalEvaluationFindingsforTempusECESM..................................................................................11

Generalfindings................................................................................................................................11

DeliverableWP1:ReviewandanalyzeexistingEuropeanpracticeforCybersecurityeducation.....12

DeliverableWP2:Raiseawarenessabouttheriskofonlineactivities..............................................13

Deliverable WP3: Develop and maintain an unrivalled, globally competitive cyber securityworkforce..........................................................................................................................................14

WP4Broadenthepoolofskilledworkerscapableofsupportingacyber-securenation..................15

DeliverableWP5:QualityControlandMonitoring............................................................................17

InternalQualityControlManagement..........................................................................................18

Inter-Tempuscoaching..................................................................................................................18

ExternalQualityControlManagement..........................................................................................18

DeliverableWP6:Dissemination.......................................................................................................19

DeliverableWP7:Sustainability.........................................................................................................22

IndicatorsofProgress............................................................................................................................23

Overallfindings......................................................................................................................................26




|Phone:+38622207105|Fax:+38622207272|

–ExecutiveSummary

ThisdocumentcontainscompactExternalEvaluationreportaimingtoassesswhethertheobjectivesof ECESM TEMPUS project “Enhancement of cyber educational system of Montenegro” 544088-TEMPUS-1-2013-1-SI-TEMPUS-JPHES are achieved in accordance to the project proposal. The timecoveredisthree-yearperiod,projectlifetime(M1-M36).

The quality control and monitoring is performed based upon all available documentation (projectproposal, LFM, reports & deliverables, web site information, intermediate report, and feedbackmonitoring visit reports by EACEA). Also, the strong pointwas given to conducted interviewswithcontactpersonsfrompartnerinstitutionsthatwereinchargeforspecifictasks.

Theindependentexternalevaluationfindingsarepresentedtogetherwiththedeliverablesachieved.Apartfromthat,thedetectedchangesanditsimpactsaregivenalongwiththeindicatorsofprogressdefinedintheLogicalFrameworkMatrixoftheproject.ThemajorityoftheactivitiesoftheTEMPUSECESM project have beenmet according to project proposal, and also themain project goals aresuccessfully delivered towards the project end. The professional and experienced projectmanagementwasoneofthemainstaysforsuccessfulprojectimplementation.




|Phone:+38622207105|Fax:+38622207272|

–

ExternalEvaluationMethodology

InordertocarryouttheExternalEvaluation,theauthorwasgivenfullaccesstoallrelevantexternaland internal documents and on the contents of the project’s website http://www.ecesm.net/.Startingfromtheoriginaldocumentationeg.projectproposal,theLogicalFrameworkMatrix-LFM,available documentation included reports & deliverables, internal quality control and monitoringreportsforworkpackages,intermediatereport,andfeedbackmonitoringvisitreportsbyEACEA,etc.

Direct communicationwith project coordinator prof.Matjaž Debevcwas obtained almost on dailybases, aswell as Skypeandemail communicationwith contactpersons fromotherprojectpartnerinstitutionsinordertocollectandsummarizetheimportantinformationforreportingandevaluationfindings. The methodology starts with project overview and analyse of planed and achieveddeliverables,andcross-matchingwithlogicalframeworkmatrixinordertodefinetowhichextentthequalityofresultsmeettheplanaccordingtotheavailableindicators. Finally,thesummaryexternalevaluatorfindingisgiven.




|Phone:+38622207105|Fax:+38622207272|

–ProjectDescription

The ECESM project has been designed to enhance the overall cyber security posture of theMontenegro by accelerating the availability of educational and training resources designed toimprove the cyber behavior, skills, and knowledge of every segment of the population. The mainproject objective is to improve, develop and implement standards, guidelines and procedures atnational system levels in Montenegro in order to enable creation of agile and highly skilledMontenegrin forces capable of responding to a dynamic and rapidly developing array of e-threats.Developed standards will be adopted and assimilated to EU standards having in mind specificcharacteristicsofMontenegrinsocietybyexpandingexistinglegal,sublegal,technicalandeconomicalcapacitiesatthenationallevel.In order to provide solid base for the implementation of newly developed cyberspace, the ECESMprojectwillbuild thehighlyawareMontenegrinsociety throughestablishinga framework forcybereducation, which will encourage competence across the nation. Furthermore, the ECESM projectprovide:

• Extensiveworkshops,presentationsandpromotionsforcitizensandalllevelsofstaffinvolvedinbroadswathsoftheeconomyandindividualactors,representativesoforganizations,etc.inordertoincreasepublicawarenessofcybersecurityrisksandunderstandspecialtyareasofcybersecurityneeds;

• Specialized training for specific groups ofworkers, related to their range of responsibilitiesand jurisdictions, ranging from state and local governments, private sector, criticalinfrastructure owners/operators, large companies, small businesses, academic institutions,andotherinterestedparties;

• Accredited master program recognized and supported by relevant international academicsociety, for continuous future education of highly skilled professionals in specific areas ofcyber security, and (iv) encouraged environment for further research activities in cybersecurityarea.

ProjectgoalsandoutcomeswillenablefulfillmentofissuesdefinedthroughtheMontenegrinprocessofexaminationswiththeEU(e.g.screeningchapter10-Informationsocietyandmedia,Dec,2012).




|Phone:+38622207105|Fax:+38622207272|

–

QualityControlandMonitoringofprojectactivitiesandresultsbyinternalandexternalmechanismswillberealizedcontinuouslythroughoutthewholedurationoftheprojectandevaluatedatseverallevels.TheprojectisaimedtomakeawideimpactonthenationalsysteminMontenegro,andthusthe synergy of national Ministries of Defense, Interior, Justice, Finance, Education and Information

Society and Telecommunications,withrelevant institutionsinEUhighlyexperiencedincybereducationand threats protection, is the best guarantee of the sustainability of project results beyond its lifetime.

Projectwiderobjective

- BreakingthelacksofeducationalsystematnationallevelinMontenegroregardingcybersecurityfield,increasingpublicawarenessaboutcyberthreatsandpotentialsforresponding,creatingnewcyberspaceinMontenegrowithhighlyspecialized,trainedandeducatedworkers,enhancededucationandresearchbyestablishingmultidisciplinaryMasterprogram

Projectspecificobjectives

- AdjustingeducationalpracticeincybersecurityatnationallevelinaccordancewithEUpracticeandstandards

- Increasingpublicawarenessofcybersecurityrisksandunderstandingofspecialtyareasofcybersecurityneeds

- Improvingknowledgeofcybersecuritywithinorganizationssothatresourcesarewellappliedtomeetthemostobviousandseriousthreats

- Accreditationofmultidisciplinarymasterprogramrecognizedandsupportedbyrelevantinternationalacademicsociety,forcontinuousfutureeducationofhighlyskilledprofessionalsinspecificareasofcybersecurity

- EncouragementofMontenegrinprofessionalsandHEandresearchenvironment,forfurtherresearchactivitiesincybersecurityarea




|Phone:+38622207105|Fax:+38622207272|

–

WorkPackages

Deliverable/ ActivityRef.N°

Activities(asindicatedintheLFM)

1 ReviewandanalyzeexistingEuropeanpracticeforCybersecurityeducation

1.1 AnalysisofexistingEuropeanpracticeandprinciplesforCybersecurityEducation

1.2 AnalysisofthepolicyleveldocumentsaddressingCybersecurity

1.3Analysisofexistingpractice inMontenegroandcross-matchingwithEuropeanpracticesandstandards

1.4DeveloppolicyandplanroadmapforimplementingandmanagingCybersecurityEducationinMontenegro

2 Raiseawerenessabouttheriskofonlineactivities

2.1 Developapubliccybersecurityawarenessframework

2.2 Improvecitizens’knowledgetoallowthemtomakesmartchoicesastheymanageonlinerisk.

2.4 Enableaccesstocybersecurityresourcesforcitizens

3 Developandmaintainanunrivaled,globallycompetitivecybersecurityworkforce.

3.1Develop a usable cybersecurity competency framework (Human Resources & Curriculumfocus).

3.2 Provideaframeworkforfocusingcybersecuritytrainingtomeetevolvingneeds.

3.3 Improveknowledgeofcybersecuritywithinorganizationsandgovernmentalbodies

3.4Study the application of professionalization, certification, and licensing standards oncybersecuritycareerfields.




|Phone:+38622207105|Fax:+38622207272|

–

4 Broadenthepoolofskilledworkerscapableofsupportingacyber-securenation.

4.1 Increasethequantityandqualityofcybersecuritycoursesinundergraduatestudyprograms.

4.2 Createmultidisciplinarycurriculumformasterstudyprogramforcybersecurity.

4.3 AcreditationmultidisciplinaryMastersudyprograminCybersecurity

4.1 Incentivize,support,andrecognizeexcellenceincybersecurityresearchanddevelopment.

1 QualityControlandMonitoring

1.1 Internalqualitymonitoring,assessmentandcontrol

1.2 Externalqualitycontrolandmonitoring

1.3 Inter-Tempuscoaching

1 Dissemination

1.1 Establishandmaintaininteractiveprojectwebsiteinenglishandlocallanguages

1.2Organizeworkshops,roundtablesandspecialsessions fordissem.of intermediateandfinalresults

1.3 Startupannualregionalconference"Cybersecurityandsociety"

1.4 StartuponlineNewsletter“Cybersecurityandsociety”

1 Sustainability

1.1 Developmentofsustainabilitystrategyofrelevantprojectresultsatthenationallevels

1.2 BuildcapacityforsecuringadequateresourcesfortheCybersecurityEducation

1.3 Theinitiativesforstart-upMonteneginCybersecurityEducationCentre–MCEC

1 Management

1.1 Overallprojectmanagementactivities

1.2 Projectcoordinationmeetings

1.3 Regionalcoordinationmeetings




|Phone:+38622207105|Fax:+38622207272|

–Projectboards

ProjectManagementBoard

1.UniversityofMaribor-MatjazDebevc

2.TallinUniversityofTechnology-MarkoKaaramees

3.GlobalCenterofCyberSecurity-MassimoCappelli

4.UniversityofRomaTre-StefanoGuarino

5.BuckinghamshireNewUniversity–FlorinIoras

6.UniversityDonjaGorica-RamoSendelj

7.UniversityMediterranean–AdisBalota

8.InstituteofModernTechnologyMontenegro–DraganDjuric

9.Ministryofinformationsocietyandtelecommunications–AnaRakocevic

10.Ministryofeducation–NadaKovac

11.ChamberofeconomyMontenegro–NadaRakocevic

QualityManagementBoard

1.UniversityofMaribor-MarkoHolbl

2.TallinnUniversityofTechnology-MarkoKaaramees

3.GlobalCenterofCyberSecurity-MassimoCappelli


5.UniversityMediterranean–AndjelaJaksicStojanovic

NationalManagementBoard(Montenegro)


2.UniversityMediterranean-AndjelaJaksicStojanovic

3.InstituteofModernTechnologyMontenegro-DraganDjuric

4.Ministryofinformationsocietyandtelecommunications-AnaRakocevic

5.Ministryofeducation-NadaKovac

6.ChamberofeconomyMontenegro-NadaRakocevic




|Phone:+38622207105|Fax:+38622207272|

–

ExternalEvaluationFindingsforTempusECESM

Generalfindings

Afterdetailed insight inoriginalprojectproposal it canbeconcluded that theprojectwascarefullyplaned and steps towards obtaining project objectives are duly defined. Also, after inspection ofLogicalFrameworkMatrix,itisobviousthatprojectteamiswellawareofpossibleobstaclesandrisks.

Theprojectthematicishighlyimportant,butcanbeconjectural,takingintoaccountthatcountryofimplementation is rathersmallwhichcanaffect thenumberofprospectivestudents,aswellas thenumber of quality experts in the field of cyber security. Nevertheless, the project team hasdemonstratedgoodawarenessontheseaspects,andtookthemintoaccountfromtheverybeginningof the project plan. The decision of project consortium to enhance cyber educational system ofMontenegroistobeappreciatedatalllevelsofeducationalsystemandstakeholders.

The three project boards (Project Management Board, Quality Management Board, NationalManagement Board) are well defined. It is shown that coordinating the project from outside ofpartner country was implemented on high level and obviously the coordinator and managementboardsweredevotedtotheprojectobjectives.

Developmentworkpackagesproducedresultsinvariousforms:reports,trainings,handbooks.Actualdelivery dates are not explicitly given in all documents, but according to the metadata, mostdeliverablesarealignedwiththeproposedtimeline.

In WP1 all deliverables are fully harmonized with the proposal. The meetings held in order ofcoordinatingofWP1activitiesshoweduptobefruitfulintermsofdispatchingtasks.

WP2wasmore practical oriented in terms of its deliverables. As result two versions of handbookwere produced and trainings were organized for wider audience primarily inME. Deliverables arealignedwithproposal.

PackageWP3isalignedwiththeproposedoutcomes.Thebasiccompetenceframeworkisdesigned.Trainings fororganizationsareplanned,preparedandconducted.Six trainingsessionswereheld inChamberofEconomy.Everytrainingisevaluatedandsurveysshownthattraineeswereverysatisfiedwithknowledgeandskillsacquired.




|Phone:+38622207105|Fax:+38622207272|

–

ThemainproposedoutputsexpectedoutoftheWP4areaccreditedstudyprogramsincybersecurity.The UDG had developed amaster study program in cyber security before the project started andadapted it afterwards, including the project results, while UNIM has adapted the already existingprogram "Master in IT" introducing cyber security elective subjects. UNIM is up to accredit a newprogramincybersecuritynextschoolyear.ThisoffsetfromoriginalproposalwasarrangedwiththeAgency

Thesufficientnumberofdisseminationactivities ishighlyevident.However, it isnotclearwhichofthese activities are targeted to the prospective students and if some of the activities involvedstudents aswell. Theonlinenewsletter “Cyber instructions for all”, andannual conference “CyberSecurityintheKnowledgeSociety”arestartedandimplementedaccordingtotheprojectplan.

It ispraiseworthythatvisibilityofTempusprogrammesupportfordevelopingcybersecuritymasterprogramishighlightedatbothuniversitywebsites,aswellasatthewebpagesforenrollmentofnewmasterstudents.

Theroleofnon-academicpartnersaswellasMinistrieshasbeenpromotedtilltheprojectend.Theirinvolvement is intensified indisseminationandsustainabilityactivities inorder tostrength thenewofferingof cyber security studyprograms. Their role is alsoevident inpromotingofCyber SecurityEducationCenteramongtheacademiccommunityandbusinessworld.

The two field monitoring visits took part in 2014 and 2015 with several recommendations andremarks. The consortium considered the remarks and took into account the suggestions to beimplemented till the project end. The feedbacks from the intermediate report were taken intoaccountinordertomaximisetheprojectbenefittoMontenegrinhighereducationsystem.

DeliverableWP1:ReviewandanalyzeexistingEuropeanpracticeforCybersecurityeducation

Thispackage isestablished inorder to setground for forthcomingactions.Allof realizedactivitiesfollowedtheproposedtimespan.Appropriatedeliverablesaredocumentedandreportsareavailableonline in form of four reports made by EU and ME partners. Deliverables are fully aligned withproposal.

ExistingEUpracticewasratherthoroughlypresented.Itoverpassedthesubjectofpracticeandwentthroughmuchwidercontext,evenencompassingcountriesoutofEU.Similarcaseiswithanalysisofeducationpractice.It isnotclearwhythisreportdidnotsticktoEU,butratherspreadovervariouscountriesonothercontinents.Itisnotalackperse,butratheraquestioniftherewasn'tenoughofsuchevidencefromEU.




|Phone:+38622207105|Fax:+38622207272|

–

There is no clear evidence if there is any law or regulation adapted so it implements the projectresults.

OnemeetingwasarrangedfororganizingthisWPinMaribor.Accompanieddocumentationisputinthe"Meetings"sectionoftheprojectwebsite.

Cross-matching of practice is presented with a comprehensive introduction. Intersection linesbetweenEUandMEpracticeisshownwithshowcasingofvariousformsofeducationprovidedinthefield of cyber security. Classification of formal/informal learning given as the summary might berevisedandtermofnon-formallearningcouldbeintroduced.

AsaPillaroftheEducationroadmapauthorschosethePDCAcycle.Wefindthisapproachashighlyreasonable, especially in area of cyber security. As the security is a process, not a product, thestandardproposedthecycleasa"roundroadmap"forinformationsecuritysystemestablishmentandthisapproachisclearlyapplicableoncybersecuritytoo.

Reports are produced in both languages (EN, SR). The local language version is officially registeredwiththeNationallibrary.

DeliverableWP2:RaiseawarenessabouttheriskofonlineactivitiesThis package is devoted to research of current state of public cyber security awareness and to itsimprovement.Alldeliverablesarecompletedintimelyfashionandarealignedwiththeproposal.Thefirst report on awareness in ME is rather comprehensive and gathers information from differentsources.However,inseveralplacesthediagramsareomitted(justplaceholders,novisibleimage)andthereportwouldbenefitinqualityifthesewereproperlyset.

The security framework was given by considering various levels of education and is suppose toprovide foundation for forthcoming design of appropriate curricula and handbook. Although it is adraft,reportcouldbemorecomplete,i.e.the"Summary"sectionisleftempty.

The trainingactivity isnext step. It is set in formofdirect andonline courseswithdifferent targetgroups (instructors, teachers, students). A group of 10 instructors went through Cisco course ininformationsecurity (Introduction tocybersecurity)andgotcertificates.We find that this step isagreat contribution in moving towards industry standards and building teachers capacity forforthcomingtrainings.




|Phone:+38622207105|Fax:+38622207272|

–Werecognizethehandbookasaveryimportantresult.Thecontentmettheproposal:thehandbookis made for average users. There are two version of handbook: one in English and the other inMontenegrinlanguage.Itisstatedthatthehandbookisfor"naiveusers".However,wefindthatthefirstversion isratheradvancedandmoreappropriateforadvancedusersandprofessionals.Ontheotherhand, the secondversion is composed inmanner that is understandablebywideauditoriumandisfullofpracticalexamplesofsecuritymechanismsavailabletoeverydayusers.

DeliverableWP3: Develop andmaintain an unrivalled, globally competitive cybersecurityworkforce

WP3 activities are pointed towards improving regulations and gaining impact in organizations andindustry.FirststeptakenwascrossmatchingEUpracticewithMEcurrentstate.Reportsaremostlyalignedwithproposal.SimilartoWP1,standardsandpracticesdidnotremain incontextofEU,butspreadtoUS.AlthoughwemayencourageusingbestpracticefromoutEU,wewouldrecommendtoclearly distinct the analysis focus (EUpractice) from someotherpossibleuseful example.Althoughthecybersecurityisaglobalissue,duetonatureandscopeofprojectitispreferabletoputemphasisontheEUspecificpractice.

The security questionnaire is made in order to get information from companies. It is in rathercomprehensive andwe find it as a great survey tool.On the other hand, it is not clear howmanyorganizations responded andwhat the result looks like.We find that this feedbackwould of beenbeneficialfornextdeliverable-thecompetencyframework.

The cyber security competency framework is build upon industry standards and is supposed to beimplemented in organizations in form of trainings. Trainings are set for different target groups,basically:forICTprofessionalsandforothers.

Six training sessionswere held on various cyber security topics in Chamber of Economy. Lecturerswere experts from EU. Trainees were public sector staff (from ministry, university) and industryrepresentatives. Every coursewas evaluatedusing a survey and theoverall feedbackaboutqualitywasverygood(above4,5outoffive).

Thetrendsareelaborated infieldofcybersecurity.Accordingtoseveralcrediblesourcesthere isawideopportunityforspecialistsinthisareainME.




|Phone:+38622207105|Fax:+38622207272|

–

WP4 Broaden the pool of skilled workers capable of supporting a cyber-securenationThispackageisdevotedtostudyprogramthatwouldgenerateskillfulindustryworkforce,whowillbefurther actively involved in improvement of general public cyber security awareness. According toproposal,bothMEpartnersweresupposedtodevelopandaccreditmasterstudyprogram incybersecurity.

There isacomprehensiveanalysisofcurrentstate incybersecurityeducationworldwide.Althoughno specific standard is established in this area, there are specialized study programs and wemayexpecttheprogressinstandardizationtoo.ThereisaanalysisofstateofeducationinthefieldattheMEpartneruniversitiesandattheselectedEUuniversities.

During the project the changes in the existing master program in cyber security at UDG wererecognizedandappropriateprogrammeupdatewassuggested.InitialideawastobringjointprogramwithUNIM,butsincethatisnot(yet)allowedbythelocallaw,atUNIMwasdecidedtofostercybersecurityintroducingnewsubjectsrelatedtothisarea.

University of Nova Gorica published a call for enrolment on master program in Cyber securityhttp://www.udg.edu.me/fist/osnovne/obavjestenja/serve/7312.Thecallexcerptisgivenbelow.

Theprogramisnotaccreditedduringtheproject,asitwassetinproposal.Itwasalreadyaccreditedwhentheprojectbegan.However,projectmanager(onbehalfofpartners)appliedformodificationofthemaster-program,byimplementingEUbestpractice.Thecurrentstudyprogrammeisgivenathttp://mastercybersecurity.me/nastavni-plan-i-program-20162017-god/




|Phone:+38622207105|Fax:+38622207272|

–

University Mediterranean offers master study program in information technology with severalsubjectsrelatedtoinformationsecurity.Sincetheaccreditationprocesstakesalongtimetocompleteand includes various procedures, it is decided that for first time cyber security subjects should beintroduced in already existing program. This programme-modification is accepted at the universitySenate.Studentsareinpositiontochooseallthreeelectivesubjectsinareaofcybersecurity.Everycyber-security subject is funded by the project and the scholarship for students that chose thesesubjectsisdecreasedaccordingly.Informationaboutthesesubjectsisuptobeingdocumentedinthediplomasupplementthatisassembledwiththediploma"masterininformationtechnology".

However,theprocessofaccreditationofthedistinctstudyprogramme"master incybersecurity" isongoingandisexpectedtostartnextschoolyear.

Thisyearcallforadaptedprogram"masterofinformationtechnology"isgivenbelow:




|Phone:+38622207105|Fax:+38622207272|

–

DeliverableWP5:QualityControlandMonitoringThe TEMPUS ECESM project Enhancement of cyber educational system of Montenegro hasestablishedtheQualityassuranceboardattheverybeginningoftheproject.TheQAboardconsistof5representativesasfollows:

• MarkoHölbl,UniversityofMaribor• MassimoCappelli,GlobalCyberSecurityCenter• RamoŠendelj,UniversityDonjaGorica,• AnđelaJakšićStojanović,UniversityMediterranean• MarkoKääramees,TallinnUniversityofTechnology

Eventhatvariousinstitutionsareinvolved,itshouldbenotedthatrepresentativesfromtheMinistryofinformationsocietyandtelecommunicationsandMinistryofeducationshouldbealsomembersoftheQAboard.




|Phone:+38622207105|Fax:+38622207272|

–InternalQualityControlManagement

Accordingtotheavailableevidencefromtheprojectwebsite,theQualityassuranceboardhasbeenworking seriously on quality assessment ofWP1 andWP2. Quality Assurance Board held separatemeetingsforeachworkpackageandproducedminutesofthemeetingwithcorrespondingreportsonWork Package 1 andWork Package 2. The reports performed in depth analyses of the produceddeliverables and provided extensive comments and recommendations. Furthermore, there wassimilarapproachforotherworkpackages.TheworkofQAboardisreallypraiseworthy.

Inter-TempuscoachingTheInter-Tempuscoachingeventswasorganizedon30thSeptember.ItwasheldduringIFOFEST2014inBudva,Montenegro.Themainaimoftheeventwastouseaccumulatedexpertiseofmembersofrunning and completed Tempus projects through sharing experiences between different projectconsortiums.TheInter-TempusCoachingwasattendedby14participantsfrom6institutions.

The Inter-Tempus coaching events was cross-matched with two Tempus projects: “BioengineeringTechnologyandMedicalInformaticsStudiesinSerbiaandMontenegro”(BioEMIS)and“SupportandInclusionofstudentswithdisabilitiesatHigherEducationInstitutionsinMontenegro”SINC@HE.

ExternalQualityControlManagement

MonitoringinMontenegro

Asapartofanon-goingmonitoringoftheTempusactivities inMontenegro,afieldmonitoringvisittookplaceonthe27thofJune2014attheUniversityDonjaGorica.MsVanjaDrljevićandMrRankoLazović from the National Erasmus + Office in Montenegro met and discussed the progress ofactivities with representatives of both Montenegrin Universities involved in the project and withrepresentativesof theMinistryof InformationSocietyandTelecommunicationsandthe InstituteofModernTechnology.ThevisitconfirmedthattheprojectisrelevantforthehighereducationcontextinMontenegro. Itwas highly appreciated that the activities are developing in accordancewith theplan, and that there are no significant delays as regards the implementations of activities. Theauditors recommended that when creating a master programme, the existing programme atUniversity of Donja Goricamust be taken into account in order to avoid overlapping. The idea ofdeveloping joint master programme is highly appreciated and suggested by auditors. It was alsosuggestedthatparticipationofallnon-academicpartnersintheprojectactivitiesneedtobestrongerinthefuture,especiallytheroleofInstituteofModernTechnology.

Aspartofouron-goingmonitoringofTempusactivitiesinMontenegro,afieldmonitoringvisittookplaceon the28thofSeptember2015 inBudva.MsVanjaDrljevićandMrRankoLazović from theNational Erasmus + Office (NEO) inMontenegromet and discussed the progress of activities withrepresentatives of the consortia. The auditors noted that the planned master programme to beintroduced has already been accredited before the official start of the project. Thus, the foreseen




|Phone:+38622207105|Fax:+38622207272|

–programmewillbeonlyreviewedwithallEUpartnersandassuchaccredited/reaccreditedduringthereaccreditationprocessin2016.Theadvisornotedthattuitionfeesmustnotbechargedduringtheprojectimplementationphase.ItisalsosuggestedthatCyberSecurityEducationCenterneedstobepromotedmoreamongtheacademiccommunityatthelevelofthewholesystem,aswellasatthelevelofbusinessworld.

Thefeedbackontheintermediateprojectreportwasreceivedon24thSeptember2015.Theauditorspointed out the positive fact that ten trainers have been selected from five partner universities inMontenegro,andthattwotrainingcourseshavebeenorganizedandCISCOtrainingcourseforCyberSecurityIntroductionModulewillbeusedtotraintheothertrainers.Theauditorscongratulatedtheconsortiumonwell-performed dissemination activities, and pointed out the need to providemoreinformation on howmaster program courses were developed. Apart from that, it is positive thatsustainability strategyhasbeenpublishedwith5-yearand10-yeargoals. The roleofnon-academicpartnershasbeenquestioned,aswellastheirinvolvementindisseminationactivities.

DeliverableWP6:Dissemination 6.1.EstablishandmaintaininteractiveprojectwebsiteinEnglishandlocallanguagesThe ECESM project website http://ecesm.net/ leaves the impression of carefully managed andupdated website on daily bases. The project website is comprehensive and user friendly. Itapparentlypresentsall important information, logos,downloadarea,andproject results.Themosttime-bound page is Outcomes/Outputs, which is well updated and administrated, so it is valuablesourceofup-to-dateprojectactivities.

Thewebsiteprovides also the information fordifferent stakeholders, so thatprospective students,companies,projectparticipants,aswellasEACEAcommittees,canfindouttheusefulinformation.

6.2Disseminationofintermediateandfinalresults

So far, the project has achieved the enviable visibility, not only in Montenegro, but also in theWesternBalkanregion.Thedisseminationactivitieswereofvariouskinds,asoriginallyplanedintheprojectproposal.Thelistofdisseminationactivitiesisasfollows:

Videopromotion,disseminationinpressandontheInternet

CyberSecurityThreats(inlocallanguage)-Video

Prepoznati značaj sajber bezbjednosti (Recognising characteristics of cyber security), Chamber ofEconomyofMontenegro(inlocallanguage)-Link




|Phone:+38622207105|Fax:+38622207272|

–Anarticleabouttheseminarwhichwasheldon16thFebruary2016attheChamberofCommerceinMontenegro,monthlymagazineGlasnik,printedin1500copiesanddistributedfreetoinstitutionsandcompaniesinMontenegro(inlocallanguage)-Link

Articles

R.Šendelj,F.Lombardi,I.Ognjanović,S.Guarino,CyberSecurityinMontenegro:Practice,FrameworkandChallenges,ProceedingsofINFOFEST2014Conference,28thSeptember-4thOctober2014,Milocer-Budva,Montenegro.

I.Bandara,F.Ioras,K.MaherI,CyberSecurityConcernsinE-LearningEducation,ProceedingsofICERI2014Conference,p.728-734,17th-19thNovember2014,Seville,Spain.

I.Ognjanović,R.Šendelj,I.Ognjanović,CybersecurityawarenessinMontenegro,ITConference,Žabljak2015,Montenegro

J.Ljucović,I.Ognjanović,R.Šendelj,AnalysisofcybersecurityeducationalsysteminMontenegro,ITConference,Žabljak2015,Montenegro

I.Bandara,F.Ioras,K.MaherI,C.Lusuardi,CyberSecurityChallengesofDistributedE-LearningSystems,INTED2015,the9thInternationalTechnology,EducationandDevelopmentConference,2ndMarch2015,Madrid,Spain.

R.Šendelj,I.Ognjanović,CyberSecurityEducationinMontenegro:currenttrends,challengesandopenperspectives,7thannualInternationalConferenceonEducationandNewLearningTechnologies(EDULEARN15),2015,Barcelona,Spain.

I.Vujačić,I.Ognjanović,R.Šendelj,"SM@RTHOME:personalsecurityanddigitalforensicissues",BISEC2016.Accesstothepaper.

Disseminationactivities(conferences)

5-6March2014,Risk2014,Lasko,Slovenia,Disseminationreportfromtheconference

12-15May2014,THENATOARW,Belgrade,Serbia,Managingterroristthreatstocriticalinfrastructure

30September2014,Infofest2014,Milocer-Budva,Montenegro,EnhancementofcybereducationalsystemofMontenegro

12-13November2014,CyberCrime2014,Tallinn,Estonia,Disseminationreportfromtheconference

23-27February2015,ITConference2015,Zabljak,Montenegro,Disseminationreportfromtheconference




|Phone:+38622207105|Fax:+38622207272|

–14-18June2015,IEEEConferenceMECO2015,Budva,Montenegro,Disseminationreportfromtheconference

22September2015,IEEEConferenceERK2015,Portorož,Slovenia,Disseminationreportfromtheconference

Disseminationactivities(other)

5-6April2016,UniversityofEconomicsinBratislava,Slovakia,Disseminationreport

5May2016,InstituteofModernTechnology,Montenegro,Disseminationreport

25May2016,Facultyoftourismandbusiness,Budva,Montenegro,Disseminationreport

24thJune2016,RadioAtlas,Podgorica,Montenegro,Disseminationreport

Nationaldisseminationevents(other)

2June2014,ChamberofEconomyofMontenegro,Podgorica,Montenegro,Disseminationreport

17September2014,ChamberofEconomyofMontenegro,Podgorica,Montenegro,Disseminationreport

10October2014,VIVIDConference,Ljubljana,Slovenia,Disseminationreportfromtheconference

16October2014,GluhinekočindanesvLjubljani,Ljubljana,Slovenia,Disseminationreportfromtheconference

28January2015,IFAMforum,Celje,Slovenia,Diseminationreportfromtheforum

11March2015,RoundtableofInformationSocietySlovenia,Maribor,Slovenia,Disseminationreportfromtheroundtable

18July2015,VisittoAtlasTelevisionPodgorica,Podgorica,Montenegro,Disseminationreportfromthevisit

8December2015,CybersecurityWorkshop(TheSlovenianSocietyInformatika),Maribor,Slovenia,Disseminationreportfromtheworkshop

16May2016,UniversityofDonjaGorica,Podgorica,Montenegro,Disseminationreport

25May2016,VisittoAtlasTelevisionPodgorica,Podgorica,Montenegro,Disseminationreport

26May2016,DisseminationattheSummerSchool"BeReadyforEUValues",Disseminationreport




|Phone:+38622207105|Fax:+38622207272|

–In total, 3 video promotions, 7 dissemination articles, 7 conference presentations, 4 otherdissemination audio/press activities, and 11 national dissemination activities various in type. Thesufficient number of dissemination activities is evident. However, it is not clear which of theseactivitiesaretargetedtotheprospectivestudentsand ifsomeoftheactivities involvedstudentsaswell.

6.3Startupannualregionalconference"Cybersecurityandsociety"

Theannualconference“CyberSecurity intheKnowledgeSociety” isnicely integrated inthepopularregional event InfoFest, and it was organized in 2014, 2015 and 2016. The conference providedinternationalspeakersonstate-of-the-arttopics incybersecurityandprovidesveryhighvisibility intheWesternBalkanregionamongdifferentstakeholders. It isnotevident ifconferencecanbeself-sustainableaftertheprojectend,soitisrecommendedthatconsortiumconsiderthisfact.

6.4StartuponlineNewsletter“Cybersecurityandsociety”

AccordingtotheDecisiononStart-uponlinenewslettertitled“Cyberinstructionsforall”,consortiumpublisheditasapeerreview,andcontainingthefollowingsections:welcomenote,articles,upcomingeventsincybersecurity,andprojectactivitiesinprogress.Uptonow,threeissuesarepublished:

- 1stOnlineNewsletter-17.5.2015- 2ndOnlineNewsletter-2.1.2016- 3rdOnlineNewsletter-18.5.2016

TheNewslettercontentisofaverygoodquality.Sinceitispublishedinnationallanguageitishighlyvaluablefornationalandregionalreadersandprospectivestudents.

DeliverableWP7:Sustainability

Aquiteextensiveeffortwasputintosustainabilityactivities.Firstofalltwoversionsofstrategywerecomposed.Thesedocumentsdefinefutureactivitiesandbasicallysuggestcontinuationofworkthatisalreadyseenintheproject:training,tuningtheeducation,promotionofcybersecurityawarenessand involving government stakeholders. The second version of strategy points out the ways ofachieving future goals. We find the strategy documents very useful and fully aligned with theproposal.

As an important part of sustainability the Montenegrin Cybersecurity Education Center (MCEC) isestablished. This activity goes fully in line with the proposal. The Center presentation is set as asubpageoftheIMTMweb-site(http://www.imtm.me/me-cyber-edu-center/).Valuablepartisthee-learningplatform,whichisbuiltforeducationincybersecurity.ThemaincontentontheplatformatthemomentisaNewsLetter"CyberSecurityInstructionsForAll",buttherearealsoworkshops.Threenewsletter issues have been published so far. GCSEC and IMTMhadmeeting onmatter of Centeractivities.




|Phone:+38622207105|Fax:+38622207272|

–IndicatorsofProgressTable below contains indicators of progress. It turned up that most deliverable met the proposalrequirements and are aligned with LFM. Some activities surpassed the requirements - such ashandbook creation, where two versions were produced, or dissemination, which was extremelyfruitful.However,thereisalackofdocumentationforthefourthwp.

IndicatorsofProgress

ReferenceNumber

TitleofDelivery/Activity Indicatorofprogress

(DEV)1.1 Existing EU practices for cybersecurity

Analysis of best practice in the field in EU: standards andrecommendationsthatmightbeusableforimplementationintheME.Report1.1produced.

(DEV)1.2 EU practice for cyber securityeducation

DiscussiononEUeducationinthefieldandidentificationofpossiblebenefits.Report1.2produced.

(DEV)1.3 Cross-matching of practice in MEwithEUstandards

Cross-analysis of degree inwhatMEmeets the current EUstandards in the field and identification of possibilities ofimprovement.Report1.3produced.

(DEV)1.4 Roadmap for new Cyber securityeducationinME

Definedlongtermandshorttermstrategiesforeducationinthefieldofcybersecurity.Report1.4produced.

(DEV)2.1 Analysis of the ME cyber securitypublicawareness

Analyses of the current state of cyber security publicawarenessinMontenegro.Findingsarenotedinreport2.1.

(DEV)2.2 Development of a draft cybersecurityframework

A draft methodological framework defined. Framework isdocumentedwithreport2.2

(DEV)2.3 Improvecybersecurityknowledge Variousformsofcommunicationwithgeneralpopulationonsubjectofcybersecurity(opendays,trainings).Trainingsaredocumented with agenda and minutes and(http://ecesm.net/trainings-within-wp2-improve-cyber-security-knowledge) aswell aswith links to online coursesusedforthissubject.Ontheawarenesstraining8teachersparticipated. On ISO 27000 training five teachers (futureinstructors) were trained. About 120 professors and ICTcoordinators participated in trainings in 5 cities in tworounds. A total of 91 university students were trained atMediterraneanUniversity.

(DEV)2.4 Cybersecurityhandbook Creation and publishing of Handbook defining roles,procedures,methodologiesandevaluationofcybersecurityresourcesforwidepopulation.Twoversionofhandbookareavailable.




|Phone:+38622207105|Fax:+38622207272|

–(DEV)3.1 Cross-matching of organizations

withEUstandardsAnalysis of current state of cyber-security knowledge incompaniesandorganizationsinME.Report3.1produced.

(DEV)3.2 Usable cyber security competenceframework

Defined the framework applicable in both industryenvironment and university (in organizations as mean ofimprovement of worker knowledge and as part of studyprogram).Frameworkisdescribedinreport3.2.

(DEV)3.3 Improve knowledge withinorganizations

Sixtrainingsessionsforworkforcewasconducted,with20-30 attendants each. Course plans described in the report.Handbookforprofessionalsproduced.

(DEV)3.4 Professionalization issues on cybercareerfields

Trends and expectations in cyber security field in MEdocumented.

(DEV)4.1 Analysis of cyber security masterprograms

AnalysisofEuropeanmasterprograms incybersecurityandcross-matchingwithME educational system documented inreport.

(DEV)4.2 Creation of multidisciplinarycurriculum

Master study program in cyber security on ME partneruniversities. Established regulation about the programs.MajoradaptationsofstudyprogramsatUDGandUNIM.

(DEV)4.3 Accreditation of multidisciplinarymasterprogram

Prepareddocumentationfortheaccreditationprocessexpected.Noreportfoundrelatedtothisactivity:theaccreditationisomitted,sincetheprocesswouldtooktoolong,howevertheadaptationsofexistingmatchingprogram(UDG)andrelatedmasterprogram(UNIM)implemented.

(DEV)4.4 Cyber security research anddevelopment

Promotionofresults,creationofaforumunitingindustryandacademy in research and student support.Already held workshops and conferences, as well ascollaborationplansbrieflydocumented.

(QPLN)5.1 Internal quality control andmonitoringreports.

QualityboardreportsonDEVpackagesproduced.

(QPLN)5.2 External quality control andmonitoring

Thisreport.

(QPLN)5.3 Inter-tempuscoaching OnemeetingorganizedwithtwootherTempus(BioEmisandSINC@HE)projects.

(DISS)6.1 Establish and maintain interactiveprojectwebsite

Web-siteisathttp://ecesm.netPageiswellstructuredandeasyfornavigationsoallresultsmaybequicklyfound.TheFacebookpage(https://www.facebook.com/TempusEcesm/)isupdatedregularlywithdifferentkindofinformation.However,ithasgotonly21subscribersasfornow.

(DISS)6.2 Dissemination of intermediate andfinalresults

A number of activities in formof conference presentations,summerschoolmeetings,TVguesting.Reportsavailable.




|Phone:+38622207105|Fax:+38622207272|

–(DISS)6.3 Startupannualregionalconference Conference"CyberSecurityintheKnowledgeSociety”isset

aspartofInfofest.

(DISS)6.4 Startuponlinenewsletter Newsletterisavailableathttp://ecesm.net/online-newsletterThreeissueshavebeenpublishedsofar.

(EXP)7.1 Sustainability strategy of projectresults

Sustainability strategyplansareproduced.Twoversionsareavailable.

(EXP)7.2 Buildcapacityforsecuringadequateresources

Assuranceofsufficientresourcesforfurtheraccreditationofstudy programs. This activity delivery date is 1.12.2016. Noreportisavailableatthemoment.

(EXP)7.3 Montenegrin Cyber securityEducation Centre

Establishment of the MCEC - ME Cyber security EducationCenter.SofarthesectionatIMTMweb-siteisdevotedtothecenter (http://www.imtm.me/me-cyber-edu-center/) withsupport for e-learning. Reports produced(http://ecesm.net/montenegrin-cyber-security-education-centre).




|Phone:+38622207105|Fax:+38622207272|

–

Overallfindings

Indicatorsofprogressexpected Indicatorsachieved

Strategy for new educational practicealigned with modern demands and EUexamples

Reports1.3and1.4produced.AroadmapbasedonEUstandardsandindustrybestpracticethatissupposedtoset ground for short term and long-term strategy forcybersecurityeducation.

Improved cyber security knowledge ofgeneralpopulation

Trainingsreportedin2.3.Ittargetedfutureinstructors,professionalsandstudents.Trainingswereorganizedinfivecities.Ahandbookforgeneralcitizenpopulationpublished.

MECyberEducationCenter

http://www.imtm.me/me-cyber-edu-center/

Improved cyber security knowledge inorganizations

Trainings prepared and conducted for industry andpublicsector.Requirementsanalysedandpresented inreportin3.3.Alsoahandbookproduced.

SixtrainingsessionorganizedinChamberofEconomy.

Cyber security incorporated in studyprograms

ExistingMasterstudyprogramincybersecurityatUDGmodified to reflect requirements alignedwith best EUpracticeinthefield.

(http://mastercybersecurity.me/sample-page/upis-na-magistarske-studije-sajber-bezbjednosti/)

Adapted program "master in information technology"ignited at UM. The accredited specialized cyber-securityprogramisuptobestartedinnextschoolyear.

ItisimportanttomentionthattheconsortiumalongwiththeProjectManagementBoard,directfromthe beginning took into consideration the Evaluation from EACEA and has proceeded with allproposedactions’revisions.

Documents

External Evaluation Report - TEMPUS ECESMecesm.net/sites/default/files/FINAL - External... · European Commission Tempus Project: 544088-TEMPUS-1-2013-1-SI-TEMPUS-JPHES This project