1

NIIF-Hungarnet

Assessing the Service Portfolioand Evaluating User Experiences/Demands

by Dedicated Static On-Line Surveysat NIIFI

Lajos Balintlajos.balint@niif.hu

NIIFI

TF-MSP meetingZagreb, 09-10.05.2012

2

Dilemmas and options

• The need of surveys (assessment and/or evaluation)• The way of surveying (motivation and methodology) • Living with and without surveys (NREN vs. users)

The big questions:

• What? (coverage, complexity, extent)• Why? (direct, indirect, and final objectives; external/internal)• When? (before-during-after preparing/executing activities)• Who? (who talks to whom, what expertise/responsibility level)• How? (traditional/online and static/dynamic/interactive)

3

Some crucial details

• Level of granularity (high-low: topics/users)

• Width of coverage (wide-narrow)

• Technique (traditional-online)

• Adaptivity (static-dynamic-interactive)

• Complexity (coverage vs. expertise)

• Output (availaility, accessibility)

• Frequency (single – repeating – periodical)

4

Recent NIIF/Hungarnet surveys (NIIFI)

Background:• ISO 9000 relations (external)• NDP project relations (external + internal)• Internal intentions (internal)

Execution:• Wide vs. narrow topical coverage (overall – VoIP)• High vs. low granularity (3000 – 30 invitees)• Single vs. periodical surveying (2010 – 2005+2n)• Static/dynamic on-line non-expert/expert responses• Internal exploitation, on-demand external access

5

Survey I – 2010-2011

• Wide SP coverage – discovering demands (RN [IP, λ], HPC, storage, cloud, VC, VoIP)

• Single execution (2010: NDP, ~2005: ISO)

• 2700 invited respondents (Y/N + annotation)(institutes, departments, research groups …)

• 90 incoming responses, mostly moderate value(applicable evaluation well supported)

• Started in 2010, concluded in 2011

• External motivation (meeting internal interest)(request by funding organisation)

6

Survey II – 2005,2007,2009,2011

• Narrow coverage – discovering directions(VoIP only, extended to collab.in 2009 by VC)

• Periodic execution(commensurable results fine tuning possible)

• 20-30 invited respondents(institutes, sometimes departments)

• 80-100 % incoming high value responses(expert, responsible respondents)

• Motivated by internal interest(+ high respondent willingness)

7

Survey II – additional info

• Evaluation (VoIP experiences):(net: size, call types in/out, technology, additional operator, cost, …)(use: , quality, support, invoicing, errors/failures, repair cycles, …)

• Development needs (VoIP demands)(improve support; improve invoicing; involve more institutes; provide directory;

apply selective promotion – user by user; introduce novel/additional technologies; …)

8

Experiences - corollaries• Result depends on motivation-devotion (both sides)• Improper objective or preparation prohibits success• Experts need demanding surveys, unlike non-experts • Even very low (3%) response level can be usable• Complex surveys mustn’t be frequently repeated• Repeated surveys should be commensurable• Interactive on-line surveys should have multilevel memory• Allocate time-energy-cost also to due post-processing• Resource demand: prepare + n * complete + nk * evaluate• Consider overall cost/benefit

Surveys sometimes can save but always do cost money!