Living Libraries Infrastructure Program Evaluation Web viewThe Living Libraries Infrastructure Program is a four-year grants program ... to the total cost of a major new library building

Evaluation of the Living Libraries Infrastructure Program Local Government VictoriaDepartment of Environment, Land, Water and Planning

3 February 2015

Contents

Contents........................................................2Executive summary........................................3

Background............................................................................................3Findings and recommendations.............................................................4

Introduction...................................................6Overview................................................................................................6Evaluation scope....................................................................................6Outcomes logic model.......................................................................8Evaluation methodology and data sources............................................8

Interviews...................................................................................................9Survey........................................................................................................9

Evaluation limitations.............................................................................9Findings........................................................10

Program delivery – summary data.......................................................10Appropriateness and relevance............................................................12

Key evaluation questions..........................................................................12Findings....................................................................................................12

Effectiveness........................................................................................15Key evaluation questions..........................................................................15Findings..................................................................................................15

Initiative delivery and efficiency..........................................................18Key evaluation questions....................................................................18Findings....................................................................................................18

Lessons and future directions..............................................................20Lessons on initiative design and delivery.........................................20Impact of stopping the initiative...............................................................21Conclusions and recommendations.................22

Conclusions..........................................................................................22Recommendations...............................................................................22

Attachments..................................................23Attachment 1: Evaluation design.........................................................23Attachment 2: Outcomes logic model..................................................30Attachment 3: Interview guide.............................................................31Attachment 4: Survey instrument........................................................33Attachment 5: Survey data..................................................................38Attachment 6: References....................................................................62Attachment 7: Australian Library and Information Association Statement on Public Library Services.........................................................................63Attachment 8: Australian National Audit Office Principles for Better Practice Grant Processes...................................................................................64

Living Libraries Infrastructure Program Evaluation Page 2 of 65

Executive summary

Background

The Living Libraries Infrastructure Program is a four-year grants program designed to assist regional libraries and local governments to provide high-quality and accessible public library infrastructure that meets the changing needs of communities. These grants are additional to and separate from core service funding to the library services by the state government.

The Living Libraries Infrastructure Program’s objectives are to:

• provide new or improved public library infrastructure;

• support the role of public libraries in strengthening communities and encouraging opportunities for community participation;

• encourage and create lifelong learning opportunities for Victorian communities; and

• facilitate free access to information and reading resources.

Grants range up to a maximum of $750,000. Around fifteen projects are approved each year. Sixty-three projects have been approved over the last four years, with $17.2m of LLIP grants contributing to projects of $180m total worth.

Local Government Victoria has commissioned an evaluation to provide information on the effectiveness of the Living Libraries Infrastructure Program and determine whether the program has met the appropriate policy objectives, planned outputs and desired outcomes. The evaluation is also to inform the government's current and future policy and decision-making around the program; a new business case will be developed early in 2015 for a proposed new round of the program.

The evaluation has assessed the program against the requirements of the Department of Treasury and Finance’s Evaluation Policy and Standards for Lapsing Programs:

• justification of the program and links to policy;• effectiveness;• funding/delivery;• risk; and• efficiency.

Evidence for the evaluation was gathered in December 2014 and January 2015. Evidence sources included:

• document and data reviews;• key informant interviews; and • an online survey of library service managers.

Findings and recommendations

Findings


The key findings included the following:

• The 17.2m of LLIP grants was evenly divided between metro projects ($8.1m, 47% of LLIP grants) and regional projects ($9.1m, 53%).

• The LLIP proportion of total projects’ expenditure was higher for regional projects (22% of total projects’ expenditure) than metro projects (6%).

• Stakeholders interviewed and surveyed for the evaluation were overwhelmingly in favour of the program and were unanimous in saying that much public library infrastructure would remain unbuilt or left in a dilapidated state in its absence.

• Grant processes were said to be open, relatively transparent, and easy to comply with. Local Government Victoria was praised for running a collaborative and simple grants system.

• Libraries that have implemented infrastructure projects with LLIP grants assistance report greater accessibility of services, increased usage rates and higher user satisfaction ratings as a result.

• The increasing cost of buildings means that the maximum grant amount of $750,000 is diminishing in significance compared to the total cost of a major new library building.

• Stakeholders interviewed and surveyed for the evaluation described the range of project sizes and scope funded by LLIP as being highly appropriate, as it responds to the range of needs across the sector.

Conclusions

The Living Libraries Infrastructure Program (LLIP) is a grant funding initiative that is seen by Victorian public library services as a vital and integral part of the sector’s annual infrastructure funding. It contributes around 10% on average of public libraries’ infrastructure spending and is considered to play a vital role in locking in counterpart funding from local Councils.

Grant processes are viewed by applicants as efficient, open and transparent, simple to comply with, and delivered in a collaborative manner between the Department and public library services.

Recommendations

It is recommended that:

1. The Department review the maximum grant size.

2. The Department review the rationale for deciding the limitations to non-fixed infrastructure for grant funding.

3. The Department review whether there is a case for articulating decision making guidelines on a policy basis for annual division of the grant funds pool between large, medium and small projects.


4. The Department articulate the grant decision-making principles in the guidelines made available to applicants.

5. The Department consider changing the timing of the grant process to better fit library services’ capital budget planning processes, for example by running multiple grant rounds each year or a continuously open round. There may be merit in running separate processes for large and small projects.


Introduction

Overview

Local Government Victoria supports public libraries through a range of grants and other funding and is also responsible for governance issues relating to regional library corporations and Mechanics' Institutes.

Local Government Victoria administers the Government's grants to libraries, including managing three year funding and service agreements with:

thirteen regional library services (each serving two or more councils);

thirty-three single-council library services; and

a state-wide library services for people with print disabilities.

These funding and service agreements cover two program areas:

core funding, which helps councils provide public library services; and

local priorities funding, which supports specific local services identified by the library as a priority.

The Living Libraries Infrastructure Program is a four-year grants program designed to assist regional libraries and local governments to provide high-quality and accessible public library infrastructure that meets the changing needs of communities. These grants are additional to and separate from core service funding to the library services by the state government.

The Living Libraries Infrastructure Program’s objectives are to:

provide new or improved public library infrastructure; support the role of public libraries in strengthening communities and encouraging

opportunities for community participation; encourage and create lifelong learning opportunities for Victorian communities;

and facilitate free access to information and reading resources.

Grants range up to a maximum of $750,000. Around fifteen projects are approved each year. The sixty-three projects approved over the last four years are at varying stages of completion.

Evaluation scope

Local Government Victoria has commissioned an evaluation to provide information on the effectiveness of the Living Libraries Infrastructure Program and determine whether the program has met the appropriate policy objectives, planned outputs and desired outcomes. The evaluation is also to inform the government's current and future policy and decision-making around the program; a new business case will be developed early in 2015 for a proposed new round of the program.

The evaluation has assessed the program against the requirements of the Department of Treasury and Finance’s Evaluation Policy and Standards for Lapsing Programs:


justification of the program and links to policy; effectiveness; funding/delivery; risk; and efficiency.

The top level questions and sub-questions have been provided in the brief, based on DTF requirements:

1. Justification / problem: What is the evidence of a continued need for the program? What is the role for government in delivering this program?

To what extent does the Living Libraries Infrastructure Program continue to address a demonstrable need and is responsive to the needs of Victorians?

To what extent have options been investigated to address the identified need or problem?

Does the Living Libraries Infrastructure Program continue to be the best way to respond to the problem and deliver the intended outcomes?

How have the economic, environmental and social conditions changed since the program was funded and how will continuation of the program meet these conditions?

2. Effectiveness: What is the evidence of the program’s progress toward its stated objectives and expected outcomes, including the alignment between program, its outputs (as outlined in Budget Paper 3), departmental objectives and any stated government priorities?

To what extent has Living Libraries Infrastructure Program progressed towards its stated objectives and outcomes it was seeking to achieve (at start-up and any revisions)?

Why was this program approach considered the best way to achieve the outcomes?

3. Funding / delivery: (a) Has the program been delivered within its scope, budget, within expected timeframes, and in line with appropriate governance and risk management practices; and b) Has the department demonstrated efficiency and economy in relation to the delivery of the program?

To what extent has Living Libraries Infrastructure Program been delivered within its scope?

To what extent has Living Libraries Infrastructure Program being delivered within its budget?

To what extent has Living Libraries Infrastructure Program been delivered within the expected timeframe?

To what extent has Living Libraries Infrastructure Program been delivered in line with appropriate governance and risk management practices?


To what extent has Living Libraries Infrastructure Program been delivered at lowest possible cost without compromising quality?

4. Risk: What would be the impact of ceasing the program?

What would the impact be if DTPLI was to successfully exit from delivering Living Libraries Infrastructure Program if the government so desired?

What strategies have been identified to minimise any negative impacts of this exit?

5. Efficiency: What efficiencies could be realised?

If ongoing funding was provided, what level of efficiencies could be realised?

Outcomes logic model

During the preparation of the evaluation plan the evaluator, in collaboration with LGV, developed an Outcomes Logic Model (OLM) for the LLIP. The OLM is attached at Attachment 2.

The OLM describes the program logic in a table to illustrate the links between policy, strategies, inputs, activities, outputs and outcomes.

The OLM provides a guide to the evaluation design and data collection by illustrating how the program is intended to work in producing outcomes for the beneficiaries and the community.

Evaluation methodology and data sources

The evaluation design is attached at Attachment 1. In brief the approach is as follows.

1. Develop Outcomes Logic Model2. Develop evaluation framework & key questions3. Develop interview questions and guide4. Develop survey questionnaire5. Collect data:

a. Performance data, including PLVN annual performance statistics.b. Key informant interviews of ten library managers.c. Online survey of library services.

6. Analyse7. Report


Interviews

Key informant interviews were conducted in December 2014 of ten library service managers selected at random1 from the list of grant recipients.

Survey

An online survey was run in January 2015. Forty-two library service managers responded, a 91% return rate.

Evaluation limitations

The evaluation has worked within several constraints:

1. The time for the evaluation research and analysis is limited (one working month).2. Evidence of the program’s long-term impact is necessarily indirect, as many

factors contribute to the creation of strong socially-inclusive, connected and active communities where people like to live, the LLIP being only one of them.

1 Grant recipients were assigned a sequence number and a random number generator app was used to select candidates for interview.


Findings

Program delivery – summary data

The LLIP has delivered $17.2 m of funding through sixty-three grants over the period 2011-14. Summary data is shown in below.

Table 1: Funding summary 2011-14

Year Number of applications

Number of grants made

Amount requested

Amount granted

2011 15 10 $5.2 m $3.4 m

2012 27 22 $10.8 m $5.5 m

2013 20 13 $8.6 m $3.5 m

2014 24 18 $8.5 m $4.8 m

TOTAL 86 63 $33.1 m $17.2 m

Significant competition for the grant funding is illustrated by the fact that applications were made for an amount of grant money almost double that available. At the same time, three-quarters of grant applications by number were successful.

The size of grants ranged from $6,000 to the maximum amount under the scheme, $750,000.

<$100,000

$100,000 to $199,999

$200,000 to $299,999

$300,000 to $399,999

$400,000 to $499,999

$500,000 to $599,999

$600,000 to $699,999

>$700,0000

5

10

15

20

25

Histogram of all LLIP grants funded 2011-2014

Size of grant

Num

ber o

f gra

nts

Figure 1: Distribution of grant size – LLIP grants 2011-2014

Table 2, over, shows projects funded by type.


Table 2: Projects funded, by type, 2011-2014

Project Type Number

New Library/Relocation 23

Redevelopment 16

Refurbishment 18

New or enhanced Mobile 6

Table 3 below shows grants made by location type.

Table 3: Grants data by location type

Metro projects 22

Metro projects total allocation $8,131,000

Metro projects total allocation (% of total $) 47%

LLIP proportion of total projects’ expenditure, metro 6%

Regional projects 41

Regional projects total allocation $9,069,000

Regional projects total allocation (% of total $) 53%

LLIP proportion of total projects’ expenditure, regional

22%

Table 3 above shows that:

Total grant monies over the period 2011-2014 are roughly evenly divided between metro and regional library services.

Grants make up a larger proportion of total projects’ funding in regional library services compared to metro library services’ projects.

Figure 2 below shows that while library services in regional locations received almost twice as many grants by number, the majority were for smaller amounts, clustered under $100,000.


<$100,000

$100,000 to $199,999

$200,000 to $299,999

$300,000 to $399,999

$400,000 to $499,999

$500,000 to $599,999

$600,000 to $699,999

>$700,000

02468

101214161820

Comparison of Metro & Regional LLIP grants 2011-2014

METROREGIONAL

Size of grant

Num

ber o

f gra

nts

Figure 2: Comparison of Metro and Regional LLIP grants 2011-2014

Appropriateness and relevance

Key evaluation questions

The key evaluation questions for Appropriateness and Relevance of the LLIP are as follows:

What is the evidence of a continued need for the program?

What is the role for government in delivering this program?





Findings

The Australian Library and Information Association (ALIA) articulates the need for public libraries in the following way (ALIA, 2009):

“Freedom of access to public library and information services is essential:


• to enable members of the Australian communities, including new residents and emerging communities to participate in all aspects of Australian life, including the democratic process;

• to actively contribute to social inclusion for all members of the Australian community; and

• to enable Australians to contribute to the economic wellbeing of their families and the nation.”

Public libraries in Victoria are entirely funded by government2, with the majority of funding, both operational and capital expenditure, by local government. The state government provides block funding for operational expenditures, and the LLIP for infrastructure capital expenditure. Library services may also be eligible for other state government grants, for example Community Support Grants.

Victorian public library services provided a total of 354 service points to the public, including 265 branches and thirty mobile services, according to the 2013-14 PLVN Performance Data report.

Significant research such as Libraries Building Communities (State Library of Victoria, 2005) and Dollars, Sense and Public Libraries (SGS Economics and Planning, 2011) has established the social and economic value of public libraries. Being The Best We Can (State Library of Victoria and Public Libraries Victoria Network, 2011), a libraries evaluation and improvement framework used by Victorian library services, summarised research as showing that libraries provide the following to their communities:

• provide gateways to information, learning and leisure;

• build individual skills, capability and wellbeing; and

• develop social connections and build social capital.

Successive governments have recognised the particular role of public libraries in fostering lifelong learning, and providing free access to information and a place for people to meet.

The LLIP is a response to identified needs around Victoria’s growth, changing community needs, the demands of ageing infrastructure, and a need to respond to modern developments in the library and information sector. Financial support is required to ensure that libraries’ infrastructure continues to meet Victorian communities’ diverse and growing needs,

The majority of Victorian public library services’ annual capital expenditure is provided by local governments: $29m of a total $33m in 2013-2014. The state government’s LLIP contributed an average $4.3m per annum to libraries’ infrastructure capital expenditure over the period 2011-2014. The LLIP funding made up 9.4% of the capital expenditure funding for projects by libraries over the period of the program. The proportion of the contribution made by LLIP to libraries’ projects is actually higher than this figure suggests, as many of the larger projects in the period were for multi-purpose buildings (e.g. Council ‘hubs’), so the library was only a portion of the total project expenditure.

2 With the exception of a small amount of funding raised through user charges and other sources.


The need for continuing investment in library services’ infrastructure is supported by interviewees and survey respondents in multiple comments about the value they have obtained from the LLIP and what would likely happen without it.

“Buildings have finite lifespans. The purpose for which they were built, and the changing nature of libraries, their buildings, and how they are being used and the purposes for which they are being used is changing and evolving. There is a need to recognise that by the state government. It is ‘public libraries’ still, despite what everyone says – it’s a partnership between local and state government I feel, and the contribution to the physical environment in which that happens is still very much a responsibility of both.” – library manager, metro Melbourne interface (high growth area).

The significance of the LLIP to the libraries’ infrastructure projects goes beyond the monetary value. Most library managers interviewed for the evaluation stated that without matching funding coming from the state government, their Councils would be much less inclined to fund the library projects.

“I think that you can’t underestimate just how critical the grants program is to actually getting commitment to capital funding. When you’ve got limited capital budget and you’ve got so many different services that are all trying to get that capital funding, when you can apply for a grant, it just adds so much weight to a capital bid to be able to have that with it …when it gets to executive or council if there’s a commitment of funding against it, they don’t want to knock back something where they’re potentially going to have co-funding.” – library manager, metro.

The continuing demand for, and responsiveness of the grant program is illustrated by the number of applications received over the period of the program. Of the forty-two respondents to the evaluation’s online survey, thirty-four (81%) had applied for a grant under the LLIP in the period 2011-2014. Thirty-three of these applicants were successful with one or more applications. Of the eight respondents who had not applied for an LLIP grant, five did not have infrastructure projects that would qualify under the LLIP, and one had an appropriate project that was already fully funded. One other respondent said that while they had not applied on behalf of their library service, three member councils of their regional library corporation had successfully done so.

Interviewees, survey respondents and research papers on Victorian public libraries reveal several other factors driving a continuing need for infrastructure investment:

Emergence of new technologies and evolution of established ones mean that buildings and library spaces need to change to meet the changing needs of ‘21 st

century literacies’.

Rapidly growing populations in metro Melbourne’s urban fringe regions are creating great demand for local services, including new and expanded public libraries.

The evolution of community use of local public spaces, especially public libraries, and their growth in use by the community as a safe ‘third space’ means that spaces in many long-established library buildings need to be reconfigured.


Effectiveness


The key evaluation questions for Effectiveness of the LLIP are as follows:

What is the evidence of the program’s progress toward its stated objectives and expected outcomes, including the alignment between the program, its outputs, departmental objectives and any stated government priorities?



Findings

The LLIP’s long-term goal is to support the creation of strong, socially-inclusive, connected and active communities where people like to live, by helping Victorian public libraries to:

strengthen communities and encourage opportunities for community participation;

encourage and create lifelong learning opportunities for Victorian communities; and

facilitate free access to information and reading resources.The LLIP does this by providing grant funding to public libraries for infrastructure, comprising part of projects’ funding requirements. As a result, public library services are able to provide new or improved library infrastructure (construction works, permanent fixtures or fittings, or renovation works) that benefits their users and the community.The LLIP has contributed $17.2m in sixty-three grants, towards infrastructure works of over $180m over the period of 2011-2014.

According to the economic modelling in Dollars, Sense and Public Libraries, for every dollar invested in Victoria’s public libraries, the average value returned is $3.56 in community benefits.

Survey respondents rated the contribution made by the LLIP grant highly, as shown in the following figure.


Strongly disagree

Disagree Neither agree nor disagree

Agree Strongly agree0

5

10

15

20

25

The LLIP grant made a significant contribution to the infrastructure project

Num

ber

Figure 3: Contribution by LLIP grant

As described in the previous section, many interviewees and survey respondents stated that the LLIP grant was vital in ensuring the project went ahead in the first place.

In addition, many smaller projects, especially in regional areas, received a significant proportion of their total project funding from the LLIP. A number of these involved small upgrades and/or expansions to smaller branches.

“We would not be able to provide the library facilities required by our growing community without Living Libraries funding.” – survey respondent.

Large library projects have the potential to deliver very significant impact:

“Our old library, when it was on this site, used to have attendance figures or visitor figures of up to 15,000 people a month. At the moment, we’re running at 25,000 visitors a month, so there’s been a significant increase in the number of people visiting the facility. Our loans were of the order of 15,000 or 16,000 a month, they’re now between 23,000 and 30,000 a month so there’s been also a significant increase in the loans. In addition to that, other things like bookings of meeting facilities, we’re probably running at something like 300 bookings a month.” – library manager, metro interface (high growth) region.

Not only have projects made impact on usage, but also on user satisfaction in the services, as shown in the following figure.


Strongly disagree



5

10

15

20

25

A measurable increase in user satisfaction has resulted from the infrastructure project

Num

ber

Figure 4: User satisfaction impact

One example of the type of impact achievable by a relatively small amount of infrastructure funding was shown by surveys of user satisfaction before and after the redevelopment of a library branch in a small coastal town. The rating given by users for the range and quality of books increased from 9% to 67%, while the ease of locating a book went from 55% satisfaction to 100%. Overall user satisfaction was 100% (‘Satisfied’ or ‘Very satisfied’).

Strongly disagree



5

10

15

20

25

Measurable increases in usage have been made possible by the infrastructure project

Num

ber

Figure 5: Impact on usage

Both quantitative and qualitative effects on usage were described by interviewees and survey respondents.


“Massive [impact] is probably the first word that comes to mind. In the first day we had 120 people visit the library, we picked up about 33 new members.” – library manager, regional area, describing the effect of opening a new static branch replacing a visiting mobile service.

“We’ll be able to make the library a breastfeeding friendly venue, and be a place where mums can duck in and their older kids can read a book while they feed their babies.” – metro library manager describing new spaces made possible by the infrastructure investment.

Another example was given where the accessibility of services was increased significantly, with infrastructure grants making it possible for two renovated branches to increase their opening hours from fourteen hours per week to five and a half days.

Initiative delivery and efficiency


The key evaluation questions for initiative delivery and efficiency are as follows:

To what extent has Living Libraries Infrastructure Program been delivered within its scope, budget and expected timeframe?



What would be the impact of ceasing the program?




Findings

The LLIP has been delivered according to its scope, budget and planned timing – see Table 1 for details.

The cost of administering the LLIP is quite low, with one Equivalent Full Time (EFT) staff administering $17.2m of grants over four years, and dealing with forty-six public library services and other sector stakeholders. This resource is supplemented by convening of a part-time Grants Moderation Panel, an independent body that assesses applications against the grant guidelines and recommends successful applications for funding.

The Commonwealth Grant Guidelines, cited in Implementing Better Practice Grants Administration (Australian National Audit Office, 2009), provide a framework of principles that underpin better practice grant processes. The principles are:


1. Robust planning and design

2. An outcomes orientation

3. Proportionality

4. Collaboration and partnership

5. Governance and accountability

6. Probity and transparency

7. Achieving value with public money

Further details of the grants principles framework are shown in Attachment 7. Overall, the LLIP performs well against these principles, as demonstrated by the following findings.

Grants are decided in an annual process that starts with an open call for applications in February/March. The application process is relatively simple, using an Expression of Interest (EOI) format. The Department works with applicants to help them focus their project applications before they lodge them.

Interviewees and survey respondents alike rated the grants processes highly.

Local Government Victoria dealt with queries and other communications from us promptly

The grant funding was made available when we needed it

The timing of the call for applications fitted into our planning cycle

The grants were advertised far enough in advance to allow adequate preparation

The grant application process was easy to understand

1 2 3 4 5

Rating of the LLIP grants processes

Rating (scale 1 to 5)

Figure 6: Survey ratings of LLIP grants processes

Assistance from Local Government Victoria was highly appreciated by library service applicants.

“It [the grants process] was really good. I think mainly because we sat down with the group straight away, and they explained the process and what was needed, and I found that process really good and helpful.” – library manager, metro Melbourne interface (growth) area.


“It’s one of the simpler processes that I’ve been through.” – library manager, regional area.

Several interviewees and survey respondents did raise some issues around timing of the grants process compared to the annual planning processes of their Councils, which sometimes made it more difficult to get projects up in a timely manner.

“Formal advice of success was not received until July 1. Councils finalise their budget process in April each year – it would be better if a result was known prior to finalising the commitments of Council for each operating year.” – survey respondent.

Decisions on grant applications are made by the Grant Moderation Panel, which contains representatives of the Department, Public Libraries Victoria Network (PLVN), the Municipal Association of Victoria (MAV) and the State Library of Victoria (SLV). Interviewees and survey respondents were satisfied that the grant requirements about the scope and type of projects eligible for funding were adequate and reasonable, and were clearly expressed in the published grant guidelines. However, several comments were made that questioned the consistency of decisions made.

A significant proportion of interviewees and survey respondents commented that the size of the maximum grant could reasonably be increased beyond $750,000 given the escalation in building costs for new libraries. This is a significant issue for larger interface Councils and Regional Library Corporations that are facing rapid population growth and are expected by the community to respond with major new buildings. Even though the LLIP contribution to such large projects may be a relatively small proportion of the whole, it can make a significant difference to the overall quality of the finished product and also to a Council’s willingness to proceed, according to a number of those interviewed for the evaluation.

Many interviewees commented that smaller grants often make very significant impacts on smaller library services, especially in regional areas. Grants even of under $100,000 can make large differences to the quality of users’ experience in smaller branches or mobile services.

Many interviewees and survey respondents commented that without the LLIP grant component, their project would not have proceeded. A common reason for this is that, on the one hand, the Council would not consider proceeding unless there was a matching component form the state government, and on the other hand that once funding was approved from the state government, it acted as a strong incentive to proceed, because Council did not want to forego promised funds. These comments were reinforced by those made by unsuccessful applicants, more than three-quarters of whom said their project did not proceed because they failed to win the LLIP funds.

All those interviewed and surveyed for the evaluation agree that were the LLIP to be terminated without replacement, the consequences for public libraries’ infrastructure in Victoria would be significantly negative. Political fallout could be expected as a result, according to a number of interviewees.


Lessons and future directions

Lessons on initiative design and delivery

Overall, the design appears appropriate and adequate for the purpose expressed in the program objectives. Applications run at about double the amount of funding available each year, so there is sufficient competition between grantees to ensure that good applications proceed. Unsuccessful applications are often fine-tuned between grant calls, resulting in better submissions that are more likely to succeed the following year. The Department plays an appropriate role in ensuring applicants put their best proposal forward.

A wide variety of comments were made by interviewees and survey respondents about potential improvements to the LLIP. The main themes were to:

Increase the maximum grant size (some suggested a figure of $1m).

Widen the scope of applicable infrastructure to cover a defined range of furnishings (restrictive definitions of fixed infrastructure can limit libraries’ ability to respond to changing usage needs).

Many of those interviewed and surveyed commented on the wide range of projects funded under the LLIP. This was very much seen as a positive, because no-one was excluded. The differing levels of matching funding on offer were widely commented upon as being appropriate, as they reflected differing levels of need and financial circumstances between library services in metropolitan Melbourne, regional cities, metro interface areas, and regional and rural councils.

Impact of stopping the initiative

It is unlikely the Department could successfully stop delivering the initiative without there being significant backlash from the public library sector, local government and the community, potentially leading to political fallout. The state government is seen to be responsible for providing public library services in collaboration with local government, and cessation of the LLIP could be interpreted as withdrawal of the state government from its publicly stated commitment to supporting and enhancing the quality of public library services in Victoria.


Conclusions and recommendations

Conclusions

The Living Libraries Infrastructure Program is a grant funding initiative that is seen by Victorian public library services as a vital and integral part of the sector’s annual infrastructure funding. It contributes around 10% on average of public libraries’ infrastructure spending and is considered to play a vital role in locking in counterpart funding from local Councils.

Grant processes are viewed as efficient, open and transparent, simple to comply with for applicants, and delivered in a collaborative manner between the Department and applicant public library services.

Recommendations

It is recommended that:

1. The Department review the maximum grant size.

2. The Department review the rationale for deciding the limitations to non-fixed infrastructure for grant funding.

3. The Department review whether there is a case for articulating decision making guidelines on a policy basis for annual division of the grant funds pool between large, medium and small projects.

4. The Department articulate the grant decision-making principles in the guidelines made available to applicants.

5. The Department consider changing the timing of the grant process to better fit library services’ capital budget planning processes, for example by running multiple grant rounds each year or a continuously open round. There may be merit in running separate processes for large and small projects.


Attachments

Attachment 1: Evaluation design

The evaluation design detailed below was designed to meet DTF requirements, including recommendations for future program delivery improvements, based on evidence gathered in the evaluation.

Purpose of the evaluation.

The purpose of the evaluation is to provide information on the effectiveness of the Living Libraries Infrastructure Program and determine whether the program has met the appropriate policy objectives, planned outputs and desired outcomes. The evaluation will also inform the government's current and future policy and decision-making around the program – a new business case will be developed early in 2015 for a proposed new round of the program.

Scope of the evaluation.

The evaluation will comprise:

1. Collection and analysis of data relating to library services that have received funding as part of the Living Libraries Infrastructure Program;

2. Analysis of relevant reports and surveys of library professionals; and

3. A review of relevant documents.

The main quantitative data collected will be derived from the 2013–14 public library statistics compiled by PLVN on behalf of Local Government Victoria. Additional data may be sourced from specific library services via formal surveys to help ascertain the benefit of facilities built as part of the living libraries infrastructure program. Qualitative data will be obtained through evaluation research, and LGV will contribute case study information from a representative sample of eight libraries from around the state.

Target audiences.

The main audience for the evaluation is Local Government Victoria, which will use the evaluation as part of its budget submission preparation.

Key evaluation questions.

The evaluation will assess the program against the requirements of the Department of Treasury and Finance’s Evaluation Policy and Standards for Lapsing Programs:

1. justification of the program and links to policy;

2. effectiveness;

3. funding/delivery;

4. risk; and

5. efficiency.


The detailed evaluation questions to be answered are as follows:

1. Justification / problem: What is the evidence of a continued need for the program? What is the role for government in delivering this program?





2. Effectiveness: What is the evidence of the program’s progress toward its stated objectives and expected outcomes, including the alignment between program, its outputs (as outlined in Budget Paper 3), departmental objectives and any stated government priorities?



3. Funding / delivery: (a) Has the program been delivered within its scope, budget, within expected timeframes, and in line with appropriate governance and risk management practices; and b) Has the department demonstrated efficiency and economy in relation to the delivery of the program?

To what extent has Living Libraries Infrastructure Program been delivered within its scope?

To what extent has Living Libraries Infrastructure Program being delivered within its budget?

To what extent has Living Libraries Infrastructure Program been delivered within the expected timeframe?



4. Risk: What would be the impact of ceasing the program?




5. Efficiency: What efficiencies could be realised?


Logic model.

The program outcomes logic model for the Living Libraries Infrastructure Program is shown in Attachment 2.

Interview guide and survey questionnaire.

The interview guide is shown in Attachment 3; the survey questionnaire is shown in Attachment 4.

Evaluation plan and timetable

The evaluation plan timetable is shown in Figure 7 below, and explained in the following paragraphs.

Week CommencingDec Dec Dec Dec Dec Jan Jan Jan Jan Feb

Outputs 01 08 15 22 29 05 12 19 26 02

1. PLANCommencement meeting Contract & draft workplan

Review documentationDevelop program logic model PLM

Stakeholder analysis Stakeholder engagement plan

Develop evaluation framework & questions Framework & KEQs

Develop interview & survey questions & tools Questionnaires, interview guides

Finalise methodology and evaluation plan Evaluation Plan

2. COLLECT DATAConduct stakeholder interviews Qual. data

Collect & review library data Quant. data

Conduct online survey Survey data

3. ANALYSE & REPORTAnalyse quantitative & qualitative data Thematic analysis

Assess improvement opportunities Recommendations

Draft evaluation report Draft evaluation report

Circulate for review Revisions requested

Finalise report Final evaluation report

PHASE & Tasks

1

3

2

= milestone deliverable

Figure 7: Evaluation timetable

Commencement meeting

At the commencement meeting we will confirm the evaluation scope as being to determine the effectiveness, efficiency and equity of program delivery and to assess achievements against intended program objectives and outcomes.


The evaluation will consider:

Program justification: the need that the program addresses;

Program effectiveness: the level of achievement of the program outcomes, including an assessment of success factors and constraints and linkages to departmental objectives and stated government priorities;

Funding/delivery and efficiency: program management and processes;

Risk: impact of ceasing the program; and

Efficiencies realisable in a continuing program.

Review documentation

At the commencement meeting, or immediately following it, LGV will provide as much of the relevant program documentation as is practicably available for us to review.

Develop program logic model

As this evaluation examines outcomes and also seeks to discover program improvement possibilities, we will start by articulating in summary form the program design and implementation steps and the objectives they were intended to achieve.

Working from program documentation and discussions with LGV, we will formulate an overview of the program logic, articulating how resources were intended to be mobilised, what processes were used for implementation, what outcomes were expected, and what longer-term outcomes are anticipated.

Stakeholder analysis

In consultation with the project manager, ACIG will develop a list of stakeholders to be consulted and develop a consultation plan.

We expect that we will consult selected stakeholders through an appropriate mix of face to face and telephone interviews.

Develop evaluation framework & key questions

Having confirmed the overall program logic, we will use it to inform the evaluation framework.

To frame the evaluation, we will firstly confirm in consultation with the evaluation project manager the set of questions that the evaluation is to answer. The top level questions and sub-questions have been provided in the brief, based on DTF requirements.

The detailed list of evaluation questions will form the basis for data collection tools such as interview plans and surveys.


Develop interview questions & tools

To ensure consistency we will develop an interview plan that lists key discussion points to be covered during the course of the interview, without circumscribing the conversation. We will use the interview plan as a prompt, then probe the interviewee for further depth or follow up on issues raised.

Finalise methodology and evaluation plan

We will then finalise the detail of the evaluation plan, working from the agreed framework. The evaluation plan will include:

1. Purpose of the evaluation.

2. Scope of the evaluation.

3. Target audiences.

4. Key evaluation questions.

5. Logic model – diagram and narrative.

6. Evaluation indicators – both quantitative and qualitative.

7. Methods for key informant engagement and data collection, aggregation and analysis.

8. Interview guides.

Collect Data

Both quantitative and qualitative data will be collected for analysis in the evaluation. Our primary sources of evidence will be: quantitative data provided by LGV; documents provided by LGV; survey data; and key informant/stakeholder interviews. We will create a data collection matrix that will serve both as a data collection guide and as cross-referencing summary table for the initial analysis.

Quantitative data analysis

We will analyse data provided by LGV that may include (but is not limited to):

Program funding and resourcing;

Data evidencing before-project and after-project situations, to enable assessment of change attributable to a sample of projects funded by the Living Libraries Infrastructure Program grant, for example:

- Spaces (e.g. amount, type, availability for specific usages)

- Collection stock (e.g. type, quantities, average age)

- Membership numbers

- Loans

- Availability and accessibility data


Qualitative data analysis

The following qualitative data will also be analysed:

Existing qualitative feedback received from grant applicants;

Review of grant project outcomes;

Information from surveys of and interviews with grant applicants and LGV program officers;

Information on the Living Libraries Infrastructure Program grants process, including program guidelines and evidence of the implementation of the process; and

Examine the most efficient program delivery mechanism (i.e. the market place or government) where possible.

Online survey

We propose developing a brief online survey to ascertain the benefit of facilities built as part of the Living Libraries Infrastructure Program. The survey will be sent directly to grant recipients. We have over 20 years of experience in survey design and analysis, so we know how to structure surveys in order to obtain clear and practical results.

Key informant interviews

Based on our understanding of the brief, we anticipate up to twelve key informant/stakeholder interviews, depending on the final stakeholder analysis. Although the Terms of Reference do not explicitly mention interviews, we regard interviews with key stakeholders an essential part of the data collection, for triangulation purposes.

We will seek LGV’s assistance in initially contacting interviewees and validating our interview request. Most interviews will be less than one hour long, and we expect to be able to conduct most (if not all) of them by telephone.

Document review

We will carefully examine the program documentation provided to seek evidence about processes that were implemented, and outcomes that have been achieved through those processes. These data will be cross-referenced in the data collection matrix. We will include review of information on best practice government grants administration processes such as the ANAO guidelines.

Preliminary analysis

Having tabulated the relevant evidence in the data collection matrix, we will examine each evidence cluster to assess the extent of achievement of intended outcomes to date. We will review the documentation for evidence of unintended outcomes (positive or negative) and, if necessary, follow up with enquiries for clarification.


Assess improvement opportunities

We will review the complete data set – quantitative and qualitative – to develop a list of potential program improvements. We will consult further with LGV to refine those opportunities for improvement, as it is important at this time that we leverage the in-depth program knowledge held by department staff.

Draft report

We assume that the audience for this evaluation report is internal to government. The report is to meet DTF’s Evaluation Policy and Standards for Lapsing Programs and (as specified by DTF) shall have no more than thirty pages in the body. Our findings will be documented in this report, which will address the core lapsing program questions.

The draft report will include the following sections.

Executive Summary

Introduction

Findings

Conclusions and recommendations

We will present a final draft of the report to the evaluation project manager for any final comments, before circulating it for wider review.

Circulate report for review

We will rely on DTPLI/LGV to ensure the review is completed and comments and revisions returned to us in time for the report to be finalised according to the department’s timetable.

Finalise report

Following the report review period we will finalise the report.


Attachment 2: Outcomes logic modelStrategies

Encourage investment in new and improved public library infrastructure by part-funding projects by library services with grants up to $750,000

Goal

Provide support by renewing public library infrastructure and addressing gaps in service provision.

Policy

The Victorian Government recognises the particular role of public libraries in fostering lifelong learning, and providing free access to information and a place for people to meet.

Problem

Victoria is growing, and support is required to ensure that libraries continue to meet Victorian communities’ diverse and growing needs, and respond to the demands of ageing infrastructure, changing community needs, and modern developments in the sector.

Inputs

Government funding.

Activities

Annual infrastructure grants program.

Outputs

Grant funding provided to public libraries for infrastructure, comprising part of projects’ funding requirements.

Clients

Potentially – all fifty-two Victorian public library services.

Key stakeholders

Victorian public library services.

Local Government Victoria

Public Libraries Victoria Network

Municipal Association of Victoria

OutcomesShort-term

Public library services provide new or improved library infrastructure (construction works, permanent fixtures or fittings; or renovation works).

OutcomesMedium-term

Victorian public library facilities improved.

Regional libraries and local governments provide high-quality and accessible public library infrastructure.

OutcomesLong-term (Impact)

Public libraries supported in:

strengthening communities and encouraging opportunities for community participation;

encouraging and creating lifelong learning opportunities for Victorian communities; and

facilitating free access to information and reading resources.

Strong socially-inclusive, connected and active communities where people like to live.


Attachment 3: Interview guide

Objective

Grantee interviews for the evaluation explore the following key evaluation questions: To what extent has the program met stakeholders’ expectations? What outcomes has the program achieved? What would happen if the program (or similar infrastructure upgrade/replacement

funding process) did not exist? Is there a continuing need for the program and does LGV have a continuing role to

play in improving public libraries’ infrastructure?

Introduction

Thank the interviewee for giving the time for this interview. Ask permission to record the interview, to enable accurate note-keeping.

Local Government Victoria has engaged the Australian Continuous Improvement Group (ACIG) to undertake an evaluation of the Living Libraries Infrastructure Program. ACIG is an independent management consultancy engaged in evaluation, benchmarking and continuous improvement for clients for more than twenty-two years.

This independent evaluation will be conducted within the Department of Treasury and Finance guidelines and will be used to inform the development of the 2015 budget funding submission.

Data collection for the evaluation includes document review, surveys and interviews. You have been chosen as part of a random sample of the 63 projects funded over the past four years of the program. We would like to ask you some questions about the program and your experience of it.

Please feel free to be open and frank in their answers. All information will be de-identified in our report to LGV.

(Be prepared to answer questions about what access there will be to the final evaluation report.)

Opening

Confirm the interviewee’s role and the type and scope of their involvement with the program: grants applied for; grants won; projects completed and in-progress.

Grant application(s)

What has your experience been with the grants process?

Were there any problems?

How could the process be improved?

Project delivery

Was funding made available when it was needed? As planned?

Were any obstacles encountered? How were they dealt with?

What could have been done better? In what way?


How well do you feel the LGV program office communicated with stakeholders?

Were there any issues? What caused them? What could have been done better?

Outcomes

What outcomes were achieved as a result of the grant funding?

Do you have data on improved outcomes, e.g. increased usage of library services, changes in community satisfaction, increased access to resources, etc.?

How significant to the project outcomes was the LLIP program funding received?

What would have happened if you had not received the LLIP funding for the project? Would the project have still proceeded? What changes would have been necessary?

Do you have any comments about:

the size of the grants offered, the conditions of the grant, or what types of library services are or should be eligible for them?

Do you feel there is a continuing need for this type of grant funding for library infrastructure from the State Government?

If the program ceased, what would be the impact?

Closing

Thank the interviewee once again for their time. Leave them with contact details, should they have any further comments or questions.


Attachment 4: Survey instrument







Attachment 5: Survey data
























Attachment 6: References

Australian Library and Information Association. 2009. Statement on public library services. Canberra.

Australian National Audit Office. 2010. Implementing Better Practice Grants Administration. Commonwealth of Australia. Canberra.

SGS Economics and Planning. 2011. Dollars, Sense and Public Libraries. State Library of Victoria. Melbourne.

State Library of Victoria. 2005. Libraries Building Communities – The Vital Contribution of Victoria’s Public Libraries – A Research Report for the Library Board of Victoria and the Victorian Public Library Network. Melbourne.

State Library of Victoria and Public Libraries Victoria Network. 2011. Being The Best We Can – Key Results for Public Library Services. Melbourne


Attachment 7: Australian Library and Information Association Statement on Public Library Services

Principle

Freedom of access to public library and information services is essential: to enable members of the Australian communities, including new residents and

emerging communities to participate in all aspects of Australian life, including the democratic process;

to actively contribute to social inclusion for all members of the Australian community; and

to enable Australians to contribute to the economic wellbeing of their families and the nation.

Statement

Each member of the Australian community has an equal right to public library and information services regardless of age, race, gender, religion, nationality, language, disability, geographic location, social status, economic status and educational attainment.

A public library services its community through the provision of access to knowledge, information and works of imagination through a range of resources and services. It does this through access to materials in all formats in order to meet the needs of individuals and groups for education, information and personal development through learning, including recreation and leisure.

Public libraries have an important role in the development and maintenance of a democratic society by giving individuals access to a wide and varied range of information, ideas, opinions, and skills.

The role of public libraries is essential in developing an educated society through programs that improve literacy and information literacy including lifelong learning opportunities. Public libraries contribute to economic prosperity by helping people improve their skills and life chances.

Public libraries provide the first point of access for information for the general public and for the public's access to the national and international system of library and information services.

The satisfaction of a person's information needs must be independent of an ability to pay.

Local, state/territory and Commonwealth governments have an obligation to work in partnership to provide agreed public library services to all members of the library's clientele without direct charge to the user.

The Australian Library and Information Association believes that public library services have particular responsibilities to monitor and proactively respond to the changing demographic characteristics and trends of their communities, to consult with their communities and to meet information, learning and recreational needs of an increasingly diverse society. Public library services should ensure that they have policies, guidelines, and procedures in place to respond to and meet relevant legislative requirements.


Attachment 8: Australian National Audit Office Principles for Better Practice Grant Processes

The Commonwealth Grant Guidelines, cited in Implementing Better Practice Grants Administration (Australian National Audit Office, 2009), provide a framework of principles that underpin better practice grant processes.

The principles are:

1. Robust planning and design which underpins efficient, effective and ethical grants administration, including through the establishment of effective risk management processes.

2. An outcomes orientation in which grants administration focuses on maximising the achievement of intended government outcomes from the available funding.

3. Proportionality in which key program design features and related administrative processes are commensurate with the scale, nature, complexity and risks involved in the granting activity.

4. Collaboration and partnership in which effective consultation and a constructive and cooperative relationship between the administering agency, grant recipients and other relevant stakeholders contribute to achieving more efficient, effective and equitable grants administration.

5. Governance and accountability in which a robust governance framework is established that clearly defies the roles and responsibilities of all relevant parties; establishes the policies, procedures and guidelines necessary for defensible funding recipient selection and administration processes that comply with all relevant legal and policy requirements; and supports public accountability for decision-making, grant administration and performance monitoring.

6. Probity and transparency in which program administration reflects ethical behaviour, in line with public sector values and duties; incorporates appropriate internal and fraud control measures; ensures that decisions relating to granting activity are impartial, appropriately documented and publicly defensible; and complies with public reporting requirements.

7. Achieving value with public money which should be a prime consideration in all aspects of grant administration and involves the careful consideration of costs, benefits, options and risks.


Documents

Living Libraries Infrastructure Program Evaluation Web viewThe Living Libraries Infrastructure Program is a four-year grants program ... to the total cost of a major new library building