92

6

Objective Evaluation of Recreational Facilities: Development and Reliability of the Recreational Facility Audit Tool

Volume 31, Number 4 pp. 92-109

Journal of Park and Recreation AdministrationWinter 2013

Ka Yiu LeeDuncan MacfarlaneEster Cerin

EXECUTIVE SUMMARY: The characteristics of recreational facilities are determinants of facility use and physical activity, yet there are few validated and extensive audit tools gauging characteristics of recreational facilities. This study aimed to describe the development of a 111-item audit tool that objectively evaluates the characteristics of recreational facilities across 10 domains, including the availability of sports facilities, accessibility to the facility, availability of supportive amenities, conditions of changing rooms, conditions of toilets, management, policy, environmental safety, aesthetics, and social environment. The intra- and inter-rater reliability indices of the audit tool were also examined. In the intra-rater reliability tests, 20 recreational facilities were examined twice by the same auditor, with one week apart between each audit. The inter-rater reliability was assessed using data from 20 recreational facilities that were examined twice by two independent auditors. Kappa statistics, percent agreement and intra-class correlation coefficients were computed. The average intra- and inter-rater percent agreement was 96.9% and 90.6% respectively. The majority of items (n=107 and 101) showed either good percent agreement or acceptable kappa values in the intra- and interreliability tests respectively. The overall intra- and inter-rater reliability of the audit tool was acceptable. Subjective and time-sensitive audit items may need clearer descriptions and multiple repeated measures to improve their reliability. The audit tool can be used to reliably assess indoor sports centres, parks, sports grounds, playgrounds and swimming pools. The objective data collected by this audit tool would contribute to the understanding of relationships between characteristics of recreational facilities, usage and physical activity.

KEYWORDS: Objective measures, recreational facilities, physical activity, characteristics, reliability, audit tool

AUTHORS: Ka Yiu Lee is with the School of Nursing, University of Hong Kong, Room 432A, 4/F, William MW Mong Building, 21 Sassoon Road, Hong Kong, phone: 852-98354542, e-mail: kyle2012@hku.hk. Duncan Macfarlane is with the Institute of Human Performance, University of Hong Kong. Ester

93

6

Cerin is with the Institute of Human Performance, University of Hong Kong, and School of Exercise and Nutrition Sciences, Deakin University, Burwood, Australia.

ACKNOWLEDGMENTS: The authors would like to thank Justin Cheong (former research assistant in the University of Hong Kong) for his contribution in the data collection.This research was primarily supported by the RGC (#HKU747807H, #HKU740907H), and also by the University of Hong Kong’s URC-SRT (Public Health).

Engagement in regular physical activity has been associated with a variety of health benefits (Dwyer et al., 2011; Hamilton, Swan, & Jamal, 2010; Pate et al., 1995; Whelton, Chin, Xin, & He, 2002). Although the benefits of engaging in physical activity have been well documented, inactive lifestyles are prevalent in many developed countries. For example, in the United States, less than 5% of adults accumulate health-enhancing levels of physical activity (Troiano, Berrigan, Dodd, Masse, Tilert, & McDowell, 2008). Consequently, potential determinants of physical activity have been examined and efforts have been made to identify aspects of the built environment predictive of physical activity behavior (Adams et al., 2011; McCormack, Rock, Toohey, & Hignell, 2010; Sallis et al., 2009).

Some key aspects of the built environment that correlate with physical activity include the availability of recreational facilities (Sallis et al., 2009). However, the mere presence of nearby recreational facilities may not be enough to influence physical activity (Hoehner, Brennan Ramirez, Elliott, Handy, & Brownson, 2005) as the specific characteristics of recreational facilities may actually determine their usage and hence the amount of physical activity performed there. For example, any signs of incivility such as graffiti and vacant buildings near recreational facilities are believed to discourage facility use (Coen & Ross, 2006), while the presence of people being active, absence of illegal activities (Ries, Gittelsohn, Voorhees, Roche, Clifton, & Astone, 2008), good aesthetics, and the availability of amenities (McCormack et al., 2010) have been found to encourage facility use. Similarly, recreational facilities with good lighting (Cohen et al., 2006) and neighbourhood safety (Booth, Owen, Bauman, Clavisi, & Leslie, 2000) were also positively associated with physical activity. Given the significant role of recreational facilities on physical activity, reliable and validated audit tools are essential to study the relationships between different characteristics of recreational facilities, facility use and physical activity.

Literature ReviewSeveral audit tools have been developed and used to assess the characteristics

of recreational facilities (Bedimo-Rung, Gustat, Tompkins, Rice, & Thomson, 2006; Cavnar et al., 2004; Crawford et al., 2008; Cunningham, Michael, Farquhar, & Lapidus, 2005; Saelens et al., 2006; Troped et al., 2006). The assessments generally focus on physical characteristics and quality. Examples of physical characteristics included the number and type of sport facilities available (Crawford et al., 2008), accessibility which has been positively associated with facility use and physical activity (Giles-Corti & Donovan, 2002), as well as the presence of amenities such as drinking fountains, rubbish bins, seating and toilets (Bedimo-Rung et al., 2006). Although these amenities may not be directly associated with the amount of physical activity performed within recreational facilities, they are likely to encourage or discourage people to make visitations (Bedimo-Rung et al., 2006). In contrast, the assessment of quality include the aesthetics, cleanliness, safety, general conditions (Cavnar et al., 2004; Saelens et al., 2006) and social environment within a recreational facility, all of which were found to influence facility use (Ries et al., 2008).

94

Bedimo-Rung et al. (2005) conceptualized a model to describe environmental characteristics of the recreational facilities which may correlate with physical activity and facility use. These characteristics included the features, condition, accessibility, aesthetics, safety and policies within specific geographical areas (i.e., activity areas in the park and the nearby neighborhood). Subsequently, several recreational facility audit tools emerged and extended the dimension and depth of environmental measurement of recreational facilities. For instance, Crawford et al. (2008) introduced the Children’s Public Open Space Tool (C-POST) which examined several aspects of public open spaces, including the number of sport facilities, presence of amenities such as rubbish bins, seating, drinking fountains, barbecue facilities, kiosks, toilets, sheltered areas, water features, trees and shade, lighting along the path and signage of restricted access to dogs and other activities, with most items based on a dichotomized scale.

A larger audit tool called the Environmental Assessment of Public Recreation Spaces (EAPRS) which contains 1,088 items (Saelens et al., 2006), evaluated the characteristics of parks and playgrounds, including trails, paths, the availability of amenities, water areas and shade, and playground equipment. For each, their corresponding number and conditions (e.g., cleanliness, aesthetics, flatness, dimension, visibility and proximity) were evaluated. Some items measuring the dimension of amenities were based on a 3-point scale, while those measuring the quality of facilities were evaluated using a 5-point Likert-scale. The inter-rater reliability of the EAPRS was good, particularly for the assessment of physical characteristics (e.g., the presence or absence of a facility), but the reliability was lower for the quality assessment (e.g. the aesthetics and cleanliness). The results indicated that the audit tool was more reliable with regards to objective environmental characteristics which do not require subjective ratings.

Lower reliability of subjective items was also observed in another study (Bedimo-Rung et al., 2006) which, based on a 181-item audit tool, examined the characteristics of green spaces, paths, playgrounds, sports fields and courts across five domains, including the features, access, aesthetics, condition and safety. The Bedimo-Rung Assessment Tool (BRAT) showed in general good inter-rater reliability of 86.9% (Bedimo-Rung et al., 2006) for the assessments of domains and 87.5% for the assessment of a variety of geographical areas. Similarly, Troped et al. (2006) introduced the Path Environment Audit Tool (PEAT) which assessed the design, amenities, aesthetics and maintenance of trails and paths, while Cavnar et al.’s (2004) 61-items audit tool assessed the safety, maintenance and conditions of various types of recreational facilities, including parks, playgrounds, football and soccer fields, basketball and tennis courts, as well as aquatic facilities.

Despite the variety of audit tools that have been developed, the applicability of these specific instruments is limited. For example, Crawford et al. (2008) focused on the number and availability of sports facilities and amenities, while Bedimo-Rung et al. (2006), Troped et al. (2006) and Saelens et al. (2006) focused mainly on the characteristics of parks, trails and playgrounds. In contrast, Cavnar et al.’s (2004) audit tool examined a variety of recreational facilities, but it focused only on their safety, maintenance and condition. Consequently, the existing recreational facility audit tools are not well suited to examine other types of facilities (e.g., swimming pools), nor other relevant characteristics of recreational facilities that potentially could facilitate or hinder facility use and, hence, users’ physical activity habits. These additional factors could include: policy (i.e., regulations imposed on facility users), conditions of toilets and changing rooms (i.e., provision of clean toilets and hot showers), social environment (i.e., crowdedness of the facility), management (i.e., regular maintenance practices), and accessibility to the facility (i.e., availability of public transport).

In an attempt to fill the above research gaps, this study aimed to develop an objective audit tool, named the Recreational Facility Audit Tool (RecFAT), which would have the ability to evaluate a wide range of characteristics of indoor and outdoor, aquatic and non-aquatic recreational facilities in urban locations, and then to test its intra- and inter-rater reliability. The objective data collected by the RecFAT could contribute to the

95

understanding of relationships between the characteristics of recreational facilities, facility use, and physical activity.

Method

Development of the Recreational Facility Audit Tool (RecFAT)The Recreational Facility Audit Tool (RecFAT) was developed to objectively measure

the characteristics of recreational facilities across 10 domains: the availability of sports facilities, accessibility to the facility, availability of supportive amenities, conditions of changing rooms, conditions of toilets, management, policy, environmental safety, aesthetics, and social environment.

The RecFAT was designed after conducting a literature review of relevant studies that examined the features of recreational facilities (Bedimo-Rung et al., 2006; Cavnar et al., 2004; Cunningham et al., 2005; Saelens et al., 2006; Troped et al., 2006), then a series of meetings with a panel of experts were undertaken to design the items of the RecFAT. The panel included health and leisure management specialists, as well as physical activity researchers. Discussions were conducted in which relevant domains of characteristics of recreational facilities, and items underlying each domain, were selected and modified from the reviewed studies. New domains and items were designed and added to the audit tool in order to ensure the content of the RecFAT was comprehensive.

Ten Domains of the RecFATAt the start of the facility audit, the auditors were first required to specify the Context

(indoor or outdoor) and Nature (public, residential, or commercial) of the examined recreational facilities, followed by the evaluation of the 10 domains. Each domain is discussed below.

Availability of Sports Facilities. Availability of Sports Facilities examined the presence of sports courts and equipment within the examined recreational facilities. The presence of the following 14 types of sports facilities was examined: tennis courts, table tennis courts, badminton courts, basketball courts, volleyball courts, soccer pitches, squash courts, walking trails, bike lanes, playgrounds for children, weight training gyms, cardio training gyms, sport climbing walls, and swimming pools.

Accessibility. Accessibility examined the degree of ease by which people can access the recreational facility. It was important to identify potential barriers and facilitators such as the presence of slopes, staircases, highways, public transport and parking lots near or within the recreational facilities (see Appendix A for a complete list).

Availability of supportive amenities. This domain examined the presence of complementary resources that support the facility users during their visits. These amenities included free water fountains, rubbish bins, seats (Crawford et al., 2008), bins for pet-droppings, kiosks or shops for snacks or beverages, lockers, air conditioning, changing rooms and toilets.

Conditions of changing rooms. This assessed the hygiene and security as well as the presence of showers and other supportive equipment, including availability of hand dryers or paper towels. Similarly, Conditions of toilets assessed the hygiene, security, availability of flushable sit-toilets (some provide non-flushable squat toilets only), toilet paper, water taps for hand-washing, as well as the dryness of the floors. The auditors also evaluated the presence of any unpleasant smell inside the changing rooms and toilets. Although the conditions of changing rooms and toilets have not been commonly examined in previous studies, the panel of experts believed these measures may contribute to the general impression of the cleanliness and safety of the recreational facilities and hence affect usage.

Management. Management assessed the maintenance practices and operations within the recreational facilities, including items such as the repair of broken equipment, emptying of rubbish bins and daily cleaning of facilities. The availability of clear signage and well-posted regulations was also assessed, along with the presence of helpful and

96

friendly staff (Cavnar et al., 2004) by making standardized enquiries to the staff during the audits. If an indoor sports facility was examined, the indoor temperature was measured using an electronic thermometer.

Policy. Policy assessed if regulations restricted certain behaviors at the facility, including smoking, mobile phone use, dog walking, cycling, and food consumption. In addition, opening hours, the nature of the booking system, and costs of using the facilities were assessed. Although this policy domain was also not commonly included in previous studies, the panel of experts maintains that the regulations imposed on the facility users may influence facility usage.

Environmental safety. This domain assessed safety signage and conditions inside and near the recreational facilities, and included features such as the presence of undesirable gangs, well-fenced sports grounds, broken glass or needles, stray dogs and cats, good lighting within and near the facilities, plus well-maintained equipment, floors and pathways.

Aesthetics. Aesthetics assessed the pleasantness of the environment. The auditors examined if there were trees, flowers, natural sights, sounds of birds, rubbish, animal waste and unpleasant smells inside the examined recreational facilities. In addition, undesirable behaviors such as spitting or smoking visible inside the recreational facilities were also examined.

Social environment. Social environment assessed the presence of physically active or sedentary people of different ages inside the recreational facilities.

From the above 10 domains, a total of 111 audit items were included in the RecFAT using a dichotomous scale (Yes/No answers), except for four items that assessed traffic volumes, the number of public transport options, indoor temperature and the opening hours of recreational facilities, which were ordinal and continuous variables. Appendix A shows a full list of items included in the RecFAT.

Study ProtocolsIn the intra-rater reliability tests, 20 recreational facilities were examined twice by the

same auditor, who was trained research assistants, with one week between each audit. The inter-rater reliability tests were also conducted using twenty recreational facilities, which were examined twice by two independent auditors. The recreational facilities examined in the intra-rater reliability tests included public outdoor parks (n=4), sports grounds (n=1), swimming pools (n=1), playgrounds (n=3), and public (n=6) and commercial (n=5) indoor sports centers. In the inter-rater reliability tests, the recreational facilities examined included public outdoor parks (n=4), sports grounds (n=2), swimming pools (n=1), playgrounds (n=2), recreation grounds (n=2), and public (n=8) and commercial (n=1) indoor sports centers.

Written instructions were given to the auditors in an attempt to standardize the way recreational facilities were assessed. Auditors were trained by the panel of experts in a workshop, in which all items included in the RecFAT were explained and defined. Several field training assessments using the RecFAT were conducted to enhance the understanding of each item. Ethics approval was obtained from the University of Hong Kong Ethics Committee.

Statistical AnalysesKappa statistics and percent agreement were used to examine the intra- and inter-

rater reliability of the individual items that used a dichotomous scale. Percent agreement has been frequently used in reliability studies (Bedimo-Rung et al., 2006; Ijmker et al., 2008; Spanjer, Krol, Brouwer, & Groothoff, 2008), but it has been criticized as it does not consider the agreement by chance, unlike the kappa statistic (Bedimo-Rung et al., 2006). Adopting both percent agreement and kappa statistics provides a more comprehensive analysis than using only one measure, because the kappa statistic is negatively affected by any restricted variability in the ratings.

. Items with percent agreement <60%, 60-70% and >70% were considered ‘‘Poor’’, ‘‘Moderate’’ and ‘‘Good’’, respectively. Items with kappa (k) values <0.4, 0.4- 0.6 and >0.6 were considered ‘‘Poor’’, ‘‘Moderate’’ and ‘‘Good’’, respectively.

97

Kappa values greater than 0.6 were considered ‘‘good’’ agreement, while values within 0.4-0.6 were considered ‘‘moderate’’ agreement, and values less than 0.4 were considered ‘‘poor’’ agreement (Saelens et al., 2006). In contrast, a percent agreement greater than 70% was considered ‘‘good’’ agreement (Bedimo-Rung et al., 2006), values within 60-70% were considered ‘‘moderate’’ agreement and values less than 60% were considered ‘‘poor’’ agreement (Saelens et al., 2006).

Two-way random-model intra-class correlation coefficients (ICC) were computed to examine ordinal and continuous variables including ‘‘indoor temperature,” ‘‘traffic volumes,” ‘‘opening hours,’’ and ‘‘number of public transport stops available near the recreational facilities.’’ ICC values were classified as ‘‘good,’’ ‘‘moderate,’’ and ‘‘poor’’ using the same criteria as the kappa values.

Cronbach’s alphas and inter-item correlations were computed to examine the internal consistency and the extent to which the items included in each domain were inter-correlated. Cronbach’s alpha values over 0.7 (Anderson, Laubscher, & Burns, 1996) and inter-correlations over 0.3 (Zheng & Lin, 1991) were considered indicative that the items measured a common dimension. Statistical software SPSS/PASW 18.0 was used in all analyses.

Results

Intra-Rater Reliability Overall, each domain showed good percent agreement. The intra-rater reliability

measures across domains ranged from 91.0% for Accessibility and Social Environment to 100% for Availability of Sports Facilities, Nature and Context (Table 1). The average percent agreement across all items was 96.9%, indicating overall good intra-rater reliability. All individual items (sub-questions within each domain) also showed good percent agreement, ranging from 75% to 100% (data not shown). The individual item with the lowest percent agreement (75%) was from the Accessibility domain, and assessed the “presence of parking lots inside or near the recreational facilities.”

A total of 81 out of 107 items (76%) showed good kappa values, with 4 and 5 items with moderate and poor values, respectively. The 5 poor items were from the Availability of Supportive Amenities (n=1), Aesthetics (n=1), and Social Environment (n=3) domains. The kappa statistics of 17 items could not be computed due to insufficient response variability and the percent agreement of these items are shown in the left-hand columns of Table 1. Among the 26 items without kappa values (n=17) or with poor (n=5) or moderate kappa values (n=4), all showed good percent agreement, indicating that the lower kappa values obtained for these items may have been due to low response variability. All individual items showed either good percent agreement or good kappa values (Table 1).

The ICCs of the four ordinal and continuous items showed moderate-to-good intra-rater reliability. The item evaluating “traffic volume” showed moderate reliability (ICC=0.52), while the “number of public transport stops available near the recreational facilities” (ICC=0.88), “indoor temperature” (ICC=0.99) and “opening hours” (ICC=1.00) showed good intra-rater reliability.

Inter-Rater ReliabilityOverall, each domain showed good percent agreement. The inter-rater reliability

across domains ranged from 80.0% for Accessibility to 100% for Context and Nature (Table 2). The average inter-rater agreement across all items was 90.6%, indicating overall good inter-rater reliability. Ninety-nine items out of the total 107 (93%) showed good percent agreement, while 7 items and 1 item showed moderate and poor percent agreement, respectively. The only item with poor reliability was found in the Accessibility domain and assessed the ‘‘presence of parking lots inside or near the recreational facilities.’’

. Items with percent agreement <60%, 60-70% and >70% were considered ‘‘Poor’’, ‘‘Moderate’’ and ‘‘Good’’, respectively. Items with kappa (k) values <0.4, 0.4- 0.6 and >0.6 were considered ‘‘Poor’’, ‘‘Moderate’’ and ‘‘Good’’, respectively.

98

20

Table 1. Intra-rater reliability of the Recreational Facility Audit Tool (RecFAT) (n=20).

Items without sufficient response variability

(only % agreement shown, as k could not be

calculated)

Items with sufficient response variability

(data shown for both % agreement and k)

Overall

Domains (No. of items) No. of items

with poor

% agreement

No. of items

with moderate

% agreement

No. of items

with good

% agreement

No. of items

with poor

% agreement (k)

No. of items

with moderate

% agreement (k)

No. of items

with good

% agreement (k)

Average

%

agreement

No. of items with good

% agreement or k

(% of domain total)

Context (Indoor/Outdoor) (1) 0 0 0 0 (0) 0 (0) 1 (1) 100 1 (100)

Nature (Public/Residential/Commercial) (1) 0 0 0 0 (0) 0 (0) 1 (1) 100 1 (100)

Availability of Sports Facilities (14) 0 0 1 0 (0) 0 (0) 14 (13) 100 14 (100)

Accessibility (5) 0 0 0 0 (0) 0 (1) 5 (4) 91.0 5 (100)

Availability of Supportive Amenities (18) 0 0 4 0 (1) 0 (0) 18 (13) 98.9 18 (100)

Conditions of Changing Rooms (12) 0 0 0 0 (0) 0 (0) 12 (12) 97.9 12 (100)

Conditions of Toilets (9) 0 0 0 0 (0) 0 (0) 9 (9) 98.3 9 (100)

Management (6) 0 0 0 0 (0) 0 (1) 6 (5) 94.2 6 (100)

Policy (9) 0 0 2 0 (0) 0 (0) 9 (7) 98.3 9 (100)

Environmental Safety (13) 0 0 5 0 (0) 0 (0) 13 (8) 97.7 13 (100)

Aesthetics (9) 0 0 4 0 (1) 0 (1) 9 (3) 93.3 9 (100)

Social Environment (10) 0 0 1 0 (3) 0 (1) 10 (5) 91.0 10 (100)

Overall (107) a 0 0 17 0 (5) 0 (4) 107 (81) 96.9 107 (100)

a Four ordinal and continuous items were not included in this table. Items with percent agreement <60%, 60-70% and >70% were considered ‘‘Poor’’, ‘‘Moderate’’ and ‘‘Good’’, respectively. Items with kappa (k) values <0.4, 0.4- 0.6 and >0.6 were considered ‘‘Poor’’, ‘‘Moderate’’ and ‘‘Good’’, respectively.

Table 1

Intra-Rater Reliability of the Recreational Facility Audit Tool (RecFAT) (n=20)

99

A total of 55 out of 107 (51%) items showed good kappa values, while 12 items showed moderate values, and 12 items showed poor values. The latter items appeared in the Accessibility (n=1), Availability of Supportive Amenities (n=1), Management (n=1), Environmental Safety (n=3), Aesthetics (n=4), and the Social Environment (n=2) domains. Kappa values of 28 items could not be computed due to low response variability and the percent agreement of these items are shown in the left-hand columns of Table 2. Among the 52 items without kappa values (n=28) and with moderate (n=12) and poor (n=12) kappa values, a total of 7 items and 45 items still showed moderate and good percent agreement, respectively. A total of 101 items out of the total 107 (94%) showed either good percent agreement or good kappa values (Table 2).

The ICCs of the four ordinal and continuous items showed moderate-to-good inter-rater reliability. The items evaluating the ‘‘traffic volume’’ (ICC=0.58) and ‘‘number of public transport stops available near the recreational facilities’’ (ICC=0.47) showed moderate reliability, while ‘‘indoor temperature’’ (ICC=0.99) and ‘‘opening hours’’ (ICC=1.00) showed good inter-rater reliability.

Cronbach’s Alpha and Inter-item CorrelationsThe Cronbach’s alpha values of each domain ranged from 0.45 for Conditions of

Changing Rooms to 0.85 for Management (Table 3). The average inter-item correlations ranged from 0.12 for Availability of Sports Facilities to 0.47 for Management. Acceptable alpha values and high inter-item correlations were found in only two domains, Management and Social Environment, indicating that the items within these two domains assessed a single common dimension. Items representing other domains that were not significantly inter-correlated depict conceptually-unique characteristics of recreational facilities. While the Management and Social Environment domains can be considered as uni-dimensional scales underlined by a common factor, the other 8 domains can be interpreted as multi-dimensional indices in which different aspects of characteristics are assessed. Items within the multi-dimensional indices can be summed and form a score which reflects the level of a conceptually linked characteristic. For example in Accessibility, if there are several types of public transport stops and parking lots available near the recreational facility with few slopes and staircases, the level of accessibility is high.

DiscussionThe current literature suggests that the mere presence of recreational facilities

or resources in the neighborhood may not be sufficient to enhance the overall physical activity levels of residents (Hoehner et al., 2005). The characteristics of recreational facilities may determine facility use (Ries et al., 2008) and the subsequent physical activity accrued there (Cohen et al., 2006). Further evidence suggests that the characteristics of recreational resources may also influence health status (Coen & Ross, 2006), implying that they may have a role in promoting physical activity and health. It is therefore important to develop a reliable instrument that can provide a detailed assessment of various aspects of recreational facilities that are plausibly related to facility use and, hence, physical activity. The audit tool we present here, the RecFAT, assesses ten different domains that may impact the likelihood of using a recreational facility.

In general, the intra-rater reliability of the RecFAT was good and 100% of items showed either good percent agreement or good kappa values. Of the five items showing poor intra-rater reliability when using kappa statistics, most were of a subjective and time-sensitive nature, including items evaluating the presence of unpleasant odors and people. These findings imply that subjective and time-sensitive items may need multiple repeated assessments at different time intervals in order to improve the reliability of the data. Despite the low kappa values obtained for these items, they still showed good percent agreement, suggesting that the low kappa values may also have been partly due to low response variability.

100

Table 2

Inter-Rater Reliability of the Recreational Facility Audit Tool (RecFAT) (n=20)

21

Table 2. Inter-rater reliability of the Recreational Facility Audit Tool (RecFAT) (n=20). Items without sufficient response variability

(only % agreement shown, as no k able to be

calculated)

Items with sufficient response variability

(data shown for both % agreement and k)

Overall

Domains (No. of items) No. of items

with poor

% agreement

No. of items

with moderate

% agreement

No. of items

with good

% agreement

No. of items

with poor

% agreement (k)

No. of items

with moderate

% agreement (k)

No. of items

with good

% agreement (k)

Average

%

agreement

No. of items with good

% agreement or k

(% of domain total)

Context (Indoor/Outdoor) (1) 0 0 0 0 (0) 0 (0) 1 (1) 100 1 (100)

Nature (Public/Residential/Commercial) (1) 0 0 0 0 (0) 0 (0) 1 (1) 100 1 (100)

Availability of Sports Facilities (14) 0 0 1 0 (0) 0 (1) 14 (12) 92.1 14 (100)

Accessibility (5) 0 0 1 1 (1) 0 (1) 4 (2) 80.0 4 (80)

Availability of Supportive Amenities (18) 0 1 7 0 (1) 3 (1) 15 (8) 90.6 16 (89)

Conditions of Changing Rooms (12) 0 0 3 0 (0) 0 (2) 12 (7) 90.4 12 (100)

Conditions of Toilets (9) 0 0 5 0 (0) 0 (1) 9 (3) 94.4 9 (100)

Management (6) 0 0 1 0 (1) 1 (1) 5 (3) 85.8 6 (100)

Policy (9) 0 0 2 0 (0) 0 (1) 9 (6) 93.3 9 (100)

Environmental Safety (13) 0 0 3 0 (3) 0 (2) 13 (5) 91.2 13 (100)

Aesthetics (9) 0 0 3 0 (4) 2 (1) 7 (1) 86.1 7 (78)

Social Environment (10) 0 0 1 0 (2) 1 (1) 9 (6) 92.2 9 (90)

Overall (107) a

0 1 27 1 (12) 7 (12) 99 (55) 90.6 101 (94)

a Four ordinal and continuous items were not included in this table. Items with percent agreement <60%, 60-70% and >70% were considered ‘‘Poor’’, ‘‘Moderate’’ and ‘‘Good’’ respectively. Items with kappa (k) values <0.4, 0.4- 0.6 and >0.6 were considered ‘‘Poor’’, ‘‘Moderate’’ and ‘‘Good’’ respectively.

101

The inter-rater reliability of the RecFAT was also good across the majority of items. The only item showing poor percent agreement was the item evaluating ‘‘presence of parking lots inside or near the recreational facilities,’’ which requires the auditor to judge whether the parking lots are near the recreational facilities or not, if they are located outside the facilities. A more precise definition of “near” (e.g., within 200m of the recreational facility) may be needed to improve the inter-rater reliability of this item. When using kappa statistics, a total of 12 inter-rater items showed poor values, and included items such as: ‘‘presence of regular maintenance practices,’’ ‘‘smoothness of the sports grounds,’’ ‘‘presence of stray animals,’’ ‘‘presence of natural sights,’’ ‘‘presence of rubbish,’’ ‘‘presence of animal waste,’’ ‘‘presence of noise,’’ ‘‘presence of traffic aids,’’ ‘‘presence of children,’’ ‘‘emptiness of the facility,’’ ‘‘sale of sandwiches,’’ and ‘‘presence of parking lots inside or near the recreational facilities.’’ The subjective and time sensitive nature of some of these items (e.g., ‘‘smoothness of the sports grounds’’ ‘‘presence of noise,’’ ‘‘presence of children,’’ and ‘‘emptiness of the facility’’) are likely to have contributed to their relatively low agreement. These subjective and time-sensitive items may need multiple measures at different time intervals and by multiple raters in order to improve their reliability. Despite the low kappa values of these items, a total of 11 out of 12 items showed moderate-to-good percent agreement, indicating that the low kappa values of these items may again have been partly due to the low response variability. Only the item evaluating ‘‘presence of parking lots inside or near the recreational facilities’’ showed poor percent agreement and kappa values simultaneously. However, over 94% of the total 107 items showed either good percent agreement or good kappa values, indicating that different raters reliably assessed the majority of items.

The four ordinal and continuous variables examined using ICCs showed moderate to good reliability. It was encouraging that items evaluating ‘‘indoor temperature’’ and ‘‘opening hours’’ showed good reliability both within and between-raters, and although the item evaluating ‘‘number of public transport stops available near the recreational facilities’’ also showed good reliability within-rater, it only showed moderate reliability between-raters. These findings indicate that, as mentioned earlier, there is a need for improving the definition of “near” by replacing it with a specific distance such as 200m. It was expected that the reliability of the item assessing ‘‘traffic volume’’ would be relatively low due to its time-sensitive nature, but the item was still moderately reliable, indicating that traffic volume surrounding the audited facilities was relatively stable across time.

The low Cronbach’s alpha values and inter-item correlations shown in most domains were due to the audit items evaluating multiple dimensions in each domain, and in some situations this was anticipated. For example, the items included in the domains of

Table 3Cronbach’s Alpha and Average Inter-Item Correlations Across 10 Domains Assessed by the Recreational Facility Audit Tool

Domains Cronbach’s Alpha Inter-items Correlations

Availability of Sports Facilities 0.66 0.12Accessibility 0.65 0.29Availability of Supportive Amenities 0.72 0.18Condition of Changing Rooms 0.45 0.18Condition of Toilets 0.67 0.21Management 0.85 0.47Policy 0.70 0.21Environmental Safety 0.71 0.13Aesthetics 0.58 0.15Social Environment 0.82 0.38

102

Conditions of Changing Rooms and Conditions of Toilets, evaluating cleanliness and safety, were not expected to be inter-correlated. In such cases, the low level of internal consistency indicates that specific domains are to be treated as multiple-dimensional concepts (measured by multidimensional indices), while the domains showing high internal consistency can be considered representing unidimensional concepts underlined by a common factor (and measured by a unidimensional scale). Items within the multi-dimensional indices can be summed and generate a factor score which reflects the level of a conceptually linked characteristic, and it can be used to detect any correlations with facility usage and physical activity. Practitioners can also compare the factor score of a characteristic in a recreational facility under their charge with the score of other facilities before setting up goals and objectives for their management team.

This study has several limitations. These include the small number of repeated measures (two per facility) and the fact that only two auditors were involved in the inter-rater reliability tests. The low response variability of some items was due to several features being commonly available, for example, chairs for resting, rubbish bins, and prohibition of smoking. The low reliability of a few items may also be indicative of insufficiently detailed descriptions of specific features and some improvement to these items should be considered.

The strengths of this study pertain to the extensive evaluation of the recreational facilities across ten domains making the instrument applicable to a wide variety of facilities. The inclusion of the domains such as Policy, Management, Conditions of Changing Rooms, Conditions of Toilets and Social Environment provides a more comprehensive assessment of recreational facilities compared to previous audit tools. Attributes that are typically assessed using rather ‘‘subjective’’ items, such as safety and helpfulness of staff were gauged using items that were designed so to increase the level of objectivity. For example, safety of the changing rooms and toilets was determined by the presence of security devices (e.g., door locks, alarm buttons) or guards near the entrances, while facility staff were considered helpful if they were able to answer standardized questions raised by the auditors. Similarly, traffic volume and indoor temperature were examined using systematic observations and electronic thermometers, respectively, to increase the objectivity. In addition, a wide range of recreational facilities were examined, including the public or commercial indoor sports centers, public parks, swimming pools, sports grounds, recreation grounds, and playgrounds with different ball game courts, supporting the applicability of this audit tool to a wide range of facilities.

Implications for PracticeThe audit tool can be used by recreation practitioners and researchers who aim to

promote facility use and active healthy lifestyles. Using this audit tool, the strengths and weaknesses of a recreation facility may be identified, and the results may assist resource allocation for specific areas. For example, if people are often seen smoking inside or around the recreation facility, increasing staff presence and implementing sanctions may be deemed necessary to maintain a smoke-free environment. Similarly, when amenities are found to be broken or in poor condition, additional resources may be needed to support more frequent maintenance.

This audit tool can also be used on a regular basis in an attempt to identify changes in quality of a specific area. For example, the conditions of toilets and changing rooms can be assessed once per week to detect any improvements or deterioration in conditions. If deterioration occurs, recreation practitioners should identify the reasons and propose improvements, such as increasing the number of cleaning staff. In contrast, if improvements are observed, practitioners should identify the factors, which enhance the quality of services, and include these practices in daily operations. Consequently, this audit tool can help setting goals and objectives for the recreation practitioners who identify the weaknesses of the facilities under their charge.

103

Recreation practitioners and researchers can also use this tool to identify factors determining facility use and subsequent physical activity. This can be done when the characteristics of recreational facilities are assessed and their corresponding statistics on facility use and physical activity level of users are also measured by direct observations or other means. Using this audit tool, characteristics of several recreational facilities in a community can be assessed and centralized in a database (i.e., making use of geographical information systems) in which disparities in terms of quality of recreational facilities between different geographical areas can be identified. This may lead to resource re-distribution by policy makers, which may improve the quality of recreational facilities and hence physical activity in those underserved areas.

While this audit tool focuses on objective measures, it can easily be transformed into a subjective survey due to its simplicity, and used to assess the perceived characteristics of recreational facilities, which can be important determinants of facility use and physical activity. Regular facility users and less-frequent users can be interviewed to assess their perceptions on a particular recreation facility and discrepancies in perceptions can be identified. For example, regular facility users may perceive there to be good environmental safety within the recreational facility, while less-frequent users may have opposite opinions. Recreation practitioners and researchers can then propose particular policies, such as increasing staff presence, to improve the perceptions of users in order to enhance facility use. By continually surveying and auditing the recreation environment, practitioners and researchers can also recognize any changes in perception of users when new practices are undertaken. This audit tool, therefore, can also assess the effectiveness of new measures and policies undertaken in recreational facilities.

ConclusionThe results of this study found that the overall intra- and inter-rater reliability of the

RecFAT audit tool was acceptable, with 100% (intra) and 94% (inter) of items showing good percent agreement (i.e., >70%) or good kappa values (i.e., >0.6). Some subjective and time-sensitive items may need improved descriptions and multiple repeated measures to improve their reliability. In conclusion, the RecFAT is sufficiently reliable to assess indoor sports centres, outdoor parks, sports grounds, recreation grounds, playgrounds and swimming pools in urban areas.

References

Adams, M. A., Sallis, J. F., Kerr, J., Conway, T. L., Saelens, B. E., Frank, L. D., … Cain, K. L. (2011). Neighborhood environment profiles related to physical activity and weight status: a latent profile analysis. Preventive Medicine, 52(5), 326–331.

Anderson, C., Laubscher, S., & Burns, R. (1996). Validation of the Short Form 36 (SF-36) Health Survey Questionnaire among stroke patients. Stroke, 27(10), 1812–1816.

Bedimo-Rung, A. L., Gustat, J., Tompkins, B. J., Rice, J., & Thomson, J. (2006). Development of a direct observation instrument to measure environmental characteristics of parks for physical activity. Journal of Physical Activity and Health, 3(1), S176–189.

Booth, M. L., Owen, N., Bauman, A., Clavisi, O., & Leslie, E. (2000). Social–cognitive and perceived environment influences associated with physical activity in older Australians. Preventive Medicine, 31(1), 15–22.

Cavnar, M. M., Kirtland, K. A., Evans, M. H., Wilson, D. K., Williams, J. E., & Mixon, G. M. (2004). Evaluating the quality of recreation facilities: development of an assessment tool. Journal of Park and Recreation Administration, 22(1), 96–114.

Coen, S. E., & Ross, N. A. (2006). Exploring the material basis for health: characteristics of parks in Montreal neighborhoods with contrasting health outcomes. Health & Place, 12(4), 361–371.

104

Cohen, D. A., Ashwood, J. S., Scott, M. M., Overton, A., Evenson, K. R., Staten, L. K., … Catellier, D. (2006). Public parks and physical activity among adolescent girls. Pediatrics, 118(5), e1381–e1389.

Crawford, D., Timperio, A., Giles-Corti, B., Ball, K., Hume, C., Roberts, R., …Salmon, J. (2008). Do features of public open spaces vary according to neighbourhood socio-economic status? Health & Place, 14(4), 889–893.

Cunningham, G. O., Michael, Y. L., Farquhar, S. A., & Lapidus, J. (2005). Developing a reliable senior walking environmental assessment tool. American Journal of Preventive Medicine, 29(3), 215–217.

Dwyer, T., Ponsonby, A. L., Ukoumunne, O. C., Pezic, A., Venn, A., Dunstan, D., ...Shaw, J. (2011). Association of change in daily step count over five years with insulin sensitivity and adiposity: population based cohort study. BMJ (Clinical research ed.), 342, c7249.

Giles-Corti, B., & Donovan, R. J. (2002). The relative influence of individual, social and physical environment determinants of physical activity. Social Science & Medicine, 54(12), 1793–1812.

Hamilton, C., Swan, V., & Jamal, S. (2010). The effects of exercise and physical activity participation on bone mass and geometry in postmenopausal women: a systematic review of pQCT studies. Osteoporosis International, 21(1), 11–23.

Hoehner, C. M., Brennan Ramirez, L. K., Elliott, M. B., Handy, S. L., & Brownson, R. C. (2005). Perceived and objective environmental measures and physical activity among urban adults. American Journal of Preventive Medicine, 28(2, Supplement 2), 105–116.

Ijmker, S., Mikkers, J., Blatter, B. M., van der Beek, A. J., van Mechelen, W., & Bongers, P. M. (2008). Test–retest reliability and concurrent validity of a web-based questionnaire measuring workstation and individual correlates of work postures during computer work. Applied Ergonomics, 39(6), 685–696.

McCormack, G. R., Rock, M., Toohey, A. M., & Hignell, D. (2010). Characteristics of urban parks associated with park use and physical activity: A review of qualitative research. Health & Place, 16(4), 712–726.

Pate, R. R., Pratt, M., Blair, S. N., Haskell, W. L., Macera, C. A., Bouchard, C.,...Wilmore, J. H. (1995). Physical activity and public health: a recommendation from the centers for disease control and prevention and the american college of sports medicine. JAMA: The Journal of the American Medical Association, 273(5), 402–407.

Ries, A. V., Gittelsohn, J., Voorhees, C. C., Roche, K. M., Clifton, K. J., & Astone, N. M. (2008). The environment and urban adolescents’ use of recreational facilities for physical activity: a qualitative study. American Journal of Health Promotion, 23(1), 43–50.

Saelens, B. E., Frank, L. D., Auffrey, C., Whitaker, R. C., Burdette, H. L., & Colabianchi, N. (2006). Measuring physical environments of parks and playgrounds:EAPRS instrument development and inter-rater reliability Journal of Physical Activity and Health 3(1), S190–S207.

Sallis, J. F., Bowles, H. R., Bauman, A., Ainsworth, B. E., Bull, F. C., Craig, C. L., . . . Bergman, P. (2009). Neighborhood environments and physical activity among adults in 11 countries. American Journal of Preventive Medicine, 36(6), 484–490.

Spanjer, J., Krol, B., Brouwer, S., & Groothoff, J. W. (2008). Inter-rater reliability in disability assessment based on a semi-structured interview report. Disability and Rehabilitation, 30(24), 1885–1890.

Troiano, R. P., Berrigan, D., Dodd, K. W., Masse, L. C., Tilert, T., & McDowell, M. (2008). Physical activity in the United States measured by accelerometer. Medicine and science in sports and exercise, 40(1), 181–188.

Troped, P. J., Cromley, E. K., Fragala, M. S., Melly, S. J., Hasbrouch, H. H., Gortmaker, S. L., & Brownson, R. C. (2006). Development and reliability and validity testing of an audit tool for trail/path characteristics: the Path Environment Audit Tool (PEAT). Journal of Physical Activity and Health, 3, S158–175.

105

Whelton, S. P., Chin, A., Xin, X., & He, J. (2002). Effect of aerobic exercise on blood pressure: a meta-analysis of randomized, controlled trials. Annals of internal medicine, 136(7), 493–503.

Zheng, Y. P., & Lin, K. M. (1991). Comparison of the Chinese Depression Inventory and the Chinese version of the Beck Depression Inventory. Acta Psychiatrica Scandinavica, 84(6), 531–536.

106

Appendix AFull List of Items Included in the Recreational Facility Audit Tool (RecFAT)

Context1. Is the recreational facility indoor or outdoor? Indoor/Outdoor/Both

Nature1. Is the recreational facility public, residential, or commercial? Public/Residential/Commercial

Availability of Sports FacilitiesAre the following sports facilities in the recreational facility?

1. Tennis courts Yes/ No2. Table tennis courts Yes/ No3. Badminton courts Yes/ No4. Basketball courts Yes/ No5. Volleyball courts Yes/ No6. Soccer pitches Yes/ No7. Squash courts Yes/ No8. Walking trails Yes/ No9. Bike lanes Yes/ No10. Playgrounds for children Yes/ No11. Weight training gyms Yes/ No12. Cardio training gyms Yes/ No13. Sport climbing walls Yes/ No14. Swimming pools Yes/ No

Accessibility1. Do you need to climb a staircase to access the facility? Yes/ No2. Do you need to go through a slope to access the facility? Yes/ No3. Do you need to go across a highway to access the facility? Yes/ No4. How many vehicles pass by within 1 minute? (observe at the main entrance) <10, 10-20, 21-30, 31-40, 41-50, >505. How many types of public transport stops are available near the recreational facility? (e.g., MTR, KCR, bus, mini-bus, tram, excluding taxi stops) 0, 1, 2, 3, 4, 5 or above6. Are there parking lots near or inside the recreational facility? Yes/ No7. Is it easy to enter the facility? Yes/ No

Availability of Supportive AmenitiesAre there following amenities in the recreational facility?

1. Chairs for resting Yes/ No2. Benches (for audience) Yes/ No3. Shelters Yes/ No4. Water fountains (for drinking) Yes/ No5. Public toilet Yes/ No6. Changing room Yes/ No

107

Appendix A (cont.)

7. Kiosk Yes/ No8. Sale of sandwich/noodle/rice Yes/ No9. Sale of healthy food (e.g. fresh fruit/salad) Yes/ No10. Sale of bottled water Yes/ No11. Sale of sports drinks Yes/ No12. Sale of fresh fruit juice Yes/ No13. Rubbish bins Yes/ No14. Lockers Yes/ No15. Bins for pets droppings Yes/ No16. Baby-sitting room Yes/ No17. Ventilation system (air-conditioning/ ceiling fans) Yes/ No18. Music/television/radio Yes/ No

Conditions of Changing Room1. Is it clean? Yes/ No2. Are there security devices/services at the entrance? (e.g., presence of security guard at the entrance, use of password to access the entrance) Yes/ No3. Are there showers? Yes/ No4. Are the showers clean? Yes/ No5. Does it provide hot water? Yes/ No6. Is it private (individual)? Yes/ No7. Is soap available? Yes/ No8. Is a mirror available? Yes/ No9. Are paper towels present? Yes/ No10. Is a hand dryer available? Yes/ No11. Is the changing room floor dry? Yes/ No12. Any unpleasant smell? Yes/ No

Conditions of Toilet1. Are there clear labels to distinguish between male and female toilets? Yes/ No2. Is the entrance unlocked? Yes/ No3. Is it clean? Yes/ No4. Are there security devices/services at the entrance? (e.g. presence of security guard at the entrance, use of password to access the entrance) Yes/ No5. Is there a flushable sit-toilet? Yes/ No6. Is toilet paper present? Yes/ No7. Are water taps available? Yes/ No8. Is the floor dry? Yes/ No9. Any unpleasant smell? Yes/ No

Management1. Are there regular maintenance practices in the recreational facility? (e.g. repair of broken equipment, renewal of equipment, daily cleaning of facilities) Yes/ No2. Is there a reception area? Yes/ No3. Is there friendly and helpful staff present? Yes/ No4. (ask the staff two questions: 1. What are the opening hours? 2. Can children use the facility?) Does he/she answer your questions? Yes/ No

108

Appendix A (cont.)

5. Are the regulations of using the facility and behavior posted at the entrance? Yes/ No6. Are facilities clearly signposted? Yes/ No7. Indoor Temperature: ____ degree Celsius/ NA

Policy1. Is booking required to use some/all the facilities? Yes/ No2. Are fees required to use some/all the facilities? Yes/ No3. Is membership required to use some/all the facilities? Yes/ No4. Is using a mobile phone banned? Yes/ No5. Are dogs/animals banned? Yes/ No6. Are bikes banned? Yes/ No7. Is eating banned? Yes/ No8. Is smoking banned in all areas? Yes/ No9. Do users need to wear sports clothes? Yes/ No10. What are the opening hours? ____ hours

Environmental Safety1. Are the floors/paths generally in good condition? (no cracks, no any kind of damage) Yes/ No2. Is there visible damage on equipment (broken fences, benches, sports ground, playground)? Yes/ No3. Are there broken bottles, cans, needles, syringe, drugs and condoms? Yes/ No4. Are there stray dogs/cats/other animals? Yes/ No5. Are there undesirable people or gangs often near the facility? Yes/ No6. Are males often seen with naked chests? Yes/ No7. Are sport courts properly surrounded by fence/walls? Yes/ No8. Is the surface of the sport courts smooth/even? Yes/ No9. Are painted markings on courts clearly visible? Yes/ No10. Is there good street lighting near the recreational facility? Yes/ No11. Is there good lighting within the recreational facility? Yes/ No12. Are there uniformed security guard/ staff within the recreational facility? Yes/ No13. Are there traffic aids such as pedestrian crossing, traffic lights near the recreational facility? Yes/ No

Aesthetics1. Are there trees, flowers, natural sights around or inside the facility? Yes/ No2. Can you hear the sound of birds/nature? Yes/ No3. Is there rubbish on the floor (cans, food, drinks)? Yes/ No4. Is there animal waste/droppings on the floor? Yes/ No5. Is there sufficient shade? Yes/ No6. Is there any unpleasant smell (from rubbish/dogs /cars pollutants, etc)? Yes/ No7. Are people often seen smoking? Yes/ No8. Are people seen spitting? Yes/ No9. Are there noises from traffic/construction sites? Yes/ No

109

Appendix A (cont.)

Social EnvironmentDo you regularly see any of the following at the facility?

1. Children Yes/ No2. Teenagers Yes/ No3. Adults Yes/ No4. Elderly Yes/ No5. Opposite genders (males and females) Yes/ No6. People talking and greeting each other Yes/ No7. People engaging in physical activity Yes/ No8. People sitting Yes/ No9. Is the facility too crowded (too many people)? Yes/ No10. Is the facility too empty (too few people)? Yes/ No