Health Profession Opportunity Grants (HPOG) 2.0 Tribal ... · childcare and transportation assistance; and employment related supportive services such as resume development and interview

Be

Tribal Health Profession Opportunity

Grants (HPOG) 2.0 Evaluation

EVALUATION PLAN

OPRE Report #2017-37 | July 2017

Evaluation of the Tribal HPOG 2.0 Program

NORC | Tribal Health Profession Opportunity Grants (HPOG) 2.0 Evaluation

Evaluation Plan | pg. i

Health Profession Opportunity Grants

(HPOG) 2.0 Tribal Evaluation:

Evaluation Plan

OPRE Report #2017-37 | July 2017

Michael Meit, Carol Hafford, Catharine Fromknecht, Emily Phillips, Noelle Miesfeld, Tori Nadel

NORC at the University of Chicago

Submitted to:

Hilary Forster and Amelia Popham

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

Contract Number:

HHSP233201500052C

Project Director:

Michael Meit, Project Director

NORC at the University of Chicago

4350 East West Highway, Suite 800

Bethesda, MD 20814

(301)-634-9324

This report is in the public domain. Permission to reproduce is not necessary. Suggested citation: Meit, M.,

Hafford, C., Fromknecht, C., Phillips, E., Miesfeld, N., Nadel, T. (2017). Health Profession Opportunity Grants

(HPOG) 2.0 Tribal Evaluation: Evaluation Plan, OPRE Report 2016-37, Washington, DC: Office of Planning,

Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human

Services.

Disclaimer: The views expressed in this publication do not necessarily reflect the views or policies of the

Office of Planning, Research and Evaluation, the Administration for Children and Families, or the U.S.

Department of Health and Human Services.

This report and other reports sponsored by the Office of Planning, Research and Evaluation are available at

http://www.acf.hhs.gov/programs/opre.


Evaluation Plan | pg. ii

We would like to thank the following members of the Tribal HPOG 2.0 Technical Working Group. The views

expressed in this publication do not necessarily reflect the views of these members.

Mark Doescher, MD, MSPH

Stephenson Cancer Center, University of Oklahoma

Rick Haverkate, MPH

Deputy Director, Indian Health Service

Loretta Heuer, PhD, RN, FAAN

School of Nursing, North Dakota State University

Joan LaFrance, Ed.D

Mekinak Consulting

Myra Parker, JD, MPH, PhD

Center for the Study of Health and Risk Behaviors, University of Washington


Evaluation Plan | pg. iii

Introduction

The Health Profession Opportunity Grants (HPOG) Program is administered by the Administration for

Children and Families (ACF), U.S. Department of Health & Human Services. In September 2015, ACF

awarded a second round of HPOG grants to 32 organizations, including five tribal organizations. These

grant awards support demonstration projects that provide eligible individuals with the opportunity to

obtain education and training for occupations in the health care field that pay well and are expected to

either experience labor shortages or be in high demand.

NORC at the University of Chicago is leading a comprehensive implementation and outcome evaluation

of the five Tribal HPOG 2.0 grantees. The evaluation examines program implementation and participant

outcomes at both the individual and systems level in a manner that is grounded in a community-based

participatory research approach. The Tribal HPOG 2.0 evaluation design is descriptive; as such, the

results will not attribute causality between HPOG programs and outcomes.

Purpose

The purpose of this report is to describe the tribal evaluation team’s approach and methods for the

evaluation of the Tribal HPOG 2.0 Program.

Key Findings and Highlights

■ The Tribal HPOG 2.0 evaluation approach is grounded in community-based participatory

research and is guided by the seven values outlined in the Roadmap for Collaborative and

Effective Evaluation in Tribal Communities.

■ The tribal evaluation team worked closely with each grantee over the first year of the evaluation

to learn about their communities and gather input into the evaluation design and data collection

protocols to ensure that the evaluation is culturally responsive and relevant to the Tribe and

community.

■ The Tribal HPOG 2.0 evaluation will assess the structure, process, and outcomes of the grantees’

programs.

■ The tribal evaluation team will use multiple sources of data, including document reviews;

curricula reviews; semi-structured in-person and telephone interviews with grantee and partner

administrative staff, program implementation staff, and local employers; focus groups and

follow-up interviews with program participants; data collected through the HPOG Participant

Accomplishment and Grant Evaluation System (PAGES); and linkage of wage and employment

information through the National Directory of New Hires (NDNH) where possible.

■ Results from the Tribal HPOG 2.0 evaluation will be shared through a variety of reports, briefs,

and presentations.


Evaluation Plan | pg. iv

Methods

The Tribal HPOG 2.0 evaluation team initiated three knowledge development activities to inform the

development of the evaluation design. These activities were: 1) gathering feedback from the grantees

and other stakeholders on the evaluation approach and design; 2) consulting with experts that serve on

the Tribal HPOG 2.0 Technical Work Group (TWG); and 3) conducting a brief review of the literature

related to research in American Indian and Alaska Native (AI/AN) communities. The tribal evaluation

team also referred to the Roadmap for Collaborative and Effective Evaluation in Tribal Communities,

developed by the Child Welfare Research and Evaluation Tribal Workgroup, which guides the approach

to the evaluation.

Glossary

■ AI/AN: American Indian/Alaska Native

■ HPOG: Health Profession Opportunity Grants

■ NDNH: National Directory of New Hires

■ PAGES: HPOG Participant Accomplishment and Grant Evaluation System

■ TWG: Technical Work Group


Evaluation Plan | pg. v

Contents

Introduction ................................................................................................................................................1

Overall Approach for the Tribal HPOG 2.0 Evaluation ........................................................................3

Foundation for the Evaluation .........................................................................................................3

Development of the Evaluation Design ...........................................................................................5

Tribal HPOG TWG .................................................................................................................5

Grantee engagement ...............................................................................................................6

Literature Review. ...................................................................................................................6

Lessons Learned from Tribal HPOG 1.0 Evaluation .......................................................................6

Timeline for the evaluation ..............................................................................................................8

Key Evaluation Questions .........................................................................................................................9

Structure ...........................................................................................................................................9

Process ...........................................................................................................................................10

Outcomes .......................................................................................................................................11

Program Description and Logic Model..................................................................................................12

Conceptual Framework for Tribal HPOG programs .....................................................................12

Tribal HPOG Logic Model ............................................................................................................12

Data Collection Strategies .......................................................................................................................16

Qualitative Data Collection............................................................................................................19

Document Review .................................................................................................................19

Curricula Review ..................................................................................................................20

Annual Site Visits to the Tribal HPOG Grantees .................................................................20

Site Visit Planning ................................................................................................................21

Protocols ...............................................................................................................................22

Semi-structured in-person interviews and focus groups.......................................................22

Grantee and partner administrative staff. ............................................................................23

Program implementation staff. .............................................................................................24

Interviews with employers. ...................................................................................................24

Focus groups with participants. ...........................................................................................24

Interviews with Program Completers. ..................................................................................25


Evaluation Plan | pg. vi

Interviews with Program Non-Completers. ..........................................................................26

Lessons learned regarding follow up interviews. .................................................................27

Quantitative Data Collection..........................................................................................................27

Measures ........................................................................................................................................28

Evaluation Technical Assistance ............................................................................................................30

Data Analysis ............................................................................................................................................31

Qualitative ............................................................................................................................31

Quantitative ..........................................................................................................................32

Data Security and Privacy.......................................................................................................................33

Data Security ..................................................................................................................................33

Physical security. ..................................................................................................................33

Data and network security. ...................................................................................................33

Data storage. ........................................................................................................................33

Data security and privacy training. ......................................................................................33

Respondent Privacy .......................................................................................................................34

Interviewer and focus group facilitator responsibilities regarding privacy.........................34

Collection of Personally Identifiable Information (PII). ......................................................34

Data aggregation. .................................................................................................................34

Human Subjects Protections ..........................................................................................................34

IRB Review of Evaluation. ....................................................................................................34

Informed Consent. .................................................................................................................35

National Institutes of Health (NIH) Human Subjects Protection Training. .........................35

OMB Clearance. ...................................................................................................................35

Reporting Findings ..................................................................................................................................36

Site Visit Reports. ..................................................................................................................36

Practice Briefs. .....................................................................................................................36

Final Report. .........................................................................................................................36

Data Collection Schedule .........................................................................................................................38

References .................................................................................................................................................39


Evaluation Plan | pg. vii

List of Exhibits Exhibit 1. Co-creating Collaborative and Effective Evaluation to Improve Tribal Child Welfare

Programs, from the Roadmap ............................................................................................. 3

Exhibit 2. Evaluation timeline ............................................................................................................. 8

Exhibit 3. Conceptual Framework for Tribal HPOG programs ........................................................ 12

Exhibit 4. Tribal HPOG Logic Model – Linear Version ................................................................... 14

Exhibit 5. Tribal HPOG Logic Model – Medicine Wheel Version ................................................... 15

Exhibit 6. Data Collection Methods to Answer Key Evaluation Questions ...................................... 16

Exhibit 7. Projected Number of Interview and Focus Group Respondents ....................................... 22

Exhibit 8. In-person Interview Topic Areas by Respondent Type .................................................... 23

Exhibit 9. Measures to Answer Research Questions ......................................................................... 28

Exhibit 10. Draft Final Report Outline ................................................................................................ 37

Exhibit 11. Data Collection Timeline .................................................................................................. 38


Evaluation Plan | pg. 1

Introduction The Health Profession Opportunity Grants (HPOG) Program is administered by the Administration for

Children and Families (ACF), U.S. Department of Health & Human Services.1 In 2010, ACF awarded

the first round of HPOG grants (referred to hereafter as HPOG 1.0) to 32 organizations, including five

tribal organizations. ACF awarded the second round of HPOG grants (referred to hereafter as HPOG

2.0) to 32 organizations in September 2015, again including five tribal organizations. HPOG 2.0 grant

awards support demonstration projects that provide Temporary Assistance for Needy Families (TANF)

recipients and other low-income individuals with the opportunity to obtain education and training for

occupations in the health care field that pay well and are expected to either experience labor shortages or

be in high demand.

The HPOG Program focuses on a career pathways model in which participants receive post-secondary

education and training, along with related supportive services, in a series of manageable steps leading to

successively higher credentials and employment opportunities in growing occupations.2 For example,

many HPOG grantees offer training along the nursing career ladder, beginning with Certified Nursing

Assistant (CNA) and moving to Licensed Practical Nurse (LPN) and Registered Nurse (RN).

Supportive services provided by HPOG programs include financial assistance for tuition and textbooks;

academic supportive services such as mentoring and tutoring; social supportive services such as

childcare and transportation assistance; and employment related supportive services such as resume

development and interview preparation.

The authorizing legislation for HPOG calls for a comprehensive evaluation of the demonstration

projects funded under this program. In 2015, ACF awarded Abt and their partners, including NORC at

the University of Chicago, a contract to conduct the National and Tribal Evaluation of HPOG 2.0. Under

this contract, Abt is leading the National Evaluation, which includes impact, outcome, and

implementation studies of the 27 non-tribal grants awarded under HPOG 2.0. NORC is leading the

Tribal Evaluation, which includes a comprehensive implementation and outcome evaluation of the five

Tribal HPOG 2.0 grantees.

This evaluation plan describes the tribal evaluation team’s approach for the evaluation of the Tribal

HPOG 2.0 grantees. The evaluation examines program implementation and participant outcomes at both

the individual and systems level in a manner that is grounded in a community-based participatory

research approach. The tribal evaluation team designed the evaluation in collaboration with the Tribal

HPOG 2.0 grantees and a Tribal HPOG 2.0 Technical Work Group (TWG) comprised of tribal

evaluators and subject matter experts to ensure that the evaluation is both culturally responsive and

scientifically grounded. The evaluation will provide an in-depth assessment of the HPOG 2.0 programs

1 HPOG was established by the Affordable Care Act in 2010 and was extended by the Protecting Access to Medicare Act in 2014. 2 Career Pathways. (n.d.). Career Pathways. Retrieved from http://www.career-pathways.org/about-career-pathways/



administered by tribes, tribal organizations, and tribal colleges to 1) identify and assess how

programmatic health profession training operations are working; 2) determine differences in approaches

being used when programs are serving different sub-populations, including participants with different

characteristics and skill levels; and 3) identify programs and practices that seem to be successful in

supporting the target population to achieve portable industry-recognized certificates or degrees as well

as employment-related outcomes. The Tribal HPOG 2.0 evaluation design is descriptive; as such, the

results will not attribute causality between HPOG programs and outcomes. However, the results of the

evaluation can provide valuable descriptions and documentation of HPOG programs as they are

implemented in the field (ACF OPRE, 2016).

The evaluation plan includes the following: the overall approach for the evaluation; a logic model that

depicts the pathway of key inputs, environmental and contextual factors, program activities, outputs, and

intended short- and long-term outcomes for individuals; key research questions; measures; and data

collection and analysis methods. In addition, the plan describes procedures for data security and privacy

as well as for ensuring human subjects protections. Lastly, it includes a proposed schedule for data

collection, analysis, and reporting of evaluation findings.



Overall Approach for the Tribal HPOG 2.0 Evaluation FOUNDATION FOR THE EVALUATION

The tribal evaluation team’s approach to the Tribal HPOG 2.0 Evaluation is guided by the seven values

outlined in the Roadmap for Collaborative and Effective Evaluation in Tribal Communities, developed

by the Child Welfare Research and Evaluation Tribal Workgroup (Tribal Evaluation Workgroup, 2013).

A visual representation of the Roadmap is included in Exhibit 1. As noted in the Roadmap, the values

provide a pathway for partnering with tribal communities and are grounded in community-based

participatory research. The tribal evaluation team is committed to putting these values into practice and

demonstrating respect for tribal knowledge and sovereignty, ensuring reciprocity in the research process,

and sustaining relationships. The approach used by the tribal evaluation team is presented below, noting

that the values inform multiple aspects and phases of the evaluation process.

Exhibit 1. Co-creating Collaborative and Effective Evaluation to Improve Tribal Child Welfare

Programs, from the Roadmap



To ensure respect of indigenous ways of knowing, the tribal evaluation team will identify who can speak

for the Tribe in approving evaluation and research projects, and that the design of the evaluation is

appropriate for the Tribal context and addresses the research questions. This also means working with

each grantee to inform the research questions and to ensure that they are grounded in local context and

interest and gather information in a culturally respectful and sensitive manner. This will include tribal

review of measures and data collection protocols for cultural responsiveness and ensure that cultural

definitions of success are captured. Respect for indigenous ways of knowing carries through the entire

evaluation cycle, particularly in working with each grantee to interpret findings that reflect both culture

and context, and seeking guidance on appropriate and effective ways to share information.

The tribal evaluation team will demonstrate respect for tribal sovereignty by engaging the tribal grantees

and their leadership in the evaluation design process, including seeking tribal approvals (through

resolutions and institutional review boards (IRBs)) as needed. This may entail resolving differing

perspectives, negotiating data ownership, gaining access to the populations of interest, and obtaining

tribal approval for disseminating evaluation findings through reports and practice briefs. Respect of

tribal sovereignty also means conducting the evaluation in a manner that protects and benefits the tribes

involved, both for individuals and the community, and ensures confidentiality.

The strengths-based focus of the evaluation embodies the intent of the HPOG Program to promote self-

sufficiency and career pathways for TANF recipients and other low-income individuals. This also means

identifying cultural components and protective factors that support engagement, educational persistence,

and employment outcomes, as well as recognizing barriers and constraints that individuals and

communities may face as we examine the implementation and outcomes of Tribal HPOG programs. The

tribal evaluation team will examine strengths and resources within and across the Tribal HPOG

programs.

In conducting the evaluation, the tribal evaluation team will strive for cultural and scientific rigor both

in the conceptual orientation and the methods used. This means the evaluation will include application

of cultural knowledge and methods as well as the scientific method. Combining qualitative and

quantitative methods underscores the importance of oral traditions and expression in tribal communities.

It is important to both tell the story of each grantee’s experience, as well as report the quantifiable

results from the performance measurement system and other administrative data sources.

The tribal evaluation team will actively seek community engagement throughout the evaluation process

by being inclusive and reaching out to tribal partners, their leaders, and community members. Through

ongoing engagement, the tribal evaluation team strives for a mutually beneficial experience for the

Tribal HPOG 2.0 grantees in developing and conducting the evaluation. This approach to tribal

engagement recognizes that building relationships, knowledge, and skills are interdependent priorities

and essential to collaborative research.

The tribal evaluation team’s commitment to ethical research practice will be the foundation of the

evaluation protocol and codified through review by the NORC, Abt, and tribal IRBs, as well as review



and approval by the Office of Management and Budget (OMB). Ethical practice involves the

transparency of the research process, assurance of safeguards for privacy, respect for individual rights

through informed consent and community rights through tribal approval, protection from harm, and

tangible benefits to the community in the form of skills, capacity-developed, or knowledge generated.

Finally, a key goal will be to share knowledge, ensuring that the findings are accurate, grounded in

cultural context, and provide information that is mutually beneficial. The tribal evaluation team is

committed to the value of bidirectional learning between the tribal grantees and the tribal evaluation

team. The tribal evaluation team will seek to learn from each other while building technical skills that

support tribal sovereignty. Additionally, the tribal evaluation team will seek the guidance of the tribal

grantees and their communities about appropriate venues for co-presentation and dissemination of

evaluation findings.

DEVELOPMENT OF THE EVALUATION DESIGN

As noted, our approach to the evaluation will be guided by the values in the Roadmap. In addition to the

values in the Roadmap, the evaluation adapts Donabedian’s conceptual framework of evaluating the

quality of health care, which combines an analysis of the structure, process, and outcomes of programs

(Donabedian, 1966). Critical to this evaluation is a design that focuses on the efforts of projects to

establish process and structural elements that lead to improved outcomes among the communities

served. Thus, the Donabedian conceptual framework will allow for the evaluation of the Tribal HPOG

2.0 initiative as a whole, as well as each individual grantee program in their performance of developing

and implementing culturally competent career pathways programs.

A number of activities and efforts informed the development of the evaluation design. The tribal

evaluation team first developed a memo describing the proposed evaluation design, which ACF

reviewed. Once approved by ACF, the tribal evaluation team shared the evaluation memo with the TWG

as well as the Tribal HPOG 2.0 grantee staff and invited feedback on the evaluation approach and

design. In addition, the tribal evaluation team conducted a brief review of the literature related to

research in American Indian and Alaska Native (AI/AN) communities to inform the evaluation design.

These activities are further described below.

Tribal HPOG TWG. The tribal evaluation team convened a TWG comprised of five experts with

knowledge in a variety of relevant areas, including research and evaluation in AI/AN communities,

health workforce research, rural health, and program development and evaluation. In coordination with

ACF, we identified and confirmed the TWG members in February 2016 and held the first TWG meeting

in April 2016. During the meeting, the tribal evaluation team reviewed the scope of the literature review

and the evaluation design and gathered feedback from TWG members. In addition, the tribal evaluation

team held individual calls with TWG members in May/June 2016 to gather feedback on the protocols

regarding content, topical and cultural relevance, and completeness of the interview and discussion

guides, as well as ways to increase respondent participation and decrease respondent burden. The tribal

evaluation team incorporated the feedback received from TWG members into the evaluation plan and

protocols.



Grantee engagement. Community engagement and respect for indigenous ways of knowing are

important aspects of the evaluation approach and design. As such, the tribal evaluation team has worked

with grantees from the beginning of the evaluation to understand their tribe’s history and culture and to

discover what they hope to learn through the evaluation, as well as preferred strategies and methods. The

tribal evaluation team negotiated Memoranda of Understanding (MOUs) with all of the tribal grantees to

delineate the roles and responsibilities for both the evaluation team and the grantees. In developing the

evaluation design, the tribal evaluation team shared the evaluation design memo and draft protocols with

all of the tribal grantees. The tribal evaluation team consulted with the grantees about culturally

responsive ways to conduct the interviews and focus groups, discussed the protocols to ensure that they

are culturally relevant, and identified any additional questions that would be important to their program

or organization. The tribal evaluation team met with each grantee, either in person or via telephone, to

review the evaluation design and protocols in order to gather their feedback and input. The tribal

evaluation team incorporated the feedback received from grantees into the evaluation plan and protocols.

Literature Review. The evaluation team conducted a brief review of the relevant literature prior to

drafting the evaluation design. The tribal evaluation team developed the scope of the literature review

and search terms and shared these with the TWG members for review. The literature review focused on

five priority areas: research oversight and approvals, evaluation, post-secondary education, capacity

building, and career pathways. The results of the literature review provided important context and

information about appropriate methods for conducting research in AI/AN communities to inform the

evaluation design. The results of the literature review can be found in the report, Informing the Tribal

Health Profession Opportunity Grants (HPOG) 2.0 Evaluation Design: A Brief Review of the

Literature.3

LESSONS LEARNED FROM TRIBAL HPOG 1.0 EVALUATION

The tribal evaluation team applied lessons learned from the Tribal HPOG 1.0 evaluation to the Tribal

HPOG 2.0 evaluation plan. Lessons learned address multiple topics, including engaging with both

grantee staff and stakeholders early in the evaluation design process, broadening the circle of key

informants, documenting staff turnover, enhancing methods for participant outreach and follow-up, and

holding on-site data review sessions.

Building on lessons learned from the Tribal HPOG 1.0 evaluation, the tribal evaluation team engaged

multiple HPOG stakeholders early in the Tribal HPOG 2.0 evaluation design process. At some grantee

sites, this has included stakeholders beyond the immediate grant staff, such as academic training partners

or potential employers. Engaging stakeholders early in the evaluation process will help ensure that the

research questions posed reflect broader consensus in the community and help gain the cooperation of

key informants in Year 2 of the evaluation when annual site visits begin.

3 Informing the Tribal Health Profession Opportunity Grants (HPOG) 2.0 Evaluation Design: A Brief Review of the Literature. (2017, Forthcoming). Administration for Children and Families.



Another important component is to identify which stakeholders and partners refer participants to the

HPOG grantees so that they can be interviewed during the annual site visits. In addition, it will be

important to gain an understanding of whether and how those partners track outcomes for their client

population. During HPOG 1.0, the tribal evaluation team learned that sometimes the referral partners do

not track the outcomes of their clients, or did not have the capacity to do so. Additionally, grants

management and project administration staff changed over the course of HPOG 1.0. Therefore, for

HPOG 2.0, it will be important to track and document staff turnover and approaches that grantees have

taken to ensure continuity and stability in program implementation.

For HPOG 2.0, it will be important to work closely with employment specialists and employers to track

participant employment outcomes and gather qualitative information on employment-related supportive

services and other strategies adopted by grantees to assist participants in obtaining employment (e.g.,

resume preparation, mock interviews). As the grantees began efforts to locate program completers for

follow-up interviews in the final year of HPOG 1.0, the tribal evaluation team found it helpful to have

them adopt some of their successful outreach methods to contact prior participants, such as using the

grant project’s Facebook page to send an announcement. However, the tribal evaluation team found it

could be difficult to conduct follow up interviews via phone and therefore plan to conduct these

interviews in person for HPOG 2.0.

In terms of gaining a better understanding of grant project facilitators and challenges, the tribal

evaluation team will emphasize the importance of speaking with internal institutional partners during

site visits, especially those that control or influence key processes that may help or inhibit educational

attainment (e.g., financial aid office, registrar). The tribal evaluation team will emphasize during

planning calls the need to interview stakeholders in their own place of work to both increase

participation in evaluation activities and to reduce respondent burden. While bringing key informants to

a central location may facilitate scheduling, the tribal evaluation team found that informants are more

likely to participate in the evaluation and more likely to provide open feedback when in their own space

as opposed to a grantee’s.

Finally, given some of the concerns identified with data quality for the HPOG Performance Reporting

System4 in HPOG 1.0, the tribal evaluation team found it helpful to dedicate time during the annual site

visits to facilitate data quality review sessions and to work closely with data coordinators to review data

that would be analyzed for the site visit report. These sessions will continue as part of HPOG 2.0

evaluation activities through grantee technical assistance and evaluation support. These sessions support

the capacity building efforts that are part of the HPOG 2.0 Tribal Evaluation by helping grantees to use

and understand their data.

4 The Performance Reporting System (PRS) is a federal management information system for the HPOG 1.0 program and was designed for both performance management and program evaluation. Data was entered into the PRS by staff at each grantee program.



TIMELINE FOR THE EVALUATION

The evaluation period is six years; the first five years coincide with the grantees’ program

implementation. During the first year of the evaluation, the tribal evaluation team will focus on

engagement with the grantees, the development of the evaluation design and data collection protocols,

and obtaining the proper approvals, including IRB and OMB approval. Years 2-5 of the evaluation will

focus on data collection and sharing preliminary findings from the evaluation. During the final year of

the evaluation, which will occur after the HPOG 2.0 programs end, the tribal evaluation team will

complete data analysis and develop the final report. Exhibit 2 shows the overall activities and timeline

for the evaluation.

Exhibit 2. Evaluation timeline



Key Evaluation Questions The research questions for the Tribal HPOG 2.0 evaluation are based on the questions of interest to the

overall evaluation (National and Tribal) provided by ACF in the solicitation. In addition, some of the

questions are adapted to align more closely with the implementation and outcome focus of the Tribal

HPOG evaluation. As noted, the tribal evaluation team also shared draft research questions with ACF,

the TWG, and grantees, who provided feedback that has been incorporated. The tribal evaluation team

used Donabedian’s framework as a guiding structure for the evaluation approach in order to assess the

structure, process, and outcomes of the grantees’ programs (Donabedian, 1966). The key evaluation

questions are presented below, followed by sub-questions for each area. The Tribal and National

Evaluation will address similar research questions; the core overlapping questions are indicated in bold

and italics.

STRUCTURE

1. In what ways was the program designed or modified for Tribal organizations?

2. To what degree do the HPOG programs conform to the career pathways framework? What are the

pathways?

3. What changes to the service delivery system are associated with program implementation?

■ What is the administrative structure of the Tribal HPOG program?

■ What are the roles of grantee and partner organizations in delivering services? What partners

are involved in the Tribal HPOG program (public workforce investment system; human

service systems; employer and employment agencies; unions; educational institutions; faith-

based and community-initiatives; service providers; etc.)?

■ What is the nature of the labor market in which the HPOG programs operate? What tribal or

regional economic conditions or opportunities affect the design of the Tribal HPOG program?

■ What healthcare occupations are targeted through the Tribal HPOG program? How did the Tribal

grantees decide to offer trainings for these occupations (e.g., based on labor market information

and local demand)?

■ What strategies or instructional model(s) are used to deliver instruction in the targeted

healthcare profession (e.g., accelerated learning models/interventions, technology-enabled

learning)? Which seem to be associated with positive outcomes? What is the evidence base for

the strategy or model? Was the strategy or model adapted? Was the strategy or model adapted in

any way to be culturally relevant?

■ How are potential employers engaged by the lead entity?



■ Are two-generation approaches5 used? How do programs provide educational and skills training

for adults and address family needs?

■ Were there staffing changes made to implement the Tribal HPOG program (e.g., support staff,

faculty)? Did the grantee utilize resources to support distance learning? Were there changes or

improvements made to facilities or equipment used for the training programs? Were there

changes in the curricula?

■ What transfer and articulation policies are in place?

■ What funding sources support the implementation of the Tribal HPOG program? What in-

kind resources are used? Are other institutional grant programs or resources leveraged in

support of HPOG participants?

■ How does the social, economic, geographic, and political context influence program design

and delivery?

■ What are the main lessons learned and recommendations for program improvements?

PROCESS

1. How are health professions training programs being implemented across the grantee sites?

2. What occupational training opportunities are available to HPOG participants? What is the nature

of pre-training, supportive services, job placement and retention services?

3. Which program components do stakeholders believe to be the most effective in improving

outcomes?

■ What groups are targeted for HPOG? What are eligibility criteria and processes? What

outreach and recruitment strategies are used by the Tribal HPOG program? Which seem to be

associated with positive outcomes?

■ What retention strategies are used by the Tribal HPOG program? Which seem to be associated

with positive outcomes?

■ When a student enrolls in the Tribal HPOG program how is educational readiness identified?

How are supportive service needs identified?

■ What academic and non-academic supportive services are offered as part of the HPOG

program (i.e., financial, academic, personal, referrals to public benefits, career transition

support)? What is the duration of these supports?

■ Were the strategies or model(s) for the Tribal HPOG program implemented as intended?

5 Two-generation approaches are designed to address the problems associated with poverty by combining child-focused services intended to improve children’s school readiness and longer term outcomes, and adult-focused services aimed at supporting parents’ well-being, typically with a focus on economic self-sufficiency (St. Pierre, R. G., Layzer, J. I., & Barnes, H. V. (1995). Two-generation programs: Design, cost, and short-term effectiveness. The Future of Children, 5, 76-93). [NOTE: This definition was provided in Exploration of Two-Generation Approaches to Improving Self-Sufficiency solicitation issued by ACF in July 2015, Retrieved from http://www.acf.hhs.gov/programs/opre/two-generation-approaches-to-improving-family-self-sufficiency-2015-2017].

http://www.acf.hhs.gov/programs/opre/two-generation-approaches-to-improving-family-self-sufficiency-2015-2017

http://www.acf.hhs.gov/programs/opre/two-generation-approaches-to-improving-family-self-sufficiency-2015-2017



■ What factors seemed to help with implementation of the Tribal HPOG program? What

challenges were experienced?


OUTCOMES

1. What are the individual-level outputs and outcomes for participants in the Tribal HPOG

programs?

2. Do some programs or program components appear to be associated with positive outputs and

outcomes for tribal populations? If so, what are the hypothesized reasons for differences between

outcomes?

3. Do different program models, strategies, or components appear to lead to different outcomes for

participants?

4. Is there evidence that participation in the program is positively associated with successful

employment and work force capacity building outcomes?

■ What supportive services do Tribal HPOG program participants use? What services are used

most frequently? What proportion of participants receive services?

■ In what training courses/programs do Tribal HPOG program participants enroll? What

proportion complete training and receive a credential? What proportion enroll in a second

training to continue along a career pathway?

■ What proportion of Tribal HPOG program participants become employed? Become employed

in healthcare jobs? Have sustained employment?

■ What workforce capacity building outcomes were achieved (e.g., sustainable curricula and

learning modalities, increased instructional capacity, equipment or facility upgrades, supportive

services, transfer or articulation agreements, community college or workforce organization

partnerships, employer/industry partnerships or alliances, strategic alignment with public

workforce investment system, revenue generation, program growth or replication)?

■ How does Tribal HPOG build on local/regional healthcare capacities? Increase access to

culturally competent services and care?

■ Do different Tribal HPOG models (e.g., tribal college-based versus social service organization-

based) result in different strategies and outcomes?

■ Which components of the Tribal HPOG program will be sustained?




Program Description and Logic Model CONCEPTUAL FRAMEWORK FOR TRIBAL HPOG PROGRAMS

Exhibit 3 presents a high-level view of the service design and delivery aspects of the Tribal HPOG

programs and shows how the certification and training programs are expected to lead to educational

attainment and employment for program participants. A Tribal HPOG grantee organization administers

each program, working in coordination with strategic partners and leveraging financial and personnel

resources. All Tribal HPOG programs provide educational training programs, services to support student

retention, and various forms of employment-related activities to foster job readiness and employment.

As envisioned by ACF and the Tribal HPOG grantees, the training and support services are intended to

result in an educational degree or industry certificate, and employment that will enhance or sustain self-

sufficiency. Contextual factors, including tribal community characteristics, public/private partner

characteristics, program focus and structure, labor market conditions, promotion of protective factors,

and the local employer/industry base, will have an influence on the implementation and outcomes of the

Tribal HPOG programs. Participant outcomes will be influenced by their individual and family

characteristics and their unique histories, experiences, assets, strengths, and risk factors.

Exhibit 3. Conceptual Framework for Tribal HPOG programs

TRIBAL HPOG LOGIC MODEL

The logic model is a critical tool for understanding the relationships between program resources,

activities, and the changes a program intends to achieve (W.K. Kellogg Foundation, 2004). The tribal

evaluation teams present two versions of the Tribal HPOG 2.0 logic model that graphically depict the



pathway of key inputs, environmental and contextual factors, program activities, outputs, and intended

short- and long-term outcomes for individuals and programs. The first version linearly demonstrates

how program inputs translate to outputs which result in short- and long-term outcomes (Exhibit 4). The

second version, informed by the Tribal HPOG 2.0 grantees, resembles a medicine wheel6 to better

reflect traditional world views (Exhibit 5).

The linear version of the logic model (Exhibit 4) has key inputs that translate to grantee activities.

Program activities are listed in a generally sequential order, and range from establishing community

partners and recruiting participants to student retention efforts and tracking student data. Outputs and

short-term and intermediate outcomes are divided to reflect the grantee/program and participant

perspectives. Grantee/program perspectives are shaded in yellow, and participant perspectives are

shaded in red. The long-term grantee and participant outcomes merge together to indicate the cohesive

intended effects of the HPOG program on tribal communities. Key external factors that may influence

the level of change created by the program and assumptions about the logic model are listed at the

bottom of the model.

6 The medicine wheel symbolizes dimensions of health and the cycles of life and is used by various AI/AN tribes. For more information: https://www.nlm.nih.gov/nativevoices/exhibition/healing-ways/medicine-ways/medicine-wheel.html.

https://www.nlm.nih.gov/nativevoices/exhibition/healing-ways/medicine-ways/medicine-wheel.html



Exhibit 4. Tribal HPOG Logic Model – Linear Version

Exhibit 5 shows that the four core Tribal HPOG program components work together in a holistic manner

and are presented in the center of the medicine wheel: recruitment and retention; healthcare trainings;

support services; and employment services. These components are surrounded by key stakeholder

relationships and activities: community partnerships, including academic and training partners;

employer engagement; family and social support; and program staff and trainers. Important contextual

and environmental factors are presented at the bottom of the medicine wheel, including local and

regional job markets; tribal social, political, geographic, and cultural contexts; and family obligations

and responsibilities. At the right, short-term and long-term outcomes for participants and the tribal

grantee flow from the medicine wheel. Outcomes for participants can be understood through domains



related to changes in attitudes and perceptions, education, employment and income, skills and

knowledge, and sustainability. Outcomes for the grantees are listed on the bottom and pertain to

collaboration and capacity building. Within each domain, the list of outcomes begins with the short-term

outcomes and progresses to the long-term outcomes.

Exhibit 5. Tribal HPOG Logic Model – Medicine Wheel Version



Data Collection Strategies The tribal evaluation team will use multiple sources of data for the process and outcome evaluation:

document reviews; curricula reviews; semi-structured in-person and telephone interviews with grantee

and partner administrative staff, program implementation staff, and local employers; focus groups and

follow-up interviews with program participants, including program completers and non-completers;

program operations data collected through the HPOG Participant Accomplishment and Grantee

Evaluation System (PAGES); and when available, wage and employment information through the

National Directory of New Hires (NDNH).

Through the use of these data collection techniques, the tribal evaluation team will be able to answer the

research questions posed previously. Exhibit 6 lists the research questions and sub-questions and

identifies the data sources that the Tribal Evaluation will use to answer each of the research questions.

Exhibit 6. Data Collection Methods to Answer Key Evaluation Questions

Research Questions

Methods

Qualitative Quantitative

Do

cu

me

nt

rev

iew

Cu

rric

ulu

m r

evie

w

Inte

rvie

ws

wit

h

gra

nte

e

ad

min

istr

ati

ve s

taff

an

d p

art

ne

rs

Inte

rvie

ws

wit

h

gra

nte

e s

ite

pro

gra

m

sta

ff

Inte

rvie

ws

wit

h lo

cal

em

plo

ye

rs

Fo

cu

s g

rou

ps

an

d

inte

rvie

ws

wit

h

pro

gra

m p

art

icip

an

ts

PA

GE

S d

ata

Structure


2. To what degree do the HPOG programs conform to the career pathways framework? What are the

pathways?


What is the administrative structure of the Tribal HPOG program?

What are the roles of grantee and partner organizations in delivering services? What partners are involved in the Tribal HPOG program (public workforce investment system; human service systems; employer and employment agencies; unions; educational institutions; faith-based and community-initiatives; service providers; etc.)?

What is the nature of the labor market in which the HPOG programs operate? What tribal or regional economic conditions or opportunities affect the design of the Tribal HPOG program?

What healthcare occupations are targeted through the Tribal HPOG program? How did the Tribal grantees decide to offer trainings for these occupations (e.g., based on labor market information and local demand)?



Research Questions

Methods


Do

cu

me

nt

rev

iew

Cu

rric

ulu

m r

evie

w

Inte

rvie

ws

wit

h

gra

nte

e

ad

min

istr

ati

ve s

taff

an

d p

art

ne

rs

Inte

rvie

ws

wit

h

gra

nte

e s

ite

pro

gra

m

sta

ff

Inte

rvie

ws

wit

h lo

cal

em

plo

ye

rs

Fo

cu

s g

rou

ps

an

d

inte

rvie

ws

wit

h

pro

gra

m p

art

icip

an

ts

PA

GE

S d

ata

What strategies or instructional model(s) are used to deliver instruction in the targeted healthcare profession (e.g., accelerated learning models/interventions, technology-enabled learning)? Which seem to be associated with positive outcomes?

What is the evidence base for the strategy or model?

Was the strategy or model adapted?

Was the strategy or model adapted in any way to be culturally relevant?

How are potential employers engaged by the lead entity?

Are two-generation approaches used? How do programs provide educational and skills training for adults and address family needs?

Were there staffing changes made to implement the Tribal HPOG program (e.g., support staff, faculty)? Did the grantee utilize resources to support distance learning? Were there changes or improvement made to facilities or equipment used for the training programs?

Were there changes in the curricula?

What transfer and articulation policies are in place?

What funding sources support the implementation of the Tribal HPOG program?

What in-kind resources are used?

Are other institutional grant programs or resources leveraged in support of HPOG participants?

How does the social, economic, geographic, and political context influence program design and delivery?

What are the main lessons learned and recommendations for program improvements?

Process


2. What occupational training opportunities are available to HPOG participants? What is the nature of pre-

training, supportive services, job placement and retention services?

3. What key components or approaches seem to be successful or contribute to the success of these

programs?

What groups are targeted for HPOG? What are eligibility criteria and processes? What outreach and recruitment strategies are used by the Tribal HPOG program? Which seem to be associated with positive outcomes?



Research Questions

Methods


Do

cu

me

nt

rev

iew

Cu

rric

ulu

m r

evie

w

Inte

rvie

ws

wit

h

gra

nte

e

ad

min

istr

ati

ve s

taff

an

d p

art

ne

rs

Inte

rvie

ws

wit

h

gra

nte

e s

ite

pro

gra

m

sta

ff

Inte

rvie

ws

wit

h lo

cal

em

plo

ye

rs

Fo

cu

s g

rou

ps

an

d

inte

rvie

ws

wit

h

pro

gra

m p

art

icip

an

ts

PA

GE

S d

ata

What retention strategies are used by the Tribal HPOG program? Which seem to be associated with positive outcomes?

When a student enrolls in the Tribal HPOG program how is educational readiness identified? How are supportive service needs identified?

What academic and non-academic supportive services are offered as part of the HPOG program (i.e., financial, academic, personal, referrals to public benefits, career transition support)? What is the duration of these supports?

Were the strategies or model(s) for the Tribal HPOG program implemented as intended?

What factors helped with implementation of the Tribal HPOG program? What challenges were experienced?


Outcome

1. What are the individual-level outputs and outcomes for participants in the Tribal HPOG programs?

2. Do some programs or program components appear to be associated with positive outputs and outcomes for

tribal populations? If so, what are the hypothesized reasons for differences between outcomes?

3. Do different program models, strategies, or components appear to lead to different outcomes for

participants?

4. Is there evidence that participation in the program is positively associated with successful employment and

work force capacity building outcomes?

What supportive services do Tribal HPOG program participants use? What services are used most frequently? What proportion of participants receive services?

In what training courses/programs do Tribal HPOG program participants enroll? What proportion complete training and receive a credential? What proportion enroll in a second training to continue along a career pathway?

What proportion of Tribal HPOG program participants become employed? Become employed in healthcare jobs? Have sustained employment?



Research Questions

Methods


Do

cu

me

nt

rev

iew

Cu

rric

ulu

m r

evie

w

Inte

rvie

ws

wit

h

gra

nte

e

ad

min

istr

ati

ve s

taff

an

d p

art

ne

rs

Inte

rvie

ws

wit

h

gra

nte

e s

ite

pro

gra

m

sta

ff

Inte

rvie

ws

wit

h lo

cal

em

plo

ye

rs

Fo

cu

s g

rou

ps

an

d

inte

rvie

ws

wit

h

pro

gra

m p

art

icip

an

ts

PA

GE

S d

ata

What workforce capacity building outcomes were achieved (e.g., sustainable curricula and learning modalities, increased instructional capacity, equipment or facility upgrades, supportive services, transfer or articulation agreements, community college or workforce organization partnerships, employer/industry partnerships or alliances, strategic alignment with public workforce investment system, revenue generation, program growth or replication)?

How does Tribal HPOG on build local/regional healthcare capacities? Increasing access to culturally competent services and care?

Do different Tribal HPOG models (e.g., tribal college-based versus social service organization-based) result in different strategies and outcomes?

Which components of the Tribal HPOG program will be sustained?


QUALITATIVE DATA COLLECTION

As noted above, the tribal evaluation team will use a variety of qualitative data collection methods to

answer the research questions. The approach for conducting each of these data collection strategies is

described below.

Document Review

Document review will occur throughout the duration of the evaluation. The purpose of the review is to:

1. Provide information on the structures and processes the grantees have established to implement and

administer the elements of their programs;

2. Determine levels of commonality across programs in regard to their structure, processes, and

performance indicators;

3. Provide details to customize communication with each grantee, including the tailoring of evaluation

instruments per grantee (e.g., interview and focus group protocols) as appropriate; and

4. Inform questions to address as the program evolves and the document changes over time.

The tribal evaluation team began the document review process with a review of the grantee applications

in November 2015. The tribal evaluation team developed an abstraction template to collect standard

information across each Tribal HPOG grantee (see Appendix A). Data abstracted from the grantee



applications included, but was not limited to, program description, program goals and objectives,

partnering organizations, anticipated outcomes, population served, staffing plans and needs, support

services offered, and program processes (e.g., curriculum development, implementation approach, and

strategies for assessing learning). During the initial engagement calls with grantees, the tribal evaluation

team reviewed this information with the grantees and gave the grantees an opportunity to provide

corrections or updates. Most grantees made some changes to their program since the grants were

awarded. The tribal evaluation team will review program materials to capture the continuity and any

changes to the HPOG programs on an annual basis. The tribal evaluation team will continue to

document review activities, including the following types of documents:

■ Program materials, such as recruitment and retention materials, tribal newsletters, websites,

brochures, and other media;

■ Administrative materials, such as grant applications, organizational charts, and partnership

agreements; and

■ Organizational materials, such as strategic plans and policy and procedural handbooks.

Curricula Review

The tribal evaluation team will conduct a review of the curricula of training programs offered by each of

the grantees and the career pathways for these programs. The purpose of the review is to:

1. Document requirements for training and credentialing;

2. Determine ways in which the curricula were modified for the target population; and

3. Identify the career pathways offered as part of each grantee’s HPOG program.

The tribal evaluation team will request a copy of program curricula, including materials, instructor

guides, learner materials, etc., from each grantee or partner organization prior to the initial site visits and

subsequent site visits, as appropriate. The tribal evaluation team developed a draft curriculum

abstraction template, which will be used to collect standard information about each program offered

(Appendix B). This process will identify learning objectives, strategies and modalities for

implementation of career pathway models to understand how HPOG programs are tailored to each

grantee’s goals and circumstances.

Annual Site Visits to the Tribal HPOG Grantees

Annual site visits with the Tribal HPOG grantees will be a key component of data collection. The tribal

evaluation team will conduct annual in-person site visits at each grantee in Years 2 through 5 of the

evaluation. A two-person team has been designated to work closely with and learn from each of the five

Tribal HPOG grantees over time. Each team is comprised of a senior and junior researcher with the goal

of building rapport and strong relationships with the grantees. To date, each team has conducted an

initial engagement call with grantees in late 2015, met face-to-face with grantees at the HPOG Grantee



Meeting in February 20167, and communicated via email regarding evaluation activities. In addition,

each team has met individually with the grantees to clarify information in their grant application and

gather input on the draft evaluation design and protocols via teleconference or an initial grantee visit (no

data collection occurred during initial visits). The tribal evaluation team determined which grantees to

visit in Year 1 based on our existing relationships with grantees, the extent to which the grantee changed

their program from HPOG 1.0 to HPOG 2.0, and whether the grantee had new program partners. Two-

person teams conducted visits to the two new grantees to establish relationships and two returning

grantees as a new team will be working with them throughout the course of the evaluation. These visits

provided an opportunity to build rapport among grantees and the tribal evaluation team as the same two-

person team will conduct the evaluation site visits. The remaining grantee was part of the HPOG 1.0

evaluation, and given the existing relationship the tribal evaluation team has with the grantee staff, it

was feasible to clarify program information contained in their application and solicit feedback on the

evaluation design and protocols via phone.

Data collection will take place primarily during annual site visits with each Tribal grantee. The tribal

evaluation team anticipates that each site visit will be approximately three to five days in length,

depending on the number of partners and their geographic locations.

Site Visit Planning

Prior to conducting site visits, the tribal evaluation team will conduct a planning call with each grantee

to determine the best approach for recruiting respondents for the interviews and focus groups. Each

grantee will be asked to provide a comprehensive list of grantee and partner administrative staff, staff

responsible for implementing the program, local employers, and students, and to discuss the most

appropriate method for recruiting each individual. Some grantees may prefer to reach out to their

partners, staff, and students to facilitate the recruitment. If so, the tribal evaluation team will provide the

recruitment materials as described to facilitate outreach activities. If not, the tribal evaluation team will

ask the grantees for contact information to reach out using recruitment letters and follow-up phone calls.

Internally, the tribal evaluation team will ensure that all members understand the goals and objectives of

the site visits. Prior to data collection, the tribal evaluation team will hold a training session for the team

to review the data collection protocols that will be used for interviews and focus groups; the informed

consent procedures; the approach for post-site visit data cleaning and analysis; and data security

procedures. The training will also address cultural competency. In addition to general site visit training,

each two-person team will review grantee-specific information obtained through the document review,

curricula review, and PAGES prior to conducting the site visit.

7 On February 17-18, 2016, ACF held the HPOG Annual Grantee Meeting. Topics included grant requirements; the National and Tribal evaluation of HPOG, including key components, learning to date, and PAGES tools; and program implementation, including recruitment, adult education, partnerships, and employer engagement.



Protocols

The tribal evaluation team has developed draft semi-structured interview protocols for the collection of

qualitative data during the initial evaluation site visits and follow-up site visits. These instruments are

described below; see Appendix C for the protocols. All protocols will begin with a brief introductory

script that summarizes the overall evaluation, the focus of each interview, how respondent privacy will

be protected, and how data will be aggregated. The tribal evaluation team will obtain verbal informed

consent from interview respondents and written informed consent prior to conducting participant focus

groups. The senior member of the evaluation team will be responsible for seeking consent from

participants. See Appendix D for the verbal informed consent forms for interviews and written informed

consent forms for focus groups. All data collection instruments and informed consent forms developed

for the interviews and focus groups have been reviewed by the NORC and Abt IRB as well as any

grantee IRBs that require approval. In addition, OMB approval will be obtained prior to beginning data

collection.

Semi-structured in-person interviews and focus groups

During the site visits, the tribal evaluation team will meet with an array of stakeholders to better

understand the structure, process, and outcomes of the Tribal HPOG programs. In-person interviews will

help uncover the experience of those involved in planning, implementing, and participating in the

programs. The tribal evaluation team will collect data from the interviews and focus groups via

extensive detailed notes. Given the respondent population, and in some cases, the nature of the questions

being asked, the tribal evaluation team will not record the interviews or focus groups. Detailed notes

were sufficient for data collection during Tribal HPOG 1.0. Exhibit 7 provides an overview of the

respondent types, the length of the interview or focus group, and the anticipated number of respondents.

Exhibit 7. Projected Number of Interview and Focus Group Respondents

Respondent type Length of interview

or focus group Expected Interviewees

Per Grantee Number of Interviews Annually

Across All Grantees

Grantee and partner administrative staff

60 minutes 3-7 staff members 15-35 staff members

Program implementation staff 90 minutes 3-10 staff members 15-50 staff members

Employers 45 minutes 3-6 employers 15-30 employers; may vary based on number located and identified

Focus groups with participants 90 minutes 1-3 focus groups, 5-9 participants per group

25-135 participants

Program completers 60 minutes 20 participants 100 participants

Program non-completers 60 minutes 10 participants 50 participants

Exhibit 8 provides an overview of the stakeholders the tribal evaluation team anticipates interviewing

during the site visits and the main topics to be discussed.



Exhibit 8. In-person Interview Topic Areas by Respondent Type

Evaluation Topic

Grantee and Partner Administrative Staff Implementation Staff Employers

Program Participants

Initial Visit

Follow-up Visits Initial Visit

Follow-up Visits

Initial and Follow-up

Visits

Initial and Follow-up

Visits

Program Structure

Background

Planning

Program Type

Administrative Structure

Partnerships

Qualification of Implementation Staff

Program Design and Curriculum

Skills and Competencies for Local Health Care Workforce

Contextual Factors

Program Processes

Supportive Services

Recruitment

Orientation

Use of Program Data

Implementation Facilitators and Challenges

Quality of Instruction

Program Outcomes

Participant Educational Attainment

Participant Employment Outcomes

Role of Tribal HPOG program in building Tribal health workforce capacity

Satisfaction with Tribal HPOG program

Grantee and partner administrative staff. The tribal evaluation team will conduct in-person

interviews on an annual basis to gain insight from grantee and partner administrative staff on high-level

program strategies, program development, and lessons learned. Partners may include public and private

health care employers, education and training organizations, community-based organizations, labor

organizations, and national, state or local foundations that provide assistance to AI/ANs. The tribal

evaluation team will obtain verbal informed consent from each participant. Initial interviews will likely

focus on the overall strategic approach of the program as well as processes used to develop the program

curricula. These interviews will provide important information about the structure of the program and

any evidence that informed the structure of the program and special considerations and modifications for

Tribal populations. Interviews in subsequent years will likely focus on program modifications, overall



challenges and successes, as well as lessons learned which, at a high level, will inform analyses about

the success of the program as a whole. See Appendix C for the initial and follow-up site visit protocols.

Program implementation staff. The tribal evaluation team will conduct in-person interviews with staff

responsible for coordinating and implementing the program at each site. These individuals may include,

but are not limited to, program instructors, recruitment and orientation staff, and providers of supportive

services. Depending on the structure of the program, we may interview multiple staff at each

implementation site. The tribal evaluation team will obtain verbal informed consent from each

participant. These interviews will focus on program processes including recruitment, orientation, and

program implementation. Interviews in subsequent years will likely focus on program modifications,

overall challenges and successes in implementation, as well as lessons learned which, at a high level,

will inform analyses about the success of the program as a whole. See Appendix C for the initial site

visit and follow-up site visit protocols.

Interviews with employers. The tribal evaluation team will work with grantee sites to identify potential

employers in the region where HPOG participants will seek employment and conduct interviews with

appropriate staff at these facilities. This respondent group will not include employers that have formal

partnerships with the HPOG grantee and are part of program implementation; those employers would be

considered grantee and partner administrative staff. These interviews will be used to assess employers’

general impressions of program graduates, their degree of awareness of the program as a whole, and

their views on the extent to which programs are making an impact on the creation of a workforce that is

equipped to meet the demands of the current health care needs of AI/AN communities. The tribal

evaluation team will obtain verbal informed consent from each participant. The content of the interviews

will remain consistent for each annual site visit. See Appendix C for draft interview protocol.

Focus groups with participants. The tribal evaluation team will conduct focus groups with program

participants during the annual site visits. The focus groups will gather program participants’ perceptions

around the following key evaluation topics: program design and curriculum; supportive services;

recruitment and orientation; quality of instruction; participant educational attainment; and satisfaction

with the HPOG program.

To prepare for the focus groups, the tribal evaluation team will discuss with each grantee the best and

most culturally appropriate recruitment techniques as well as whether or not they would prefer

evaluation team staff contact potential participants or if they would prefer to contact participants

themselves. If the grantee provides a list of student participants, the tribal evaluation team will reach out

to the potential participants using recruitment letters and follow up phone calls if necessary. Should the

grantee prefer to have potential participants contacted by program staff, the tribal evaluation team will

provide recruitment materials to facilitate outreach activities. One potential challenge with grantee staff

recruiting focus group participants is that it may result in a disproportionate number of HPOG

participants who are more engaged with the program. However, during the Tribal HPOG 1.0 evaluation,

this recruitment method provided increased focus group participation and a diverse range of

perspectives.



Conducting the focus groups will require considerable communication and logistical planning.

Incentives for their participation will be a non-cash honorarium, such as gift cards, valued at $50

(amount pending OMB approval). The tribal evaluation team will work with each grantee to identify the

most appropriate incentive for their students (e.g., gas cards; retail gift cards; vouchers for the college

bookstore) to ensure maximum participation and responsiveness to community preferences. The tribal

evaluation team will also work with each site regarding the best time of day to conduct the focus groups

and address any special logistical arrangements that are needed to facilitate participation (e.g., child-

care; transportation; meals; refreshments). In addition, the tribal evaluation team will further consult

with each grantee about how best to conduct the focus group in a culturally competent manner that

stimulates discussion and full participation (e.g., allowing opportunities for self-reflection and privacy in

composing responses to questions).

The tribal evaluation team will obtain written informed consent for each participant. Once consent is

obtained from all participants, the facilitator will introduce participants to the overall purpose and

structure of the gathering. In addition, the facilitator will re-emphasize to the assembled group that their

comments will be aggregated in the site visit summaries and reports and not directly attributable to

them. The tribal evaluation team will collect data from the focus groups via extensive detailed notes,

with one member of the tribal evaluation team facilitating the discussion and the other team member

taking notes. First names of focus group participants will be collected upon registration and will be used

for note-taking purposes. To launch the discussion, the tribal evaluation team will prepare a set of

opening “ice-breaker” questions designed to stimulate conversation and spur an open and informal give-

and-take among the group assembled. The tribal evaluation team anticipates that the focus groups will

last no longer than 90 minutes. See Appendix C for the draft Focus Group Guide.

The tribal evaluation team anticipates some potential challenges for conducting the focus groups. Four

of the five Tribal HPOG 1.0 grantees were tribal colleges and the HPOG participants completed training

programs at the grantee institution. Therefore, the tribal evaluation team could host focus groups at the

tribal college, which was a comfortable and convenient location for students. For HPOG 2.0, only two of

the five grantees are tribal colleges. The remaining three grantees will be supporting students at various

training institutions and may or may not have cohorts of students at these institutions. Therefore, focus

group recruitment and coordination may be more difficult. The tribal evaluation team will work with the

grantees to determine how best to bring students together. For grantees supporting students at multiple

training locations in larger cities, it may be possible to identify a central location to host the focus group.

Similar to HPOG 1.0, the tribal evaluation team will also travel to partner sites that are located further

away and conduct focus groups at those sites, thereby minimizing participant burden.

Interviews with Program Completers. In addition to using PAGES and NDNH data, the tribal

evaluation team will conduct in-person interviews with participants who have completed a training

program in order to obtain information on key program outcomes related to educational attainment and

employment. The purpose of these interviews is to assess the current employment status of the

participants who have completed training programs and to capture their voice and perspectives on



whether the program adequately prepared them for employment in the health sector and, if applicable, to

serve tribal communities.

The tribal evaluation team anticipates that some participants in the annual focus groups will also

participate in the interviews; others may have had no previous contact with the tribal evaluation team.

The tribal evaluation team will work with the grantees to recruit participants during the annual site visit

planning period and conduct interviews in-person during the site visit.

For interviews with program completers, the tribal evaluation team lead will seek verbal informed

consent. All IRBs reviewing this study have approved this approach. Respondents will receive a non-

cash honorarium valued at $50 for an in-person interview (amount pending OMB approval). The tribal

evaluation team will consult with each grantee to determine the most appropriate method of payment

(e.g., gas cards; retail gift card) to provide to the participant.

Topics that will be addressed during the interview include program design and curriculum; recruitment

and orientation; supportive services; quality of instruction; participant educational attainment;

participant employment readiness; participant employment outcomes; and satisfaction with Tribal

HPOG program. The tribal evaluation team will collect data from the interviews via extensive detailed

notes. See Appendix C for the draft protocol for the interviews with program completers.

Interviews with Program Non-Completers. In order to understand factors that led to non-completion,

the tribal evaluation team will conduct in-person interviews with participants who started but did not

complete a training program through HPOG. The tribal evaluation team will work with the grantees to

recruit participants during the annual site visit planning period and conduct interviews in-person during

the site visit.

For interviews with non-completers, the evaluation lead will seek verbal informed consent. All IRBs

reviewing this study have approved this approach. Each non-completer participating in an interview will

receive a non-cash honorarium valued at $50 for an in-person interview (amount pending OMB

approval). The tribal evaluation team will consult with each grantee to determine the most appropriate

method of payment (e.g., gas cards; retail gift card) to provide to the participant.

Topics that will be addressed during the interview include reasons for leaving the program; challenges

experienced; elements of the program that interviewees felt were effective or non-effective;

identification of any short-term outcomes resulting from program participation; how the program could

be improved; and whether the non-completer plans to re-enroll in the program or pursue an alternative

course of study. In addition, implementation of this interview will provide the grantees with important

formative information to develop a profile of early leavers to address participant attrition and to develop

strategies that will foster retention or re-engage non-completers. The tribal evaluation team will collect

data from the interviews via extensive detailed notes. See Appendix C for the draft protocol for the non-

completer interview.



Lessons learned regarding follow up interviews. During the HPOG 1.0 evaluation, recruiting

program completers and non-completers for follow up interviews was challenging and resource

intensive despite offering an incentive for participants. The evaluation team received a list of completers

and non-completers from grantee staff and recruited participants via phone to schedule a phone

interview. However, it was difficult to contact participants for a number of reasons, including temporary

phone numbers on pay-as-you-go phone plans, scheduling conflicts, and no-show interviewees. Based

on TWG and grantee recommendations, for HPOG 2.0 the tribal evaluation team proposes recruiting

completers and non-completers to participate in in-person interviews during the annual site visit. In

addition, grantee staff will be asked to provide assistance with recruitment of participants for interviews

as grantee staff have stronger relationships and are in more frequent contact with the participants,

particularly program completers. Another potential recruitment strategy could include reaching out to

program participants via Facebook; one grantee facilitated the use of this outreach method in HPOG 1.0.

Similar to the focus group participants, one potential challenge with grantee staff recruiting focus group

participants is that it may result in disproportionate involvement from HPOG participants that are more

engaged with the program and more likely to have positive experiences. Non-completers may continue

to be challenging to recruit given that they are no longer involved with the HPOG program. One

additional strategy for addressing this challenge is to offer a higher incentive for non-completers. In

HPOG 1.0, the study team offered completers $25, while non-completers were offered $10. As part of

the OMB package, the tribal evaluation team proposed $50 for an in-person interview for both

completers and non-completers for HPOG 2.0.

QUANTITATIVE DATA COLLECTION

Abt led the development, implementation, and maintenance of the Management Information System

(MIS) for HPOG 2.0, referred to as PAGES. All HPOG 2.0 grantees will use PAGES, a web-based

system, to collect program operations data. Managed and secured by Abt, the PAGES system will

support both performance management and evaluation efforts across all tribal and non-Tribal HPOG

grantees. The PAGES team has granted the tribal evaluation team access to PAGES. The tribal

evaluation team is able to view non-identifiable disaggregated data and aggregate data but is not able to

see any personally identifiable information. Tribal grantee staff will administer informed consent and

collect participant data at intake and throughout a participant’s enrollment. Data that will be collected

through PAGES includes participant demographics at intake; educational enrollment and completion;

number and type of supportive services received; employment status at intake and after training; and

changes to participant wages.

Grantee staff will collect data on an ongoing basis. The tribal evaluation team will coordinate with the

PAGES support team to provide technical assistance to Tribal HPOG 2.0 grantees around the use of the

system. Study participants must provide written consent for the study team to report data collected in

PAGES. Further, data quality, completeness, and consistency are concerns when using program data for

evaluation purposes. The tribal evaluation team will work with grantees to ensure they have received

guidance on the use of PAGES and any questions or concerns are addressed. Additionally, the tribal

evaluation team will monitor collection of consent forms and review the PAGES data with grantees



during the annual site visits. This will help to ensure the tribal evaluation team has consent to use the

PAGES data, that the data in PAGES are reported consistently, and that the data reflect the grantees’

knowledge of the number of students who enrolled, completed, and are employed.

When possible, the evaluation team will work with Abt to use the information in PAGES to link to the

NDNH to provide longer-term data on participant earnings and employment. The team will use this data

to assess if participants have continued and advanced in the health professions, a key outcome of the

career pathways model.

MEASURES

The tribal evaluation team plans to use a core set of measures for evaluating the Tribal HPOG 2.0

programs. The measures will describe key implementation and outcomes of each HPOG grantee, and are

designed to help answer the key research questions. Exhibit 9 aligns the measures with the research

questions that they will address and the data sources.

Exhibit 9. Measures to Answer Research Questions

Key Research Questions Measures Data Sources

Structure


2. To what degree do the HPOG programs conform to the career pathways framework? What are the pathways?


■ HPOG grantee organizational model (e.g., type of lead organization and partnership structure)

■ Number and type of

● Partnerships

● Curricula developed and implemented

● Capacity building efforts (e.g., staff hires, equipment or facilities upgraded, resources leveraged)

● Transfer and articulation agreements

■ New or strengthened relationships with partners (through MOUs or informally)

■ Grantee Applications

■ Interviews with grantee and partner administrative staff

■ Interviews with program implementation staff

Process


2. What occupational training opportunities are available to HPOG participants? What is the nature of pre-training, supportive services, job placement and retention services?

3. What key components or approaches seem to be successful or contribute to the success of these programs?

■ Types of program processes implemented for recruitment, orientation, retention, etc.


● supportive services provided

● instructional modes implemented

■ Key program facilitators and challenges

■ Quality of instruction

■ Grantee Applications

■ Interviews with grantee and partner administrative staff

■ Interviews with program implementation staff



Key Research Questions Measures Data Sources

Outcomes

1. What are the individual-level outputs and outcomes for participants in the Tribal HPOG programs?

■ Participant demographics (e.g., Race/ethnicity, gender, age, education level, marital status, children, receipt of public benefits, employment status at intake)


● participants recruited for Tribal HPOG programs

● participants enrolled in Tribal HPOG programs

● participants retained in Tribal HPOG programs

● participants who completed a training program

● participants who received a professional, state, or industry certification or license

● participants who received an occupational certificate or diploma

● participants that re-enroll in a career pathway training program

● participants that no longer received public benefits (obtained through interviews and internal case management records for each program, if available)

● participants employed

● participants employed in a healthcare occupation

■ Amount of post-program wages earned by program completers (that complete one or multiple career pathways programs) (6 months, 12 months, 24 months, etc.)

■ Length of time employed for program completers that complete one or multiple career pathways trainings

■ Number and type of participants receiving supportive services and types of supportive services

■ Ability of training programs to address workforce shortages/demand (through document reviews and interviews with grantees and employers)

■ Increased partnerships

■ Increased capacity to deliver workforce development programs (as applicable to grantee)

● Staffing, new hires, and faculty (# of staff hired)

● Facilities and infrastructure developed (#of investments)

● Curricula developed (type of # of curricula developed)

● Transferability and articulation agreements (for credit and credentials) (# of agreements developed)

● Supports and services developed and provided (type and # services)

■ Amount of leveraged resources (type and $ value)

■ Integration of Tribal HPOG program with local or regional healthcare work force

■ Sustainment of Tribal HPOG program or sustainment of certain elements or components of Tribal HPOG program

■ Program satisfaction

■ Interviews and Focus Groups with Program Participants

■ PAGES data

■ NDNH data



Evaluation Technical Assistance The Tribal HPOG 2.0 evaluation presents an opportunity for bidirectional learning, wherein the Tribal

HPOG 2.0 grantees and the evaluation team learn from one another throughout the process. The tribal

evaluation team will work with the grantees to increase understanding of evaluation activities (including

ensuring consistent collection and reporting of PAGES data), increase program buy-in, and increase

knowledge of how findings can be used to promote local program performance improvement. The tribal

evaluation team will provide technical assistance (TA) to grantees to support the Tribal HPOG 2.0

evaluation and to build tribal research and evaluation capacities by building skills and providing

mentorship in multiple scientific methods. Through the evaluation, the tribal evaluation team will learn

about each grantee’s local programs, culture, and community, which informs the process and protocols for

conducting evaluation activities in each tribal community. Bidirectional learning provides the opportunity

to learn from the grantees and understand that every tribal community has its own culture, norms, and

preferred methods for evaluation and TA.

The types of TA provided will vary both in format and content. At the beginning of the evaluation, the

tribal evaluation team will focus TA efforts on explaining the evaluation approach, including the

evaluation framework. Potential TA or capacity-building needs will arise as the tribal evaluation team

engages the grantees in the design of the evaluation, and capacity building efforts will be responsive to

grantee requests. The tribal evaluation team will identify grantee needs through site visits, phone calls with

grantees, and PAGES support requests. As there may be some staff turnover over the five-year evaluation

period, the tribal evaluation team will explore with each grantee the need for guidance or documentation

on the evaluation plan methods to ensure continuity with implementation of the evaluation design.

Another key area for TA will be around the qualitative and quantitative data collection needed for the

evaluation, as well as grantee use and understanding of the data. TA will be provided on the PAGES

system both at the beginning and throughout the evaluation as needed. This TA will ensure the grantees

are able to consistently input data as well as understand, interpret, and use the data. The tribal evaluation

team will also conduct training on obtaining and recording informed consent prior to implementation of

the evaluation.

When the tribal evaluation team requests information from grantees, such as information for the

document or curricula review, detailed guidance for each grantee will be provided. The tribal evaluation

team will work with grantees to identify the easiest methods for them to share documentation and

modify the guidance as needed. The guidance will include information about what types of documents

are requested as well as the format and instructions for sharing the documents with the evaluation team.

This approach will ensure grantees have a thorough understanding of the requests and minimize burden

on the grantee staff.

Throughout all phases of the evaluation, the tribal evaluation team will maintain an open line of

communication with grantees and be available to answer questions about any component of the evaluation.



Data Analysis The tribal evaluation team will conduct analyses of qualitative and quantitative data to be included in

evaluation reports and briefs. As detailed in previous sections, data from grantees will come in three

different forms. First, the tribal evaluation team will systematically review the grantees’ documents and

program curricula to record information about the way they intend to structure and implement their

programs. Second, the tribal evaluation team will collect qualitative data that reflect the perspectives of

key stakeholders, including administrators, local partners, implementation staff, employers, and

participants. Third, the tribal evaluation team will analyze the data collected in PAGES, both the

grantee-level data about the program and individual-level data about the participants. The team will use

the following qualitative and quantitative methods in the analysis:

Qualitative

The tribal evaluation team will use a systematic approach to analyze the data obtained through the

document and curriculum review and interviews and focus groups conducted during and following

annual site visits. The tribal evaluation team will use NVivo software to store and analyze the large

volume of data that will be collected over the course of the evaluation. NVivo allows users to develop a

set of thematic codes organized around the research questions, described below. Once themes have been

identified, the tribal evaluation team will pilot test these themes by applying them to a small portion of

the data to assess relevance and refine as needed.

All data will be coded by two evaluation team members. The first coder will be the analyst from their

respective grantee, and the second will be a Tribal HPOG 2.0 evaluation team member not part of the

two-person grantee team. Having multiple evaluation team members code the same data will help

validate the codebook and increase the rigor and objectivity of code application. All tribal evaluation

team members coding the data across grantee sites will have regular meetings to discuss the codes and

ensure consistency in coding. The following key qualitative analysis approaches will guide our analyses

(Patton, 1987; 2002; Strauss and Corbin, 1988; Ratcliff, 1994; Spradley, 1979):

■ Identification of Key Themes. The coding scheme will be initially organized around evaluation

topic areas derived from the evaluation questions. The coding scheme will be applied to all data,

and new codes will be created and applied throughout the coding process to capture

unanticipated themes that emerge. The tribal evaluation team will identify emergent themes on

two levels: those themes which seem to dominate a particular grantee site; and those themes

which relate to the overall evaluation questions across all grantee sites.

■ Expansion and Description of Key Themes. Next, the tribal evaluation team will determine if

results from a particular interview or focus group are consistent with the emergent key themes

and will add to these key themes if appropriate, noting any exceptions. By the end of this phase,

the evaluation team will have developed a rich description of each theme as it pertains to the

scope of the evaluation.



■ Thematic Analysis. The evaluation team will explore relationships between themes across

respondents, across sites, and across time by using matrices in order to understand a holistic

perspective of the grantees’ and participants’ experience as it relates to the evaluation questions.

Additionally, the team will go beyond identifying key themes to describe how the social,

political, geographic, and cultural settings of grantees may have played a role in the key themes

that emerged from the evaluation.

In addition to the analytic approach describe above, the evaluation team can utilize the text search and

query functions in NVivo to examine the data and search for themes across different data sources. For

example, NVivo has a word frequency query that can be used to find the most common words used to

answer a particular question. Further, it is possible to group codes together in a model, which can help

the tribal evaluation team visualize the data in different ways. Like in the pilot phase, as new patterns

and questions arise the tribal evaluation team will refine our codes to guide additional analyses in

NVivo.

Quantitative

The PAGES team has granted the tribal evaluation team access to the PAGES system so that the team

can routinely assess and report descriptive statistics and performance indicators for each Tribal grantee

and across all Tribal grantees. The PAGES team is in the process of developing a series of standard, pre-

programmed reports that will provide basic information about the grantee programs (e.g., enrollment,

completion, employment). The aggregate reports generated from PAGES will enable the tribal

evaluation team to assess grantees’ progress towards meeting their goals. In addition, the tribal

evaluation team will export de-identified raw data from PAGES and conduct descriptive analyses to

describe and document program successes and challenges. The tribal evaluation team will continue to

evolve our quantitative data analysis plan as we receive additional training on the PAGES system.

Finally, when possible, the tribal evaluation team will work with Abt to obtain employment and earnings

data from NDNH. The NDNH is a national database of wage and employment information, including

files on new hires, quarterly wages, and unemployment insurance. Collection of social security numbers

in PAGES will enable the evaluation team to link to NDNH data. Four out of the five Tribal HPOG 2.0

grantees are collecting social security numbers for some or all participants. When possible, the

evaluation team will use the information in PAGES to link to NDNH to provide longer-term data on

participant earnings and employment. The team will use this data to assess if participants have continued

and advanced in the health professions.



Data Security and Privacy The tribal evaluation team will follow all data security policies and protocols to protect study

participants from injury or harm or breach of privacy, as detailed below. This includes all data collected

from the interviews and focus groups as well as participant data collected through PAGES. NORC will

store data from the interviews and focus groups. Abt led the development, implementation, and

maintenance of the Management Information System (MIS) for HPOG 2.0, referred to as the PAGES.

All HPOG 2.0 grantees will use PAGES, a web-based system, to collect program operations data. The

following section outlines: 1) key data security structures and processes in place at NORC and Abt, 2)

proposed processes for maintaining respondent privacy, and 3) human subject protection processes that

will be sought for the evaluation.

DATA SECURITY

Both NORC and Abt have policies in place to ensure that data security is maintained and only

authorized individuals are able to access private data. The measures in place to protect data security are

described below; additional information about data security is documented in the Data Security Plan

prepared for ACF.

Physical security. Keycard access restricts access to all Abt and NORC facilities. Only members of the

NORC tribal evaluation team will have access to the specific project data and materials; individually

identifiable data are separated from data collected for all evaluation activities. Physical copies of the

documents will be stored in locked file cabinets accessible only to authorized project staff. At the

conclusion of the project, data are disposed of in accordance with the project-specific requirements.

Data and network security. All NORC authorized network users are issued an encrypted, challenge-

response user-id and password for signing into project applications and data areas on the network.

Employees requiring remote access to the NORC network must connect through NORC’s firewall using

both Virtual Private Network (VPN) technology and an encrypted, challenge-response technology. As

required, the tribal evaluation team must report a suspected or confirmed breach to the ACF Contracting

Officer’s Representative (COR) immediately (within one hour of discovery). Intensive monitoring will

be done to ensure compliance with privacy protocols. Abt has similar measures in place to ensure data

security.

Data storage. Electronic files of interview notes will be stored on NORC’s secure network. Only

members of the tribal evaluation team will have access to these files. Any hard copies of interview or

focus group notes will be stored in a locked file cabinet. Any paper documents that contain sensitive

data will be shredded three years after the final contract payment. At the end of the evaluation, the tribal

evaluation team will follow NORC’s data destruction procedures; data will be stored until September

2021.

Data security and privacy training. All NORC employees are required to take a series of trainings

related to security awareness and data use and privacy annually. These trainings ensure all employees



are aware of NORC’s policies and procedures to protect sensitive information from unauthorized access,

alteration, and misuse.

RESPONDENT PRIVACY

The tribal evaluation team places a high priority on respondent privacy. All members of the tribal

evaluation team have extensive experience handling sensitive data and all NORC staff are required to

sign a statement of confidentiality as a condition of their employment.

Interviewer and focus group facilitator responsibilities regarding privacy. All NORC staff on the

tribal evaluation team are trained on NORC’s data security and privacy policies. Interview and focus

group participants are informed before data collection that their information will be kept private. This is

reiterated in the informed consent form.

Collection of Personally Identifiable Information (PII). PII, including names and contact information,

will be collected for each group of respondents solely for recruitment purposes. The tribal evaluation

team will preserve the privacy of any persons and entities identified in the interviews and focus groups.

Unique case numbers will be assigned to respondents; respondent identification information will kept in

a separate file. This identifiable information will not be reported to ACF or the public.

PII, including name and date of birth, will be collected in PAGES in order to track participants over

time. Four of the five tribal grantees will also be collecting Social Security Numbers of participants. The

collection of PII will allow the grantee staff and evaluation team to follow participants over time,

measure key outcomes, and potentially link participants to other data sources, including NDNH. Grantee

staff will gather and input participant information into PAGES. The PAGES Support team provides

training to all grantee staff to ensure security procedures are followed. The tribal evaluation team will be

able to access PAGES, but will not be able to view any PII.

Data aggregation. The tribal evaluation team will not reference individuals when reporting findings in

order to protect the identity and privacy of respondents. Data will be reported at the grantee level,

outlining findings across all respondents, as well as findings by each respondent group.

HUMAN SUBJECTS PROTECTIONS

IRB Review of Evaluation. All data collection and privacy procedures for qualitative data collection

will be approved by the NORC IRB, a Federally Approved IRB. The IRB at Abt will also review the

protocol. The initial IRB protocol was submitted to NORC’s IRB in March 2016. After coordination

with the Abt IRB, a revised version of the study protocol was submitted and approved in July 2016. In

addition, the study protocol was submitted to Research Review Boards and IRBs associated with three

grantees: Turtle Mountain Community College (TMCC), Cankdeska Cikana Community College

(CCCC), and Great Plains Tribal Chairman’s Health Board (GPTCHB). For TMCC, a submission was

prepared for the Tribal Nations Research Group (TNRG) Research Review Board, which reviews any

research conducted on the Turtle Mountain Band of Chippewa lands. At CCCC, IRB approval was



needed at one of the partner institutions, Untied Tribes Technical College. For GPTCHB, the regional

Indian Health Service IRB reviewed the study protocol.

Informed Consent. Informed consent will be obtained from all respondents. The informed consent

procedures will address the following: the purpose of the study; who is conducting it; risks and benefits

of participation; how the data will be used; that participation is voluntary; that respondents’ privacy will

be protected; and that the information collected will be reported without identifying any individual

interviewee or focus group participant. Verbal consent will be obtained for in-person interviews

conducted with program staff and stakeholders, as well as interviews conducted with program

completers and non-completers. The tribal evaluation team will obtain written informed consent for

participants in the focus groups; a copy of the written consent form will be provided to the participant

and the original consent form will be stored at NORC. Grantee staff will administer the informed

consent process for use of participant data stored in PAGES. Grantee staff will provide a copy of the

informed consent form to the participant, maintain one copy in a locked cabinet at their institution, and

ship the original consent forms to NORC. NORC will store all written consent forms from the

evaluation in a locked cabinet until three years after the final contract payment, at which point the

consent forms will be shredded.

National Institutes of Health (NIH) Human Subjects Protection Training. All members of the tribal

evaluation team have completed the NIH Office of Extramural Research online tutorial on Protecting

Human Research Participants. This course includes key historical events and current issues that impact

guidelines and legislation on human participant protection in evaluation; ethical principles and

guidelines that should assist in resolving the ethical issues inherent in the conduct of evaluation with

human participants; the use of key ethical principles and federal regulations to protect human

participants at various stages in the evaluation process; a description of guidelines for the protection of

special populations in evaluation; a definition of informed consent and components necessary for valid

consent; a description of the role of the IRB in the evaluation process; and the roles, responsibilities and

interactions of federal agencies, institutions and researchers in conducting evaluation with human

participants.

OMB Clearance. In coordination with Abt, the tribal evaluation team will obtain OMB approval for the

data collection instruments to be used for the implementation and outcome evaluation in 2016. The tribal

evaluation team anticipates that OMB approval will be obtained in January 2017.



Reporting Findings The tribal evaluation team will develop a variety of reports and briefs to disseminate findings from the

evaluation, each with a distinct purpose and audience. The three types of reports to be developed include

annual site visit reports, practice briefs, and the final report. Each of the three report types are described

below.

Site Visit Reports. Following each annual site visit, the two-person team will develop a site visit report

that details the findings from the interviews and focus groups conducted. The report will include a

summary of the grantee program and findings organized by structures, process, and outcomes. The tribal

evaluation team will share the site visit reports with the Tribal HPOG grantees and ACF following each

site visit; we will ask the Tribal HPOG grantees to review the information in the report for accuracy. If

the grantees have concerns with information presented in the site visit report, the evaluation team will

add footnotes to provide additional context related to the grantee’s perspective. Site visit reports will not

be shared publically. The purpose of the site visit reports is to provide ACF with an interim update of

preliminary findings and to provide the tribal grantees with a summary of findings, to advance the

collaborative process of the evaluation, and to share knowledge. Preliminary findings may be reported in

other formats, such as the practice briefs, and this data will be included in the final report.

Practice Briefs. Over the six years of the evaluation, the tribal evaluation team will develop seven

special reports/briefs related to the evaluation that will be published on the ACF website and shared

widely with stakeholders and in accordance with the dissemination plan developed as part of the

evaluation. The tribal evaluation team will propose topics for the practice briefs and will coordinate

topics with the national evaluation team as appropriate. The tribal evaluation team will also seek input

from grantees on potential topics and appropriate methods and venues for dissemination of findings.

ACF will approve all practice brief topics prior to development. Two practice briefs will be developed

during the first year of the evaluation prior to data collection. The first practice brief will be related to

findings from Tribal HPOG 1.0 evaluation and an overview of the Tribal HPOG 2.0 evaluation. The

second practice brief will be a discussion around conducting respectful and historically informed

evaluations with tribal populations. Additional practice brief topics could potentially include HPOG 2.0

program components, site-specific findings and accomplishments, or topics related to HPOG, such as

career pathways or workforce development.

Final Report. The tribal evaluation team will develop a final report that reflects the aggregated analysis

of all qualitative and quantitative data collected throughout the evaluation. The final report will

summarize findings across grantees and include grantee-specific examples. A draft outline of the final

report is included in Exhibit 10.



Exhibit 10. Draft Final Report Outline

Overview/Executive Summary Brief overview and key findings

Overview of Tribal HPOG 2.0 Program and Evaluation

Overview of the HPOG 2.0 Program

Overview of Tribal HPOG 2.0 grantees and programs

Description of HPOG 2.0 evaluation and approach to working with Tribal HPOG 2.0 grantees

Evaluation Methodology Research questions

Data collection and analysis methods

Study limitations

Key Findings Key findings organized by Structure, Process, Outcomes

Implementation facilitators and challenges

Conclusion and Discussion Overall outcomes from HPOG 2.0

Lessons learned

Sustainability

As stated in the MOUs executed with each of the tribal grantees, the evaluation team will share grantee-

specific evaluation findings with the grantees and their larger communities throughout the evaluation. In

addition to reviewing the site visit reports, the tribal grantees will have the opportunity to review and

comment on reports and practice briefs that will released publicly.



Data Collection Schedule Data collection will occur in Years 2-5 of the HPOG grants, beginning in the spring of 2017. Prior to

starting data collection, the tribal evaluation team will prepare a submission for OMB approval. The

OMB submission will include two components of the overall HPOG evaluation—the Tribal Evaluation

and the National Evaluation implementation study. The OMB package will be prepared in close

coordination with Abt, who is leading the National Evaluation implementation study.

The tribal evaluation team anticipates that OMB approval will be obtained by January 2017. The tribal

evaluation team will make any changes to the data collection protocol stemming from OMB review and

amendments will be submitted to NORC’s IRB, the Abt IRB, and the Tribal IRBs at that time.

The tribal evaluation team anticipates that the annual site visits will begin in March 2017. The tribal

evaluation team will plan site visits in coordination with the Tribal grantees to ensure that they are

productive and do not create a burden. The evaluation team will work with the grantees to ensure the site

visits do not conflict with other events or deadlines or occur during times (e.g., spring break) when

students are not available for focus groups and interviews.

Data analysis will occur in the months following the annual site visits. The tribal evaluation team will

code all site visit data within three weeks of the site visit and submit the site visit report to ACF within

six weeks of the site visit.

Exhibit 11. Data Collection Timeline

HPOG 2.0 Tribal Evaluation Task Timeline

OMB Submission Fall 2016

Annual site visits to tribal grantees and development of site visit reports Annually Spring 2017, 2018, 2019, 2020

Conduct data analysis 2017-2021

Develop Practice Briefs Annually September 2017-September 2021

Develop Final Report September 2021



References Office of Planning, Research & Evaluation, Administration of Children and Families, U.S. Department

of Health and Human Services (ACF OPRE). (2016). “The Administration of Children & Families

Common Framework for Research and Evaluation.” Retrieved from:

http://www.acf.hhs.gov/sites/default/files/opre/acf_common_framework_for_research_and_evaluation_

v02_a.pdf

Donabedian, A. (1966). "Evaluating the Quality of Medical Care." Milbank Memorial Fund Quarterly

44 (1): 166-203.

Patton, M. Q. (1980). Qualitative evaluation methods. Beverly Hills, CA: Sage.

Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.).Thousand Oaks, CA: Sage.

Ratcliff, D. (1994). Analytic Induction as a Qualitative Research Method of Analysis.

Spradley, J. (1979) The Ethnographic Interview. United States, Wadsworth Group/Thomas Learning

Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and

techniques. Newbury Park, CA: Sage.

Tribal Evaluation Workgroup. (2013). “A Roadmap for Collaborative and Effective Evaluation in Tribal

Communities.” Children’s Bureau, Administration for Children and Families, U.S. Department of

Health and Human Services.

W.K. Kellogg Foundation. (2004). “Logic Model Development Guide.” Retrieved from:

https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-

development-guide

Documents

Health Profession Opportunity Grants (HPOG) 2.0 Tribal ... · childcare and transportation assistance; and employment related supportive services such as resume development and interview