Part 2: Evaluating Setting-specific Programs

Population health intervention research can involve systematic evaluation. The following cases demonstrate unique methods and strategies for the evaluation of programs working to improve health in school-based settings.

• Case 4 - Healthy Transitions: promoting resilience and mental health in young adolescents
• Case 5 - H2K - The Heart Healthy Kids Program

Community-led Evaluation and Intersectoral Partnership

The Healthy Transitions project involves collaboration between health and education professionals to develop, implement, monitor and evaluate the impact of a comprehensive mental health promotion school-based intervention. Moving beyond academic arenas, mental health professionals, community members, teachers, school board administrators, social workers and public health nurses took joint ownership and responsibility to develop and test a population-based intervention that could be adopted, implemented and evaluated in other communities across Canada.

Case 4 - Healthy Transitions: promoting resilience and mental health in young adolescents

Corrine Langill, RN, B.ScN. Children's Hospital of Eastern Ontario, Ottawa, ON
Ann Croll, Ph.D, C.Psych, Child and Youth Health Network for Eastern Ontario, Ottawa, ON

Corresponding author: Corrine Langill, clangill@cheo.on.ca

Introduction

Adolescence is a time of intense developmental change as well as social and educational transition. It is also a time when mental health problems can initially surface. However, mental health promotion and mental health care services are often insufficient, and long waits for services are common. Approximately 14 percent of Canadian youth have a diagnosable mental health problem but fewer than one in four youth with a mental health problem get appropriate treatment (Waddell, McEwan, Shepherd, Offord & Hua, 2005). Funding for mental health promotion often does not reflect this need.

There are concerns about inequitable resource allocation, limited services, and the stigma associated with mental illness, all of which present barriers to treatment. We believed that a universal, comprehensive school program could address these inequities by identifying youth in need of services and linking them to appropriate supports, while at the same time promoting positive mental health and reducing the stigma associated with mental illness among all students.

Developing, testing and evaluating such a program helped determine what was effective, desirable and acceptable in a school-based intervention. The program represents a very practical example of what works in mental health promotion. We expand beyond academic arenas by sharing our program and results with broad audiences who are in a position to implement the program, namely: schools, community health centres and youth service providers. Further, a shared-use copyright and easy access to the program and evaluation tools enables other communities to adopt, implement and evaluate this intervention.

Project description and lessons learned

Healthy Transitions was developed in partnership with education and health professionals. It includes a screening and referral component as well as educational components for youth, teachers and parents. Some members of the evaluation team were also responsible for program development and implementation. Pilot testing and program evaluation was completed in the spring of 2006.

The youth curriculum included five interactive sessions:

• How thoughts, feelings and actions influence each other
• 'Rebounding' from difficult events
• Communication
• Handling stress and feelings
• Getting help when needed

Young adult facilitators were recruited from school social work departments and community health centres and received training to provide classroom sessions to 208 students in groups of 12-15 students at a time. Facilitators used screening protocols to identify and follow-up with youth in need of support or referral.

The youth curriculum is grounded in theories of cognitive behavioural approaches, positive psychology (optimism) and educational theories of social-emotional learning. The program evaluation was designed to assess changes to students' emotional state, coping and mental health promotion skills. This evaluation also considered the impact on teachers and parents, lessons learned and recommendations for future programming.

Students, parents and teachers completed surveys at various points during the program. In addition, focus groups and interviews with key informants (school principals, facilitators and advisory group members) provided qualitative data on program satisfaction, perceived benefit and recommendations for future programming.

Discussing mental health concepts with young adolescents requires careful attention to ethics. Parental consent was required for students to take part. The consent outlined our guiding principles of confidentiality along with information indicating the types of situations in which information from students would be shared with parents and schools. This was especially significant as we asked specific questions in the screening questionnaire about suicidal thoughts and behaviours. We developed detailed protocols for following up with students whom we were concerned about, ensured all facilitators had received ASIST (Applied Suicide Intervention Skills Training) and established a system of professional back up for facilitators.

Program facilitators identified and referred at-risk students to formal support services, while other students identified through the screening process received more informal monitoring and support. Most youth identified through our screening process had already come to the attention of school staff.

Throughout the program, most youth participants reported high frequencies of feeling happy, and low frequencies of negative feelings. Still, significant numbers of youth felt stressed, worried, angry or sad fairly often. After the program, a substantial proportion of students reported improvements in specific health promoting skills. Youth also reported lower levels of stress and worry, and better knowledge of supportive community resources. Students found the program useful, relevant and enjoyable.

The majority of parents and teachers who attended the workshops reported improved knowledge of healthy adolescent development, community resources, signs that youth may be struggling and how to support positive mental health in teens. Two months after the teacher in-service, most teachers had applied what they had learned.

Lessons learned

The Healthy Transitions project yielded two principal lessons. First, meaningful collaboration between cross-sectoral partners ensured everyone was strongly invested in the program's success. Our advisory group members, composed of mental health professionals from the Children's Hospital of Eastern Ontario (CHEO) and the community, teachers, school board administrators, social workers and public health nurses, guided program development and supported its implementation. The advisory group grew out of a working group of the cross-sectoral Child and Youth Health Network for Eastern Ontario, which is supported by CHEO and led by one of the program planners. Members had volunteered to take part in a collaborative working group to develop a mental health promotion program for young adolescents. Because of the positive working relationships that program planners had developed over the years with other community partners, it was not difficult to recruit additional members.

Taking joint ownership and responsibility for the program fostered stronger relationships between partners and facilitated the acceptance of programs into individual schools. Project leaders acknowledged and capitalized on the expertise of group members by having the advisory group develop follow-up protocols, determine program content and provide feedback to all curriculum drafts. All feedback was considered seriously and incorporated into the program.

The second lesson from Healthy Transitions was that support and cooperation from school staff are critical. The willingness of school principals to accommodate the program was essential to success. Fortunately, the curriculum sold itself to principals, who were quickly convinced of its potential to meet a number of student needs. They welcomed the positive approach to promoting mental health, and endorsed the learning objectives, teaching strategies and program content.

In general, it was very important to earn the principal's trust through clear, reliable communication, professionalism, flexibility and by following through on commitments. The principal's support was reflected in the dedicated, enthusiastic school staff who organized timetables, assigned rooms, made arrangements for non-participating students, followed up on attendance, obtained consents and fielded questions from parents and teachers.

Implications for research, policy and practice

To facilitate uptake of the program, we developed a Facilitator Resource Guide. The guide included everything needed to implement Healthy Transitions in a school community:

• background information on evaluation, consent and how to get started
• follow-up protocols and resources
• an evaluation guide and tools
• sample consent forms
• curriculum for five classroom sessions (including handouts)

We distributed hard copies of the guide (along with a CD) to organizations at national, provincial and local conferences and upon request. We met with officials from the Ontario Ministries of Education and Children and Youth to outline the program (the entire program is online). As a result, several local schools have piloted the program independently. We have received requests for the curriculum from schools, public health units, community health centres and youth-serving organizations across Canada.

Healthy Transitions was designed for use in any school and could easily be adapted for youth who are in their mid to late teens, attending a youth program or experiencing a mental health problem. The program is suitable for broad implementation across school boards or provincial ministries of education. The Facilitator Resource Guide highlights where modifications may be required to meet legislative or policy requirements (for example, provincial reporting requirements for children in need of protection). Local jurisdictions would also need to highlight local mental health resources for youth.

We suspect that the screening and referral component is a barrier to uptake. The process requires a thorough review of all completed screening questionnaires, application of follow-up criteria, follow-up with individual students, assessment of immediate risk, and appropriate referral. The time, skill and commitment needed to safely screen, assess and refer students may discourage some considering introducing Healthy Transitions. The facilitator-led model was recommended by youth in our focus groups but this also may be challenging for schools to organize. Eliminating the screening component or having teachers help deliver the curriculum might make things easier.

The scope of this project was limited to program development and initial evaluation. While our program evaluation was promising, it lacked the rigour that provincial ministries and school boards look for when allocating scarce resources. Researchers interested in this area might compare youth who receive the program with those who do not, assess the long-term impact of the program, and determine if the program could be delivered effectively by classroom teachers with larger groups of students (for example, an entire class). We would also like to know if the screening/referral component is essential to the program's success. Schools currently devote considerable resources to anti-bullying programming; it would be interesting to compare the overall impact of existing anti-bullying programs with a more generic mental health promotion program such as Healthy Transitions.

The Healthy Transitions project represents a first step toward policy changes needed for comprehensive mental health promotion in schools. It established an effective partnership model, highlighted processes critical for successful program delivery and demonstrated positive impact on students' emotional states and coping skills. It provided essential knowledge not only about the elements a program should include, but also about how to implement a mental health promotion program in schools. More rigorous demonstrations of program impact are needed to effect policy changes related to school curriculum, public health activities and resource allocation.

It is a challenge for service-providing health professionals to engage in population health intervention research. Most have workloads that do not allow for research beyond the realm of program evaluation. These professionals tend, therefore, to limit research activities to developing or testing practical solutions that address a specific problem, rather than planning publishable research projects from the outset. This leaves potentially valuable interventions in a 'promising practices' kind of purgatory, where decision makers are not likely to implement them broadly without higher-quality evidence. The collaboration and support of experienced principal investigators for community-identified projects would be most helpful. Such leadership and guidance would be very welcome to community professionals for planning, conducting and securing appropriate funding for high-quality population health intervention research.

Acknowledgements

The Healthy Transitions curriculum was written by Corrine Langill and Ann Croll. Funding was provided by the Provincial Centre of Excellence for Child and Youth Mental Health at CHEO. Organizational members of the advisory group include:

• The Children's Hospital of Eastern Ontario
• The Child and Youth Health Network for Eastern Ontario
• Carlington Community Health Centre
• Ottawa Carleton District School Board
• Ottawa Carleton Catholic School Board
• Western Ottawa Community Resource Centre
• Centre for Addiction and Mental Health
• Ottawa Public Health
• Canadian Mental Health Association

References

Waddell, C., McEwan, K., Shepherd, C.A., Offord, D.R., Hua, J.A. (2005). A public health strategy to improve the mental health of Canadian children. Canadian Journal of Psychiatry 50(4) 226-233.

Scaling Up a School-based Pilot

The Heart Healthy Kids (H2K) program brings together school, public health and community groups to evaluate the effectiveness of a school-based pilot program. Using a mixed methods approach, the evaluation team explored the value in supporting behavioural change through individual education and the creation of more supportive environments and social structures. The pilot is being scaled up for implementation in more contexts, laying the foundation for future population health intervention research.

Case 5 - H2K - The Heart Healthy Kids Program

Rebecca Spencer, MA Health Promotion Student, Dalhousie University, Maritime Heart Center, Halifax, NS
Jenna Bower, OT Reg (NS), Maritime Heart Center, Halifax, NS
Jennifer Hoyt, Halifax Infirmary, QE II Health Sciences Centre, Halifax, NS
Jennifer Miller, Halifax Infirmary, QE II Health Sciences Centre, Halifax, NC
Gillian Yates, Halifax Infirmary, QE II Health Sciences Centre, Halifax, NS
Sara Kirk, School of Health Administration, Dalhousie University, Halifax, NS
Camille Hancock Friesen, Department of Surgery, Dalhousie University; Division of Cardiac Surgery, IWK Health Centre; Halifax Infirmary, QE II Health Sciences Centre, Halifax, NS

Corresponding author: Becky Spencer, becky.spencer@dal.ca

Introduction

Only 12 percent of Canadian children are sufficiently active (Active Healthy Kids Canada, 2009) and the prevalence of childhood obesity and type 2 diabetes is growing. To respond to these health issues, the Heart Healthy Kids (H2K) research program was designed as an intensive physical activity intervention involving the assessment of children's physical activity levels, heart health knowledge and cardiovascular fitness. Our long-term goal is to improve levels of childhood physical activity sufficiently to enact a culture change that, in turn, will decrease sedentary behavior and, ultimately, decrease the prevalence of chronic disease.

The project has adopted a population health approach to target youth as a vulnerable group needing intervention. To target the group as a whole, the H2K program is school-based, allowing it to reach all children in the involved schools, regardless of socioeconomic status or other factors that typically influence participation in research programs (Sonneville, LaPelle, Taveras, Gillman & Prosser, 2009). While H2K is a research project, some program components are offered to all students (whether they have consent for participation or not) allowing everyone equal opportunity to participate. As the program expands its enrolment from 150 to, potentially, 1,200 students, additional components will be made available to a larger group of students.

To increase the relevance of the research project and more generally promote awareness of the applicability of a population health approach, the H2K program has integrated various knowledge translation activities. H2K is helping to support population health research by focusing on vulnerable populations and using an ecological approach with a focus on the social environment.

Project description and lessons learned

The H2K program was developed by a team of health professionals to improve levels of childhood physical activity. The program began in 2006 as a one-school, three-year pilot with grade 4, 5 and 6 students, and is based on social cognitive and ecological theory. These grades were chosen to address a gap in local research, which has shown that grade 3 students attain adequate physical activity but grade 7 students do not. In this school, an activity challenge was coupled with biannual educational sessions given by health professionals. The activity challenge was for teams of students to wear pedometers daily, for the entire school year and compete for virtual distance traveled. This pilot was conducted with an adult mentor present four hours per week in Year 1 and without an adult mentor in Year 2. A second pilot was conducted in the 2009-2010 school year, employing peer mentoring instead of adult mentoring. Peer mentors were trained at a one-day, in-school workshop to develop leadership skills and learn about H2K. Peer mentors then had the opportunity to mentor teams of students to increase their physical activity. Employing peer mentors helps researchers understand the previously unstudied role of peer mentoring in improving physical activity. Peer mentoring also addresses the issue of sustainability in attempts to effect cultural change.

Both pilots saw increases in participants' average steps per school day in the presence of mentoring as well as short- and long-term knowledge gains with respect to heart health. In addition, measured cardiovascular fitness improved. Qualitative data derived from focus groups showed positive trends in the theme areas of entertainment, health and motivation. These themes indicated, first, that the participants enjoyed taking part in the program, that it was fun, and that they wished to continue taking part (entertainment). Second, the H2K program supported the adoption of healthy habits and recognition of the importance of physical activity and healthy eating with regard to disease prevention (health). Finally, the qualitative data indicated that motivation through peer mentoring in the H2K program was important in strengthening social relationships at school as well as engaging in new or extra physical activity (motivation).

One of the biggest lessons we have learned is the importance of school-based intervention programs; previous research has shown that children who are least active overall get the majority of their activity during the school day (Cox, Schofield, Greasley & Kolt, 2006). Buy-in and support from teachers and school staff is critical to the success of the program and we would recommend that future school-based interventions adapt to suit the specific needs of individual schools. We have also learned of the challenges associated with collecting self-reported data from children, and of program fatigue. We have had to come up with new and innovative methods to keep interest in the program alive, such as awarding not only high-performing teams, but also teams who log the most frequently or demonstrate the most spirit. Finally, a major challenge we consistently face is in finding enough resources for the program. Because H2K is a program with a research component, we are often not eligible for resources designated strictly for either interventions or research. We have streamlined the program for efficiency, including using peer mentors, which will enhance sustainability by avoiding the resource intensiveness of hiring a registered nurse as an adult mentor.

Additionally, we have learned the importance and value of collaboration. We have a multi-disciplinary volunteer steering committee (including a cardiac surgeon, registered nurses, a dietitian, kinesiologists and an occupational therapist); we also have informal partnerships with local elementary schools, their staff and students, and formal partnerships with the Halifax Regional School Board, the Heart and Stroke Foundation of Nova Scotia, and Public Health (Capital District Health Authority). The upcoming expansion of the program will allow us to develop partnerships with medical students, who will deliver the educational sessions, and approximately 50 volunteers (community members and university students in professional health programs), who will assist with program deployment.

Implications for research, policy and practice

Our pilot work has shown that mentoring is associated with trends toward increased physical activity levels, educational sessions are associated with short- and long-term knowledge gains, students who are enrolled in organized sport are more active overall and the vast majority of students do not meet minimum activity recommendations. While the project is too preliminary to have influenced true policy change, continued expansion of the program is evidence of uptake. As of September 2010, the H2K program was expanded to 10 schools including five experimental and five control schools, which allows for a true control arm and more robust quasi-experimental design. The research objectives of the expansion are to evaluate the role and effectiveness of peer mentoring in improving levels of physical activity, with secondary endpoints of heart health knowledge, cardiovascular fitness and anthropomorphic data. A nutritional questionnaire and parent surveys have been added to further examine the complex relationship between children and their environment that negatively impact physical activity and enhance rates of childhood obesity, and to ensure more thorough results that can be generalized.

Knowledge translation is important as we expand the scope and range of the H2K program and we are dedicated to seeking opportunities to present the research as well as publish results. Our pilot data suggest that the H2K program, including peer mentoring, shows promise for future population health impact. Preparations to scale up the intervention research program have begun with receipt of institutional ethics and school board approval, as well as enrolment of 10 new elementary schools. A partnership forum occurred in September 2010 to engage further private and public interest and funding.

The research team intends to use the results of the scaled up research as evidence for policy change in relation to childhood physical activity by advocating and working with local government. Currently, in Nova Scotia, elementary students typically receive only two 30-minute periods of physical education weekly; it is recommended that children receive 90 minutes of activity daily, leaving individuals and families responsible for ensuring children receive the majority of their necessary activity (Public Health Agency of Canada and Canadian Society for Exercise Physiology, 2009). We hope that the H2K program provides enough evidence for school-based physical activity that a policy change will ensure all students have equal opportunity for a guaranteed minimum of daily physical activity.

Beyond the 10-school expansion, the H2K program offers many opportunities for future research. The research team anticipates a longitudinal follow-up of research participants to further explore the relationship between the H2K program and activity patterns, as well as the relationship with cardiovascular disease. Other research may include closer examination of mentor type, as well as frequency and duration, to better understand what can positively influence childhood physical activity levels. Beyond the research component of the H2K program, we also intend to expand the program to include more local schools from around the province of Nova Scotia, the other Atlantic Provinces and beyond.

In conclusion, the research team strongly believes that there is evidence for the use of mentoring in adopting healthy habits in childhood, and that a population health approach is necessary in order to widely improve the current level of childhood physical activity. We believe that the expansion of the H2K program will provide the evidence necessary to change policy and provide all children with equitable access to physical activity. We recommend that other researchers in this area adopt a population health approach and focus on the social environment to produce durable change.

Acknowledgements

The Maritime Heart Center, Division of Cardiac Surgery (Capital District Health Authority), Cobequid Community Health Board, Pfizer, GlaxoSmithKline, IWK Health Centre, Nova Scotia Health Research Foundation

References

Active Healthy Kids Canada. (2009).The Active Healthy Kids Canada report card on physical activity for children and youth. Retrieved from the Active Healthy Kids Canada website.

Cox, M., Schofield, G., Greasley, N., & Kolt, G.S. (2006). Pedometer steps in primary school-aged children: A comparison of school-based and out-of-school activity. Journal of Science and Medicine in Sport 9 91-97.

Sonneville, K., LaPelle, N., Taveras, E., Gillman, M., & Prosser, L. (2009). Economic and other barriers to adopting recommendations to prevent childhood obesity: Results of a focus group study with parents. Retrieved from the BioMedCentral website.

Yates, G., Cornish, W., Miller, J., & Hancock Friesen, C. L. (2009). Investing in heart healthy children: A primary prevention innovation in Atlantic Canada. Canadian Journal of Cardiovascular Nursing 19 20-25.

Date modified:

2011-11-29