EVALUATION OF THE SALMONID
ENHANCEMENT PROGRAM

6B167

FINAL REPORT
March 2015

EVALUATION DIRECTORATE


Table of Contents

ACKNOWLEDGEMENTS AND ACRONYMS


ACKNOWLEDGEMENTS

The Evaluation Directorate acknowledges and thanks all individuals who gave of their time and input for this evaluation of the Salmonid Enhancement Program (SEP), in particular, all stakeholders and staff members who shared their thoughts through site visits, phone interviews and an online survey. The Evaluation Directorate also acknowledges the time and effort on the part of SEP staff given to the evaluation team.

ACRONYMS


List of acronyms
BC British Columbia
CA Community Advisor
CEDP Community Economic Development Program
DFO Fisheries and Oceans Canada
DPI Designated Public Involvement Facility
DPR Departmental Performance Report
EMB Ecosystem Management Branch
FTE Full-time equivalent
GDP Gross domestic product
IFMPs Integrated Fisheries Management Plans
O&M Operations and Maintenance
PAR Pacific Aquaculture Regulations
PIP Public Involvement Program
PSF Pacific Salmon Foundation
PSF’s CSP Pacific Salmon Foundation’s Community Salmon Program
PST Pacific Salmon Treaty
RDG Regional Director General
SEHAB Salmon Enhancement and Habitat Advisory Board
SEP Salmonid Enhancement Program
TBSEF T. Buck Suzuki Environmental Foundation
US United States
WSP Wild Salmon Policy

EXECUTIVE SUMMARY


INTRODUCTION

This report presents the results of the evaluation of the Salmonid Enhancement Program (SEP) including Fisheries and Oceans Canada’s (DFO) contribution agreements with the Pacific Salmon Foundation’s Community Salmon Program (PSF’s CSP) and with the    T. Buck Suzuki Environmental Foundation (TBSEF). The evaluation was conducted by DFO’s Evaluation Directorate in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation.

The main objective of this evaluation is to determine if the program is still relevant and the extent to which SEP is managed effectively and efficiently and has achieved its stated outcomes. The program operates exclusively in the Pacific region (including British Columbia and the Yukon Territory). The evaluation covers a six-year period from 2008-09 to 2013-14 for SEP and a five-year period from 2009-10 to 2013-14 for DFO’s contribution agreements with PSF’s CSP and with TBSEF. Recommendations stemming from the main findings were developed to support improvements to the program and to inform future decision-making.

PROGRAM PROFILE

The program focuses on the production of Pacific salmon from hatcheries and spawning channels to directly provide harvest opportunities for all sectors (commercial, recreational and First Nations) under the Fisheries Act and the Pacific Aquaculture Regulations and to enable harvest opportunities through the provision of stock assessment information for domestic and international harvest management. The program also includes restoration and enhancement of habitat for fish production, as well as education and awareness programs to facilitate the participation of First Nations, local communities, external parties and other levels of government in cooperative fisheries and watershed stewardship activities.

SEP also supports Canada’s Wild Salmon Policy as well as international treaty negotiations and requirements such as the Pacific Salmon Treaty. SEP plays a key role in DFO’s work to conserve and manage Pacific salmon stocks. The program’s activities aim to protect vulnerable salmon stocks, provide harvest opportunities, work with First Nations and coastal communities in economic development, and improve fish habitat to sustain salmon populations.

SEP is supported by three program components: Salmonid Enhancement Operations includes the production of Pacific salmon from enhancement facilities;  SEP’s Community Involvement Program brings people from communities throughout the Pacific Region together to participate in locally-based stewardship efforts; and Salmonid Enhancement Contribution Programs provide funding to support the delivery of community-based salmon and fish habitat projects −  DFO has established contribution agreements with the Pacific Salmon Foundation and with the T. Buck Suzuki Environmental Foundation.

EVALUATION METHODOLOGY

This evaluation adopted a theory-based approach whereby program performance was measured against intended results articulated in the program logic model. The evaluation questions were determined on the basis of the Treasury Board’s Policy on Evaluation, a review of key program documents, and results from preliminary discussions with key program personnel. The chosen design was able to demonstrate the extent to which the program is achieving issues of relevance, effectiveness, efficiency and economy. Extensive use of triangulation was undertaken as an analytical method, where multiple lines of evidence helped corroborate findings.

The following lines of evidence were used for the evaluation: document review and file review; key informant interviews with senior management and program personnel, internal DFO partners and external stakeholders; two online surveys (one with SEP employees and one with SEP external stakeholders); and, site visits in two areas. Findings were determined by triangulating the multiple lines of evidence with conclusions being drawn from the findings. This approach was taken in order to demonstrate reliability and validity of the findings and to ensure that conclusions and recommendations are based on objective and documented evidence.

A number of methodological limitations and mitigation strategies were identified, such as availability of performance data and program documentation, key informant interviews and survey responses, timing of surveys, site visits and attribution challenges.

EVALUATION MAJOR FINDINGS AND RECOMMENDATIONS

Relevance

Overall, the evaluation found that there is a continued need for the Salmonid Enhancement Program through the Pacific Salmon Treaty and the Wild Salmon Policy. SEP remains relevant as its activities also contribute to the Government of Canada outcomes and supports DFO’s strategic outcomes.

Previous studies determined that the program contributes approximately $89 million to British Columbia’s gross domestic product and generate direct employment of 592 jobs. SEP, through an agreement with the Province of British Columbia, is producing Steelhead and Cutthroat trout species at its hatcheries but the management of these species is shared between the federal government and the province of British Columbia.

Activities undertaken as part of the contribution agreement with the Pacific Salmon Foundation’s Community Salmon Program share similar objectives to SEP. The contribution agreement with T. Buck Suzuki Environmental Foundation activities ultimately leads to an improved marine ecosystem, thus supporting healthy salmon stocks; however linkage between TBSEF and SEP is weak.

Performance (Economy, Efficiency and Effectiveness)

SEP immediate, intermediate and long-term outcomes have been met to some extent. The evaluation faced attribution challenges as some outcomes are out of SEP’s control. Moreover, lack of evidence did not permit to conclude on the achievement of some outcomes, such as “vulnerable salmon species populations are supported”, “the public has access to harvest opportunities”, and “sustainable marine and freshwater ecosystems provide harvest opportunities”. Additional linkages with Ecosystems and Oceans Science Sector as well as Ecosystems and Fisheries Management Sector would be beneficial as SEP is dependent on other groups’ expertise to advance some of its activities such as stock assessment.

SEP provides harvest opportunities through enhancing salmon stocks and is successful in engaging stewards to participate in cooperative fisheries and stewardship activities. Through the Pacific Streamkeepers Federation, PSF’s CSP, habitat restoration, stock assessment and enhancement, SEP helps the public support the conservation, stewardship and rebuilding of salmon and their habitat, but there is room for improvement. For example, the evaluation found that SEP has no strategy and/or vision related to habitat restoration and how the contribution of its delivery partners helps SEP to achieve its results is not clear.

SEP could not achieve the same results without additional funding from external sources, such as the Northern Boundary and Transboundary Rivers Restoration Enhancement Fund and the Southern Boundary and Transboundary Rivers Restoration Enhancement Fund. Additionally, SEP leverages funds through major facilities, Community Economic Development Program (CEDP), Public Involvement Program (PIP), Community Advisors, restoration projects and both contribution agreements.

Respondents from both surveys agree that PSF’s CSP has not achieved its outcome of “First Nations, local communities and external parties participate in cooperative fisheries and watershed stewardship activities” established in PSF’s CSP contribution agreement. In regard to the achievement of the other two outcomes established in the contribution agreement “Increased collaboration among community stakeholders in salmon enhancement, habitat conservation and restoration projects” and “Public Awareness about salmon in Pacific Region is increased”, the views of respondents are diverging. SEP employees are of the opinion that PSF’s CSP is not meeting those outcomes, while stakeholders disagree.

The program is successful in collecting and using data with respect to fish production. However, SEP’s data collection methods regarding community involvement activities are inconsistent across areas and vary from year to year. Improved documentation and data collection on how the many components of the program contribute to SEP’s outcome would be beneficial to SEP as it could better demonstrate the program’s achievements. Also, a review of performance indicators and the logic model for SEP and its contribution agreements would be necessary in order to effectively report on program achievements.

The majority of employee survey respondents believe that changes are necessary in the design and delivery of SEP. The evaluation revealed that improvements can be made to SEP’s governance and leadership as well as the communication of priorities to employees. SEP would benefit from communicating clear directions to ensure consistency in delivery throughout Area Offices. More specifically, there is a need to establish priorities and clear roles and responsibilities for Community Advisors as delivery of the program varies from one Area to another.

There is also a need to clarify the roles and responsibilities in the contribution agreement between DFO and PSF’s CSP related to SEP employees’ involvement in the project selection process as well as in the delivery of some PSF’s CSP funded projects. PSF’s CSP and ultimate recipients are dependent on DFO’s technical expertise and the large majority of respondents from both surveys are of the opinion that projects funded through PSF’s CSP could not be accomplished without SEP’s technical expertise.

Suggestions to improve SEP’s efficiency include the use of new technologies (e.g., high efficiency pumps with variable frequency drive) and amalgamating or transferring fish production of some CEDPs to major facilities, revisiting the governance model employed for non-production aspects of the program, and keeping partners more regularly apprised of program changes to increase the likelihood of stakeholder support and buy-in.

As for alternatives, the percentage of fish released from the community involvement facilities (CEDPs, Designated Public Involvement facilities and PIPs) represent only 8% of total fish released and major facilities are not producing at maximum capacity. Reviewing the role and contribution of community involvement facilities could be helpful in improving SEP’s efficiency and economy.

The identification of core versus non-core activities may also help SEP identify priorities and allocate resources. Specifically, there could be cost-sharing opportunities between SEP and the province of British Columbia for Steelhead and Cutthroat trout species. Education and awareness were activities also raised as potential non-core responsibilities.

Finally, during the period reviewed, the Salmon Enhancement and Habitat Advisory Board (SEHAB) was awarded $33,000 annually in order to provide SEP with advice from the volunteer community perspective on issues of program delivery and program policy.  As described in the Terms of Reference, SEHAB is required to provide an annual advisory report and one was provided as evidence. There is evidence that indicates SEHAB has provided advice to the program but this advice is not fully documented and did not come in the form of an annual report.

Recommendations

Recommendation 1: It is recommended that the Regional Director General (RDG), Pacific Region, review SEP’s Performance Measurement (PM) Strategy (logic model and performance indicators) to ensure that it appropriately reflects SEP’s performance story.  Any changes in the PM Strategy affecting the contribution agreements with the Pacific Salmon Foundation and with the T. Buck Suzuki Environmental Foundation should be reflected in their respective contribution agreements. Once the PM strategy is revised, ensure that data be collected and reported consistently across each area.

Recommendation 2: It is recommended that the RDG, Pacific Region, ensure that the renewal of the contribution agreement with PSF’s CSP defines the roles and responsibilities between DFO and PSF’s CSP, especially regarding the involvement of SEP employees in the selection and delivery of PSF’s CSP projects. Any changes to the contribution agreement should be communicated to all SEP employees.

Recommendation 3: It is recommended that the RDG, Pacific Region, advance the cost-sharing opportunities between DFO and the province of British Columbia with respect to the production of Steelhead and Cutthroat trout species.

Recommendation 4: It is recommended that the RDG, Pacific Region, develop a framework or strategy to identify SEP’s activities that are core to its mandate and to prioritize the non-core activities. The framework or strategy should include an action plan based on available resources, with specific timelines to implement necessary changes and re-focus SEP activities.

Recommendation 5: It is recommended that the RDG, Pacific Region, develop a habitat restoration strategy for SEP.

Recommendation 6: It is recommended that the RDG, Pacific Region, review the roles and responsibilities of the Salmon Enhancement and Habitat Advisory Board and ensure that SEHAB’s contribution to the achievement of SEP’s outcomes is clearly demonstrated.

1. INTRODUCTION


1.1 PURPOSE OF THE EVALUATION

This report presents the results of the evaluation of the Salmonid Enhancement Program (SEP) including Fisheries and Oceans Canada’s (DFO) contribution agreements with the Pacific Salmon Foundation’s Community Salmon Program (PSF’s CSP) and with the    T. Buck Suzuki Environmental Foundation (TBSEF). The evaluation was conducted by DFO’s Evaluation Directorate in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation, which requires all direct program spending be evaluated every five years. The evaluation commenced in January 2014, as per the Departmental Evaluation Plan, and was completed in February 2015. Recommendations stemming from the main findings were developed to support improvements to the program and to inform future decision-making.

1.2 SCOPE

The main objective of this evaluation is to determine if the program is still relevant and the extent to which SEP is managed effectively and efficiently and has achieved its stated outcomes.  The evaluation covers a six-year period from 2008-09 to 2013-14 for SEP and a five-year period from 2009-10 to 2013-14 for DFO’s contribution agreements with PSF’s CSP and with TBSEF.

Previous evaluations of SEP1 and the contribution agreements2 were completed in 2009-10. As part of the last SEP evaluation, three recommendations were made and a Management Action Plan (MAP) was developed by the program to address the recommendations. The recommendations were: 1) To develop and implement a succession and mentoring plan; 2) To finalize SEP’s logic model and its Performance Measurement Strategy; and 3) To undertake an assessment of SEP facilities. With respect to SEP’s contribution agreements, it was recommended that a Performance Measurement Strategy be developed with clearly defined indicators and measures to demonstrate how, and to what extent, the activities undertaken by ultimate recipients of DFO funds contribute to the achievement of the Salmonid Enhancement Program. Overall, the MAPs addressing these recommendations were implemented as planned and carried out within a reasonable time.


1 http://www.dfo-mpo.gc.ca/ae-ve/evaluations/09-10/6b105-eng.htm

2 http://www.dfo-mpo.gc.ca/ae-ve/evaluations/09-10/6b122-eng.htm

1.3 REPORT STRUCTURE

The executive summary provides a brief outline of the evaluation findings, conclusions and recommendations. Section 2 presents an overview of the program. Section 3 describes the evaluation methodology, followed by a discussion of the main findings in section 4, and conclusions and recommendations in section 5. The evaluation matrix in Annex A presents the key issues investigated (relevance, effectiveness, efficiency and economy), the specific evaluation questions addressed and the corresponding lines of evidence used for each question.

2. PROGRAM PROFILE


2.1 PROGRAM CONTEXT

DFO first launched the Salmonid Enhancement Program in 1977 to increase the catch of salmon in British Columbia and the Yukon.  The program continues to focus on the production of Pacific salmon from hatcheries and spawning channels to directly provide harvest opportunities for all sectors (commercial, recreational and First Nations) under the Fisheries Act and the Pacific Aquaculture Regulations and to enable harvest opportunities through the provision of stock assessment information for domestic and international harvest management. The program also includes restoration and enhancement of habitat for fish production, as well as education and awareness programs to facilitate the participation of First Nations, local communities, external parties and other levels of government in cooperative fisheries and watershed stewardship activities.

SEP also supports Canada’s Wild Salmon Policy (WSP), which came into effect in 2005, as well as international treaty negotiations and requirements such as the Pacific Salmon Treaty (PST) which was renewed in 2008.

2.2 PROGRAM ACTIVITIES

SEP plays a key role in DFO’s work to conserve and manage Pacific salmon stocks. The program’s activities aim to protect vulnerable salmon stocks, provide harvest opportunities, work with First Nations and coastal communities in economic development, and improve fish habitat to sustain salmon populations. SEP is supported by program components, such as Salmonid Enhancement Operations, Community Involvement Programs, and Salmonid Enhancement Contribution Programs.

Salmonid Enhancement Operations

Salmonid Enhancement Operations includes the production of Pacific salmon from enhancement facilities.  The program delivers fish production services through a number of enhancement facility types. Enhancement facilities fall into four groups:

  1. Sixteen Major (16) Enhancement Facilities (also called major facilities) managed by DFO employees and seven (7) major spawning channels;
  2. Eighteen (18) Community Economic Development Program (CEDP) facilities operated by employees of local community organizations;
  3. Three hundred and fifty (350) Public Involvement Projects (PIPs) ranging from stewardship projects to small hatcheries (57 PIP hatcheries and 25 Designated Public Involvement (DPI) facilities) which are operated mostly by volunteers (approximately 10,000 volunteers); and,
  4. Habitat enhancement projects (average of 50+ new ones added annually) such as fish ways, side channels and water control structures.   

Community Involvement Program

SEP’s Community Involvement program brings people from communities throughout the Pacific Region (including British Columbia and the Yukon) together to participate in locally-based stewardship efforts.  The program has many components, including:

  • Community Advisors who provide technical advice and financial support to community volunteers wishing to pursue salmon restoration efforts;
  • The Community Economic Development Program (CEDP) works with First Nation communities and volunteers to rebuild salmon stocks through enhancement, restoration, and education efforts.  In addition to supporting community involvement, CEDPs also support Salmon Enhancement Operations, as previously mentioned.
  • The Public Involvement Program (PIP) supports local communities in their efforts to re-establish salmonid populations in rivers and streams in their communities.  This program supports approximately 350 projects delivered by 10,000 volunteers.  Some PIP projects also contribute to Salmon Enhancement Operations.
  • The Stream to Sea Program includes a range of educational material available to educate schoolchildren on the value of salmon resources.  Salmonids in the Classroom is a component of the Stream to Sea program which is endorsed by the BC Ministry of Education and has been studied by more than one million students throughout the Pacific Region.
  • Through the Storm Drain Marking Program, local residents and children identify storm drains by painting them with a yellow fish. The objective of the program is to remind residents that storm drains empty directly into local streams, therefore harmful products should be disposed of properly.

Salmonid Enhancement Contribution Programs

Salmonid Enhancement Contribution Programs provide funding to support the delivery of community-based salmon and fish habitat projects.  DFO has established contribution agreements with the Pacific Salmon Foundation (PSF) and with the T. Buck Suzuki Environmental Foundation (TBSEF).

Pacific Salmon Foundation

The Pacific Salmon Foundation is a not-for-profit organization dedicated to promoting conservation, restoration and enhancement of Pacific salmon. DFO funds are channeled to Pacific Salmon Foundation’s Community Salmon Program (PSF’s CSP) through a contribution agreement in order to carry out salmon conservation, restoration and enhancement projects throughout British Columbia. Typical project activities funded through PSF’s CSP include: public education and awareness, habitat assessment, habitat enhancement, community stewardship and watershed planning, stock assessment and stock enhancement.

Between 2009-10 and 2012-13, DFO contributed on average $348,203 annually to PSF’s CSP through a portion of the revenues from the salmon conservation stamp fee. PSF leverages DFO’s contribution by generating additional revenue through sponsorships, cash and in-kind donations, fundraising events, etc. In addition, it is required that ultimate recipients provide matching funds of at least 50% from other sources and partners. In Budget 2013, the Government of Canada announced that the full annual revenue produced by the salmon conservation stamp (approximately $1.3M annually) would be made available to the PSF to support its work for salmon and their habitat.

T. Buck Suzuki Environmental Foundation

The T. Buck Suzuki Environmental Foundation is a not-for profit organization involved in numerous habitat-related campaigns, including efforts to halt pulp and sewage pollution, destructive logging practices and water diversions.  Habitat work carried out by the Foundation is supportive of all fish species, including Pacific salmon.  A contribution agreement between DFO and TBSEF is established in order to deliver on SEP’s immediate outcome of First Nations, local communities and external parties participate in cooperative fisheries and watershed stewardship activities, by promoting awareness and involvement in conservation, protection and restoration of Canadian Pacific fish stocks and habitat to achieve healthy, sustainable and naturally diverse stocks and habitat.

DFO contributes an average of $26K annually to TBSEF based on the revenue collected from the surcharge on Fisher’s Registration Cards. The foundation is required to provide matching financial assistance and in-kind assistance of at least 50% in the delivery of projects.

2.3 EXPECTED RESULTS

SEP has defined five specific objectives for fish enhancement (also referred to as fish production) which flow from the program logic model (see Figure 1).  Each production line (defined by project, species, run, stock, brood year, release stage and release location) of fish considered through production planning must address at least one of these objectives:

Harvest – enhancement for fisheries that are reliant on enhanced production, and would disappear or become severely constrained in the absence of enhancement. This includes harvest opportunities for First Nations, recreational, or commercial fisheries. When the objective is to provide a targeted-fishery opportunity, production targets may be set to consider both natural spawning and harvest requirements.

Assessment – fish produced for marking where stock assessment information contributed to Pacific Region assessment priorities, such as the Pacific Salmon Treaty. The information may also contribute to assessment as defined under the regional stock assessment framework, area stock assessment priorities and regional SEP assessment priorities (i.e., those produced for program performance measurement). Fish produced for assessment generally address other objectives as well but, in a few instances, fish are produced solely for marking for assessment purposes.

Conservation – enhancement of a stock highly at risk of extirpation or extinction, or a vulnerable stock that has been identified as a regional priority (e.g., populations which have an approved conservation / recovery strategy).  This includes re-establishing locally extinct populations and rebuilding populations at high risk of extirpation.

Rebuilding – enhancement of a stock that is below apparent carrying capacity.  This includes rebuilding depleted populations and mitigating for habitat loss.

Stewardship and Education – small number of fish produced to provide a stewardship or educational opportunity. Production for these purposes is assessed based on contribution to stewardship and educational goals and not on production levels.

2.4 PROGRAM LOGIC MODEL AND PERFORMANCE MEASUREMENT

As a result of the previous SEP evaluation, completed in 2009, and a DFO wide initiative, the program developed and implemented a performance measurement strategy.  The strategy provides details on the program and its objectives. The program’s logic model, shown in figure 1, informed this evaluation.

Figure 1. SEP Logic Model

Figure 1

2.5 PROGRAM STAKEHOLDERS

Through SEP’s network of partners, the program is able to leverage financial and human resources to significantly extend the reach of SEP’s delivery.  SEP continues to work on ways to maintain existing partnerships and attract new delivery partners and volunteers.

Key program partnerships include:

  • DFO sectors such as Ecosystems and Fisheries Management, Ecosystems and Ocean Science, and Aboriginal Affairs Directorate.  Through the regional production planning process, staff from these sectors provide input on the priorities that ultimately determine SEP’s annual fish production targets.
  • DFO’s Real Property, Safety and Security Directorate manages SEP’s major facilities.  SEP’s hatcheries, spawning channels and resource restoration projects comprise a significant part of DFO’s Pacific Region infrastructure, having an estimated replacement value of approximately $500 million.
  • SEP’s Community Economic Development Program (CEDP) operates small and medium sized production facilities under contract in communities across the province of British Columbia. Many CEDP facilities are run by First Nations and provide employment in rural locations.
  • SEP’s Public Involvement Projects (PIP) are operated by local community groups and First Nations and include projects that focus on education, fish production, outreach and stewardship initiatives.
  • Advisory groups such as the Salmon Enhancement and Habitat Advisory Board (SEHAB) and the Pacific Streamkeepers Federation have a formally recognized role in providing advice and client perspective to SEP on issues of program delivery and program policy.
  • The Pacific Salmon Foundation and the T. Buck Suzuki Environmental Foundation both deliver salmon restoration and rehabilitation projects through separate contribution agreements established with DFO.
  • DFO has also established an agreement with the Province of British Columbia (BC) for the production of Pacific salmon species and anadromous trout species. Therefore SEP has undertaken the production of Steelhead and Cutthroat species at SEP major hatcheries.

2.6 GOVERNANCE

SEP is a program within DFO’s Ecosystems and Fisheries Management (EFM) Sector. The program is under the responsibility of the Assistant Deputy Minister, EFM.  However, the program is regionally based in the Pacific Region and the Regional Director General, Pacific Region, is accountable for the development and delivery of SEP.

During the time period covered by the evaluation (2008-09 to 2013-14), Regional program leadership was provided by the Director of the Salmonid Enhancement Program who reported to the Regional Director, Ecosystem Management Branch (EMB) in the Pacific Region. In 2010, the SEP Director was supported in program leadership and management with five regional management positions: Managers of SEP Planning and Assessment, Community Involvement, Enhancement Operations, Resource Restoration, and Strategic Initiatives. As of April 1, 2014, the Director position of the Salmonid Enhancement Program was abolished.  SEP’s management team responsible for development and delivery of SEP programming region-wide now reports directly to the Regional Director, Ecosystems Management Branch.  There three remaining management positions within the team are: Manager of SEP Planning and Assessment, Manager of Stewardship and Community Involvement and Manager, Enhancement Operations.  The Manager of Strategic Planning was also reassigned in April 2014 from reporting to the Director of SEP to reporting to the Regional Director, Ecosystems Management Branch and now provides support to all EMB programs including SEP.

The Regional Director, Ecosystems Management Branch sits on three key regional committees including the Regional Management Committee, the Strategic Directions Committee and the Operations Committee. These regional committees have similar membership but different roles.  The Regional Management Committee is chaired by the Regional Director General and is the primary executive decision-making body and focuses on key program and corporate issues affecting the Region.  The Strategic Directions Committee focuses on issues of a tactical nature, for example the review and analysis of proposed, program-specific policy guidance.  The Operations Committee deals with issues such as program integration and business planning.  Both the Operations Committee and the Strategic Directions Committee provide recommendations to the Regional Management Committee, which holds ultimate decision making authority for issues of importance to the Region.

There are five Areas in the Pacific Region (BC Interior, Lower Fraser, North Coast, South Coast and Yukon-Transboundary Rivers) in addition to Regional Headquarters located in Vancouver.

SEP has direct linkages to a number of key programs within DFO including Science, Fisheries Resource Management, and Aquaculture Management Division.  SEP, along with other sectors such as Science, provide input into the development of annual Integrated Fisheries Management Plans for Pacific salmon stocks, which is key to ensuring that SEP meets both the access requirements of resource users and the data requirements for scientific stock assessment. Externally, SEP has a wide range of linkages to community groups, enhancement societies, first nations, and formal advisory bodies such as the Salmon Enhancement and Habitat Advisory Board and the Pacific Streamkeepers Federation.

2.7 PROGRAM RESOURCES

SEP’s total annual budget is approximately $26M of which approximately $11.7M is dedicated to operations and maintenance (O&M). The budget is allocated every year across the five areas and Regional headquarters. Table 1 provides a yearly breakdown of SEP’s FTEs, budget, actual expenditures and funding of contribution agreements.

Table 1. SEP FTEs, budget, expenditures and funding of contribution agreements from 2008-09 to 2013-14

Table 1
  2008-09 2009-10 2010-11 2011-12 2012-13 2013-14
Full-time equivalent:
FTEs (#) 217.50 217.50 212.60 212.10 212.10 214.40
Budget and Expenditures:(million $)
Total Budget 26.768 27.035 26.403 26.322 25.761 26.518
Salaries and wages (Expenditure) 13.910 14.740 14.771 14.667 14.449 15.208
O&M (Expenditure) 12.902 12.759 12.605 11.614 11.291 11.303
Total Program Expenditure 26.812 27.499 26.776 26.281 25.739 26.511
Funding Shortfall -0.044 -0.464 -0.373 0.041 0.021 0.007
Contribution Agreements:(million $)
PSF’s CSP
Contribution agreement
0.371 0.324 0.361 0.352 0.347 1.303*
TBSEF
Contribution agreement
0.028 0.027 0.026 0.028 0.026 0.027*

Source: Data provided by the program.
*The 2013-14 financial data are projected contribution figures

From its $11.7M total O&M budget, SEP dedicates $4.7M to the Community Involvement Program (CIP) which is allocated as follows: $3.2M to CEDPs, $260K to technical support contracts, $400K to Education (Stream to Sea program), $560K to Public Involvement Projects (PIPs) and $280K for individual travel, vehicle mileage and maintenance, training, equipment, etc.

For CEDP production facilities, SEP enters into contracts with First Nations and community organizations to operate these hatcheries with a total annual budget of $3.5 million.  To enable the delivery of this contracted work, SEP has a special non-competitive $1M contracting authority approved by Treasury Board (extended for 10 years in 2009).   Currently, there are 18 CEDP facilities which are situated on either private, crown or First Nations land. SEP plays a major role in the funding, planning, design and some maintenance of these hatcheries.

2.8 RISK PROFILE

SEP identified key risks3 that could have the potential of affecting the attainment of program objectives. There are specific areas of congruence between the SEP internal risk profile and the Departmental Corporate Risk profile.  The top three risks identified for SEP mirror the top three risks identified in the 2013-14 Departmental Corporate Risk profile including:

  • Financial Capacity: Due to the erosive effects of inflation, unfunded collective agreement settlements and incremental programming demands, SEP may not have the funding necessary to maintain existing levels of service delivery, compromising program objectives and client expectations.
  • Infrastructure: Due to its large, aging infrastructure holdings and insufficient maintenance and refit budget, SEP will see its facilities continue to deteriorate, creating both potential risks to the health and safety of staff, and the possibility of loss of enhancement production supporting Canada’s economic recovery.
  • Human Resources: Due to high rates of retirement related attrition, SEP risks losing key program knowledge and experience; it may not be possible to deliver programs required by clients as efficiently and as cost effectively in the absence of this knowledge and experience.

3 Salmonid Enhancement Program Risk Profile, 2012-13

3. METHODOLOGY


3.1 EVALUATION APPROACH AND DESIGN

This evaluation adopted a theory-based approach whereby program performance was measured against intended results articulated in the program logic model.  The chosen design was able to demonstrate the extent to which the program is achieving issues of relevance, effectiveness, efficiency and economy.

The analytical methods used for this evaluation were tailored to the nature and availability of the data which were in turn linked to the evaluation questions. Extensive use of triangulation was undertaken as an analytical method, where multiple lines of evidence helped corroborate findings.

3.2 KEY ISSUES AND EVALUATION QUESTIONS

The evaluation questions covered relevance and performance, including effectiveness, efficiency and economy. The evaluation questions were determined on the basis of the Treasury Board’s Policy on Evaluation (2009), a review of key program documents, and results from preliminary discussions with key program personnel. Annex A provides the detailed Evaluation Matrix.

3.3 DATA SOURCES

The evaluation drew on the following lines of evidence to gather data:

DOCUMENT REVIEW & FILE REVIEW

A review of documents and program files was conducted to assess most evaluation issues. The document review included government-wide documents such as federal legislation, regulations and policies pertaining to the program, planning documents, program performance measurement strategy, previous evaluations, etc. A file review also included administrative data such as contracts and financial reports. A sample of 71 PIP contracts was selected in order to examine alignment with SEP objectives and priorities. The sampling was designed by randomly choosing three PIP contracts per area (BC Interior, Lower Fraser, North Coast, South Coast and Yukon-Transboundary Rivers) for each year covered by the evaluation (2008-09 to 2013-14).  Only two contracts were available for fiscal year 2008-09 in the BC Interior region.

The review of relevant program documentation and files provided a comprehensive perspective of the activities, outputs and outcomes completed by the program, and were used to inform the assessment of both relevance and performance.

KEY INFORMANT INTERVIEWS

Interviews were conducted with program personnel, internal DFO partners and external stakeholders in order to gather insight about specific issues from those who work directly within the program as well as those clients and stakeholders who have accessed services from or provided services to the program.

Senior management and program managers within regional headquarters and Area Directors were selected to be interviewed based on geographical location and representation of program activities.  A total of nine program staff were interviewed (five of these interviews were carried out during the planning phase of the evaluation).  Internal DFO stakeholders from other Directorates such as Science, Aboriginal Affairs, Real Property and Fisheries Resource Management were also selected to participate in interviews based on their interactions with SEP.  A total of five internal DFO stakeholders were interviewed, including one group interview with Real Property.  Five external stakeholder organizations were also interviewed, including the Province of British Columbia, the Pacific Salmon Foundation, the T. Buck Suzuki Environmental Foundation, the Pacific Streamkeepers Federation, and the Salmon Enhancement and Habitat Advisory Board.  The interview with the Pacific Salmon Foundation was carried out in a group setting and the interview with the Province of British Columbia was carried out in written format with input from four employees.

Separate interview guides were developed and tailored to each group.  Interviews were conducted in person or by phone and took approximately 60 minutes. Interviews consisted primarily of open-ended questions, with a few ranking questions. Interviews were conducted in English, as the program is delivered exclusively in the Pacific Region.

SURVEY

Two online surveys were conducted, one with SEP employees and one with SEP external stakeholders. Surveys were available in English, as the program is delivered exclusively in the Pacific Region. The surveys helped in gathering information on staff and stakeholder perception of SEP’s relevance and performance. The surveys were launched in July 2014 and closed in September 2014. The survey tools were pre-tested by the evaluation team.  A notification e-mail was sent to all participants prior to the launch of the surveys and two reminders were sent out closer to the closing date.

Overall, the SEP employee survey had a response rate of 42.9%. A total of 90 surveys were completed out of the possible 210 staff members invited to participate in the survey.    The stakeholder survey had a response rate of 26.5%. A total of 95 surveys were completed out of the possible 359 stakeholders invited to participate in the survey.

The following terms have been used in some instances to report the proportion or frequency of responses in place of numerical data.

Terms
Proportion Terms Percentage range
All 100%
The majority 80-99%
Many 50-79%
Some 20-49%
Few 10-19%
Almost none 1-9%
None 0%

SITE VISITS

The evaluation team conducted a large amount of site visits of selected SEP major facilities, CEDPs, PIPs, habitat restoration projects and PSF’s CSP funded projects at the end of April and early May 2014.  SEP and/or PSF program representatives accompanied the evaluation team during these visits. These visits contributed to an understanding of the perceptions, opinions and experiences of stakeholders and employees who have significant experience with the program.

3.4 METHODOLOGICAL LIMITATIONS AND MITIGATION STRATEGIES

Although the evaluation encountered some challenges and limitations that are outlined below, these limitations were mitigated, as much as possible, through the use of multiple lines of evidence and triangulation of data. This approach was taken in order to demonstrate reliability and validity of the findings and to ensure that conclusions and recommendations are based on objective and documented evidence.

Availability of Performance Data and Program Documentation

Although the program collects large amounts of data regarding fish production, the data was not readily accessible in a way that aligns with performance indicators.  In addition, the program’s approach to reporting on production data through annual Departmental Performance Reports (DPR) changed from year to year, making comparisons difficult. For example, in some years, annual targets took into account all production lines, while in other years, production lines with the objective of education where removed for reporting purposes.

As for community involvement projects, data was inconsistently collected and reported across all regions.  To mitigate this issue, the evaluation team collected data from various sources (Integrated Fisheries Management Plans, program documents, PSF’s CSP annual reports, online directories, etc.) in order to address each performance indicator.  All aggregated data was validated by the program prior to analysis.

Key Informant Interviews and Survey Responses

One possible limitation for the evaluation is that staff interviewees and survey respondents (SEP employees and stakeholders) may overly emphasize the positive aspects of the program. The evaluation mitigated the potential for self-interest bias by structuring interview and survey questions so that respondents were not always commenting on areas where they have a vested self-interest. In addition, in order to mitigate the potential for bias, comments and findings were balanced with data from the other lines of evidence obtained during the evaluation.

Timing of Surveys

Both employee and stakeholder surveys were conducted between the months of July and September, a period where many employees are away on annual vacation and when stakeholders are busy working in the field.  In order to ensure as high a response rate as possible, a notification e-mail was sent to all potential respondents (employees and stakeholders) in June prior to the survey launch; and during the implementation, a minimum of two reminders were also sent.  In addition, the survey was left open for several weeks, allowing enough time for respondents to participate.

Moreover, the SEP employee survey had a response rate of 42.9% with a confidence level of 95% and a margin of error of +/- 7.9% (19 times out of 20). The stakeholder survey had a response rate of 26.5% with a confidence level of 95% and a margin of error of +/- 8.6% (19 times out of 20).4

Site visits

Considering the large geographical area, the evaluators could not visit all five areas in the Pacific Region. The selection of areas was based on the possibility to visit all different aspects of SEP activities. The two areas visited were Lower Fraser and South Coast, in addition to Regional Headquarters in Vancouver. The projects visited that were funded by PSF’s CSP were located on the Lower Mainland Area of Vancouver. To ensure the best coverage possible, areas that were not visited were compensated by in-depth telephone interviews with key personnel.

Attribution challenges

The achievement of SEP’s results can be impacted by a number of external factors such as extreme weather conditions, global warming, water levels, etc., making attribution of causality challenging. In addition, attribution challenges were encountered as other sectors are also involved and have key roles in SEP activities. To mitigate this issue, the evaluation collected data from different sources and strived to clearly articulate program stakeholder’s perspectives, their involvement as well as external factors affecting the program’s achievement of outcomes.


4 http://fluidsurveys.com/survey-sample-size-calculator/

4. MAJOR FINDINGS


4.1 RELEVANCE

Is there a continued need for SEP and its contribution agreements?

Finding: The evaluation found that there is a continued need for the Salmonid Enhancement Program due to the Pacific Salmon Treaty and the Wild Salmon Policy.  The program contributes to British Columbia’s economic activity.  Funding from the Federal Government and other organizations in support of Pacific Salmon stocks is widespread in British Columbia, however, the activities carried out by these other programs were qualified as complementary rather than duplicative.


Management of Pacific salmon has long been a matter of concern to Canada. SEP was originally introduced by the federal government in 1977 to address the rapid decline in Pacific salmon stocks.  The original long-term objective of the program was to double Canada’s Pacific Coast salmonid stocks and thereby provide economic benefit.  While the objectives and the needs of the program have evolved over time, the need to conserve and manage Pacific salmon stocks remains, as many individual salmon runs are still threatened.5

Pacific salmon stocks are of diverse and important economic, social and cultural value, yet salmon resources continue to be threatened by human incursions and natural events.  The need for salmon stock and habitat conservation, restoration and enhancement will never be completely eliminated.6

In 2010, it was determined that SEP addressed an economic need by contributing approximately $89 million to British Columbia’s gross domestic product (GDP), which included $25 million in direct economic activity, $52 million in supplier industries (indirect effects) and $11 million in industries benefitting from the spending of workers. SEP also generated direct employment of 592 jobs.  Another 872 jobs were generated by the activities of supplier industries (including those providing services to recreational anglers), while 129 jobs were created in industries benefitting from spending by workers.7  In 2012, it was estimated that the program contributed approximately $90 million of direct and indirect benefits.8

The majority of survey respondents (SEP employees and stakeholders) consider that SEP is still meeting the needs of Canadians from a moderate extent to a great extent.  All key informants interviewed agree that there is a continuing need for the program.  Key informants identified three major types of needs that are addressed by the program: economic, social/cultural, and ecological.  It is particularly needed to support economic prosperity, through its contribution to job creation and fishing revenues, to help preserve salmon’s iconic status in the Province of BC, particularly as part of First Nations’ food, social and ceremonial purposes, and to promote the sustainability of the resource through healthy rebuilding of salmon stocks and habitats.  Within DFO, SEP is said to provide needed information in support of fisheries harvest management decisions.


5 http://www.pac.dfo-mpo.gc.ca/sep-pmvs/about-sujet-eng.html

6 PSF’s CSP and TBSEF Contribution Agreement Renewal documents

7 SEP Economic Contribution, Counterpoint Consulting, 2010

8 2012 BC Statistical information from Ministry of Citizen Services. http://www.pac.dfo-mpo.gc.ca/sep-pmvs/anniv-eng.html

Pacific Salmon Treaty

Furthermore, the Pacific Salmon Treaty (PST) between Canada and the United States (US), which was renewed in 2008, also emphasizes the ongoing need for SEP.  SEP addresses Canada’s PST obligations by enhancing salmon to directly address harvest objectives by: marking fish at hatcheries and spawning channels with fin clips and/or coded-wire nose tags to collect salmon stock assessment data; providing information for salmon abundance forecasting and in-season fishery management; and, providing fishery data.  This is critical information for the domestic and international management of Pacific salmon. The purpose of the Treaty is to ensure the conservation and harvest sharing of Pacific salmon and commits both parties to a carefully planned coordinated joint enhancement program for transboundary rivers.  The treaty also ties Canada and the US to certain principles, including the implementation of an enhancement program consistent with the protection of existing wild salmon stocks, the habitat upon which they depend and the use of a variety of approaches to increasing population.  This Treaty and its principles have major implications for DFO, in particular for salmon enhancement activities where SEP is the only means for Canada to honour its commitments.

Wild Salmon Policy

The need for SEP is also highlighted in DFO’s Wild Salmon Policy9, which came into effect in 2005.  The policy describes how DFO will meet its responsibilities for the conservation of wild Pacific salmon.  It stipulates an overall policy goal for wild salmon, identifies basic principles to guide resources management decision-making, sets out objectives and strategies to achieve the goal and defines SEP’s roles and responsibilities for enhancement and other activities.

Other federal government and organizations supporting Pacific salmon

It was noted in the 2009 evaluation of SEP that the program’s activities of habitat restoration, community capacity development and public stewardship shared similar objectives to other DFO programs and contribution agreements; however, their activities were qualified as complementary rather than duplicative.  The programs and recipients do not share the same activities or the same project selection criteria, which vary between programs. For example, the SEP and the Habitat Stewardship Program (HSP) share similar overall objectives but have different criteria that distinguish them. SEP projects are involved in salmon enhancement activities, which include projects of habitat restoration, community capacity development, and public stewardship and fish production. HSP projects consider only Species at Risk Act listed species and fund projects related to these species for habitat restoration, community capacity development and public stewardship.

Funding from the Federal Government and from other organizations in support of Pacific Salmon stocks is widespread in British Columbia. The current evaluation identified a non-exhaustive list of 15 Federal Government programs and/or external organizations that also work towards protecting, restoring and/or enhancing Pacific salmon.10 In the employee survey, all of these programs were perceived as complementary to SEP by the majority of SEP staff.  For example, 57% of employees perceived the activities carried out by DFO’s contribution to PSF’s CSP as complimentary to those of SEP, while 20% of employees believed that they were similar and 22% believed they were duplicating activities of SEP.

While a few initiatives were identified by key informants as replicating some components of the program (e.g., smaller community hatcheries, community involvement activities, and First Nations-focused activities), all key informants agreed that SEP is unique in terms of its range of activities and objectives.


9 http://www.pac.dfo-mpo.gc.ca/fm-gp/species-especes/salmon-saumon/wsp-pss/docs/wsp-pss-eng.pdf

10 The 15 organizations identified in the surveys are: Species at Risk Program, Aboriginal Fisheries Strategy, Pacific Integrated Commercial Fisheries Initiative, Aboriginal Aquatic Resources and Ocean Management Program, DFO’s contribution to PSF’s CSP, DFO’s contribution to TBSEF, Recreational Fisheries Conservation Partnerships Program, Habitat Stewardship Program, Pacific Salmon Foundation, Fraser Basin Council, Pacific Salmon Commission, the Northern Boundary and Transboundary Rivers Restoration Enhancement Fund (Northern Fund), the Southern Boundary and Transboundary Rivers Restoration Enhancement Fund (Southern Fund), Salmon Enhancement and Habitat Advisory Board, and Pacific Streamkeepers Federation.

Is SEP aligned with Government of Canada and DFO priorities? Are the PSF's CSP and TBSEF contribution agreements aligned with SEP priorities?

Finding: Evidence supports SEP’s contribution to Government of Canada and DFO priorities through the Budget, Speech from the Throne, the Economic Action Plan, and alignment to two departmental strategic outcomes.  In addition, the evaluation found that activities undertaken as part of the contribution agreement with PSF’s CSP share similar objectives to SEP. TBSEF activities ultimately lead to an improved marine ecosystem, thus supporting healthy salmon stocks; however linkages between TBSEF and SEP are weak.


Activities undertaken by SEP are aligned with Government of Canada priorities as demonstrated through the Budget, Speech from the Throne and Economic Action Plan.  For example, the government of Canada indicated in the 2013 Federal Budget that it remains committed to developing policies in support of competitiveness of the fisheries sector, while promoting sustainable management now and over the long term.  The 2013 Speech from the Throne indicated that the Government of Canada will continue to support fishermen by ensuring proper management of fish stocks and by opening markets worldwide. In 2009, SEP was provided $8 million under Canada’s Economic Action Plan to make infrastructure improvements at SEP major facilities. $5.4 million was directed to 19 salmon hatcheries and spawning channels to refurbish their water supply and delivery systems, a critical component for hatchery operations.  More recently, in October 2014, the Federal Government announced an investment of $34.2 million over a five year period in order to upgrade and renew salmon hatcheries and spawning channels operated by SEP.

SEP’s activities also contribute to the Government of Canada outcome areas of Clean and Healthy Environment and Strong Economic Growth.  The program also directly supports DFO’s strategic outcome of Economically Prosperous Maritime Sectors and Fisheries by producing salmon directly for harvest as well as enhancing stocks of concern allowing commercial fisheries to proceed. Previous studies determined that the program contributes approximately $89 million to British Columbia’s gross domestic product and generate direct employment of 592 jobs.

SEP also supports DFO’s strategic outcome of Sustainable Aquatic Ecosystems by contributing directly to rebuilding salmon populations through fish culture and habitat restoration.

SEP salmon production is aligned with DFO priorities by directly supporting commercial, recreational and aboriginal access to harvest opportunities as well as stock assessment and the conservation and rebuilding of at risk wild stocks.

CEDP facilities originally had socio-economic objectives beyond fish production, primarily aimed at regional employment and capacity building in aboriginal communities.  The original social objective of the program has largely been met and SEP no longer has an outcome aimed at the delivery of social development programming. The objectives of CEDPs now focus on fish production, restoring habitat, salmon stock assessment, public awareness and education.

Pacific Salmon Foundation’s Community Salmon Program contribution agreement

The activities undertaken as part of the contribution agreement with the PSF’s CSP share similar objectives to SEP.  The activities are selected in order to advance progress towards protecting and enhancing salmon and habitat.

T. Buck Suzuki Environmental Foundation contribution agreement

A key focus of TBSEF projects has been habitat protection through pollution prevention.  In the past five years, the foundation has undertaken awareness campaigns regarding sewage treatment, urban pollution prevention and vessel pollution prevention.  Although these activities ultimately lead to an improved marine ecosystem, thus supporting healthy salmon stocks, the evaluation found that the linkage between TBSEF and SEP is weak.

For example, over the past five years, the foundation has focused on the OceanSMART Green Boating and Fuel Initiative which promotes fuel efficiency improvements on fishing vessels to reduce carbon footprint and CO2 emissions.  The foundation has also initiated green harbour projects which, for example, has helped replace wood and creosote pilings with steel and concrete alternatives.  The project in Cowichan Bay will also help increase moorage space for commercial vessels.

Is SEP aligned with federal roles and responsibilities?

Finding: Legislation governing SEP activities include the Fisheries Act, the Oceans Act, the Species at Risk Act and the Pacific Aquaculture Regulations and are therefore aligned with federal roles and responsibilities.


Through targeted enhancement efforts on key stocks, SEP actively contributes directly to Canada’s ability to meet its obligations under the Pacific Salmon Treaty and supports secure international market access objectives for Canadian salmon products.

SEP continues to focus on the production of Pacific salmon in British Columbia and the Yukon to support vulnerable stocks and provide harvest opportunities for all harvest sectors (commercial, recreational and First Nations) under the Fisheries Act.  The program also focuses on the participation of First Nations, local communities and external parties in cooperative fisheries and watershed stewardship activities under the Oceans Act. SEP activities are also authorized under the Pacific Aquaculture Regulations.

An agreement between the province of British Columbia and the federal government was signed in 1979 and renewed in 1984 authorizing the cooperative implementation of a salmonid enhancement plan. Both agreements define salmonid as “the five species of Pacific salmon, and anadromous trout”. Because Steelhead and Cutthroat trout are anadromous species, they are included in this agreement.

Both the federal government and provincial officials (Province of British Columbia) make decisions with respect to the management of the Steelhead and Cutthroat trout species. The responsibility depends on whether it concerns commercial or sport fishing, whether it concerns fishing in tidal or non-tidal waters and what is being varied (close time, quota, limits on size and weight). The province of BC has undertaken the responsibility of the administration of freshwater sport fisheries. The sport fishing of these species in non-tidal waters is  managed by the province of BC and a provincial sport fishing licence is required to fish Steelhead and Cutthroat species.

4.2 EFFECTIVENESS

To what extent has SEP achieved its three immediate outcomes?

Finding: SEP’s enhanced salmon provide harvest opportunities and is successful in engaging stewards to participate in cooperative fisheries and stewardship activities. Although SEP is working towards supporting vulnerable salmon populations, the difficulties to demonstrate change as well as attribution challenges have impacted the evaluation’s capacity to fully assess the program’s success.


Immediate outcome 1: Vulnerable salmon populations are supported

SEP’s enhancement objectives are set annually in the Integrated Fisheries Management Plans (IFMPs) with specific production targets that are to be achieved.

In 2012-13 and 2013-14, SEP achieved its target for the conservation and rebuilding of vulnerable stocks and is therefore working towards supporting vulnerable salmon populations through priority setting. Table 2 provides an overview of SEP’s achievement of its enhancement objective related to conservation and rebuilding of vulnerable stocks.

Table 2. Percentage of enhancement facility production groups in the IFMPs where the objective of enhancement is conservation and rebuilding of vulnerable stocks

Table 2
Year Target Actual results
2009-10 Not available Not available
2010-11 Not available Not available
2011-12 Not available 15.1%
2012-13 33% (by March 31, 2013) 35.3%
2013-14 25%11 (by March 31, 2014) 33.5%12

Source: Production plans. Data not available in earlier years as SEP only began to report in the Departmental Performance Report in 2012-13.

Responses from surveys and interviews are varied when asked to what extent “Vulnerable salmon populations are supported”. Key informant interviewees believe that this outcome is achieved while a greater proportion of employee survey respondents believe that SEP could improve on this outcome. Over the last two years, SEP has adjusted its target related to the IFMPs objective of conservation and rebuilding of vulnerable stocks from 33% to 25%.
 
On its own, the information presented in Table 2 does not demonstrate progress or change. The change in status of some salmon species listed as vulnerable species would help demonstrate the achievement of this outcome. In order to clearly conclude on the achievement of this outcome, SEP would need to have performance information related to the change in status. In addition, attribution of causality is challenging in this case as other DFO groups, like Science and Resource Management are also involved in the identification of priority stocks as well as in the consultation process with stakeholders. For example, Science is responsible for salmon stock assessment while SEP hatcheries’ role in stock assessment is to tag enhanced salmon in major hatcheries.

Although SEP is working towards supporting vulnerable salmon populations by ensuring the genetic diversity of some vulnerable salmon species, by ensuring geographic diversity of salmon species throughout British Columbia, etc., the evaluation team was unable to assess SEP’s achievement as performance information available was insufficient to conclude that vulnerable species are supported.


11 The methodology used by the program to report on production data through annual Departmental Performance Reports changed from year to year, making comparisons difficult.

12 As per the program, the change to the target was undertaken to address the unintended consequences of using the previous targets of 33% and 67% for respective reporting on rebuilding of vulnerable stocks and harvest opportunities. With cumulative targets of 33% and 67%, any reporting outcome that is not exactly 33% and 67% necessarily results in one of the targets being “unmet”. This unintended consequence was addressed by changing the targets to 25% and 50% respectively. This change still allows the program to report on 100% of production, but does not create artificial “unmet” performance results.

Immediate outcome 2: Enhanced salmon provide harvest opportunities

As described in Table 3 and Table 4, SEP enhanced and released close to 1.8 billion salmon of different species (Chinook, Chum, Coho, Pink, Sockeye, Steelhead and Cutthroat trout) between 2008 and 2012. Sockeye is the species most released with one billion released over five years, and is enhanced almost exclusively in SEP’s major facilities. Moreover, almost all (91.7%) enhanced salmon are released by major facilities.

Table 3. Percentage of total release of enhanced salmon by type of facilities between 2008 and 2012

Table 3
Types of facilities Number of facilities Number of salmon released Percentage of  total release
Major facilities 16 1,672,790,000 91.7%
CEDPs 18 83,114,000 4.6%
DPIs* 25 51,463,000 3.3%
PIPs 57 15,449,000 0.9%

Source: Production plans
* Designated Public Involvement Facilities (DPIs)

There are also 53 different schools that released 260,000 Chinook, Chum or Coho between 2008 and 2012.

Table 4. Percentage of total release of enhanced salmon by facilities and by species between 2008 and 2012

Table 4
  Major
facilities
%
CEDPs
%
DPIs
%
PIPs
%
Chinook 10 28 27 13
Chum 23 37 15 33
Coho 3 10 11 36
Pink 3 21 45 18
Sockeye 61 3 2 -
Steelhead / Cutthroat
trout
0.1 0.5 0.1 -
Total 100 100 100 100

Source: Production plans

Over the period evaluated, the program has undergone changes in its operating environment. Starting in 2010, the program began to develop and implement a more structured production planning process which was also influenced by the implementation of the Pacific Aquaculture Regulations (PAR).

Prior to 2012, production lines did not have an explicit enhancement objective. This changed in 2012 when a more rigorous and structured decision-making process was implemented leading to the creation of specific enhancement objectives for each line of production. DFO major facilities, CEDPs, and DPIs are now integrated into fisheries decision-making with development of regional IFMPs.

Following the implementation of the PAR, the use of target release changed as they became recommended maximums. Production targets pre-PAR were weak guidelines and most facilities attempted to at least achieve their target and exceeding this target in many cases was seen as a success. Post-PAR, the targets became recommended maximums and the program had to manage changes in its operating practices to avoid exceeding the maximum release. This was a fundamental shift in thinking and the program is still making progress.

Sometimes the incubation and rearing happens in a major facility but eggs or fry are transferred to a CEDP or PIP that then get the credit of the actual release. This practice is to give credit to the volunteers that released the fish. Therefore target release (where target release is now maximum release) vs. actual release can vary depending on which facility get the credit for production. There are also many other factors that can influence target release vs. actual release, such as poor return of some salmon species in some years, water levels, etc.

The review of SEP data demonstrated that in recent years, the majority of facilities are no longer exceeding the maximum release. In 2010, 33% of all major facilities, including spawning channels, exceeded the maximum target (previously called target release) and in 2012, only 4% exceeded the maximum target.

Table 5 provides the average maximum target for each major facility from 2008 to 2012. On average, SEP major facilities released 68% of their maximum target. This suggests that some hatcheries are not functioning to full capacity and still have room for additional production if needed.

Table 5. Average percentage of maximum target release for all major facilities from 2008-2012 for all salmon species except Steelhead and Cutthroat trout

Table 5
Major Facilities Average % Major Facilities Average %
Big Qualicum River 37 Nadina 160
Capilano River 107 Nitinat River 52
Chehalis River 83 Pinkut Creek 71
Chilliwack River 101 Puntledge River 91
Conuma River 81 Quinsam River 95
Fulton River 86 Robertson Creek 104
Gates Spawning Channel 69 Rosewall Creek 141
Horsefly Spawning Channel 20 Shuswap River 85
Inch Creek 102 Snootli Creek 92
Inch Sockeye Satellite 94 Spius Creek 111
Kitimat River 82 Tenderfoot Creek 91
Little Qualicum River 54 Weaver Spawning Channel 53
Total average:  68%

Source: Production plans

SEP is  a means for Canada to honour its commitments related to the Pacific Salmon Treaty (PST). Through the enhancement of salmon, SEP tags salmon that have been identified as a priority to meet PST obligations. In fact, there are 11 facilities involved in the production of Chinook and six facilities producing Coho related to PST. On average, from 2008 to 2012, 11% of the total number of Chinook and 6% of the total number of Coho released were tagged for PST obligations. The number of Chinook tagged has been stable in the last five years but the quantity of Coho tagged has increased over the years.

Although the 2011-12 Departmental Performance Report mentioned that SEP had not met its targets of enhanced contribution to catch, the access to harvest was not limited. In fact, in 2012-13, SEP slightly missed its target entitled “enhancement facility production groups in IFMPs where the objective of enhancement was harvest or stock assessment”. The target in 2012-13 was 67% and the actual result was 64.7% (Table 6). Over the last two years, SEP has adjusted its target related to the IFMP objective of harvest or stock assessment from 67% to 50%. In 2013-14, the actual target was achieved with 51.7%.

Table 6. Percentage of enhancement facility production groups in Integrated Fisheries Management Plans (IFMP) where the objective of enhancement is harvest or stock assessment

Table 6
Year Target Actual results
2009-10 Not available Not available
2010-11 Not available Not available
2011-12 Not available 17.4%
2012-13 67% by March 31, 2013  64.7%
2013-14 50% by March 31, 201413  51.7%14

Source: Production plans

SEP has demonstrated that enhanced salmon provides harvest opportunities. The average contribution of enhanced salmon catch from 2008 to 2012 was 28%15. This is characterized in Table 7 which demonstrates the harvest of enhanced salmon compared to total catch (enhanced + wild) from 2008 to 2012.


13 The methodology used by the program to report on production data through annual Departmental Performance Reports changed from year to year, making comparisons difficult.

14 See footnote #12

15 SEP Contribution to Harvest (provided by program).

Table 7. Harvest of enhanced Pacific salmon contribution to total catch

Table 2

Source: Production plans

The majority of respondents in both surveys were of the opinion that CEDPs are successful in delivering their key activities of fish culture, project operations and public stewardship which contribute to the immediate outcome of “Enhanced Salmon provide harvest opportunities”. However, the other two CEDP activities of habitat conservation and stock assessment are perceived by SEP employees as well as stakeholders as not being achieved.

SEP employees, stakeholders and key informants interviewees all consider that the program has achieved its outcome of enhancing salmon that provide harvest opportunities at least to a moderate extent. One particular limitation mentioned is the limited effectiveness of enhancements when compared to habitat restoration to ensure long term increase of salmon stocks.

The evaluation found that SEP has been successful in enhancing salmon stocks and has demonstrated that, through enhanced salmon, SEP contributes to harvest opportunities.

Immediate outcome 3: First Nation, local communities and external parties participate in cooperative fisheries and stewardship activities

SEP engages First Nations, local communities and external parties through major facilities, CEDPs, PIPs, Stream to Sea Program, etc. which leads these community groups to participate in cooperative fisheries and stewardship activities. In addition, SEP through its contribution agreement with PSF’s CSP, encourages community groups (ultimate recipients) to participate in cooperative fisheries and stewardship activities.

For example, some of SEP’s major facilities contribute to the public education and awareness activities of this immediate outcome by providing eyed eggs for classroom incubators, office spaces, visitor centres, fish culture supplies and equipment, and expert advice to people involved in stewardship and education initiatives. Specifically at the Quinsam River Hatchery “an estimated 10,000 people visit and follow self-directed tours of the hatchery annually, another 1,500 people participate in interactive programs under the direction of hatchery staff”16.

As part of the public education and awareness activities, SEP’s Stream to Sea program has approximately 826 schools with 900 incubators used annually. In addition, there are between 325 and 350 groups participating in Community Involvement as well as approximately 10,000 volunteers participating in salmon enhancement projects annually. The availability of the Stream to Sea program data was challenging for the evaluation as the collection of data was not consistent across all Areas. To mitigate this issue, the evaluation team collected data from various sources (program documents, PSF’s CSP annual reports, online directories, etc.) in order to address performance indicators.  All aggregated data was validated by the program prior to analysis.

Through site visits, it was possible to see evidence of how CEDPs and PIPs help community groups participate in cooperative fisheries and stewardship activities. For instance, when visiting CEDPs it was possible for evaluators to understand that some CEDPs not only contribute to the enhancement of salmon, but to community involvement by having groups of students learn about salmon in their establishment. The evaluators also visited local PIPs that reared salmon with the goal of releasing them with the help of community stewards. By participating in the rearing process as well as releasing the salmon in the stream, communities become more knowledgeable about salmon. This was also supported by media articles that raise the positive implications of local hatcheries in communities. Although there are activities delivered in CEDPs and PIPs, SEP does not document how the results in both types of facilities contribute to the program.

The opinion of SEP employees concerning the achievement of this immediate outcome is diverse: 42% considered that the outcome was achieved to a limited or moderate extent while 53% considered that it was achieved to a considerable or great extent. Most key informant interviewees considered that SEP has achieved this immediate outcome to a considerable or great extent.

PSF’s CSP has reported that on average, volunteers contribute 23,000 hours of work per year. However, in 2013-14 the number of hours decreased substantially to just over 10,000 hours. Volunteer hours are spread across different project types such as habitat rehabilitation, stock assessment, stock enhancement, habitat assessment steward training, community stewardship and watershed planning, and public education and volunteer steward training.

There are diverging views regarding the achievement of the PSF’s CSP outcomes established in their contribution agreement.  In fact, SEP employees are of the opinion that they are not meeting their outcomes, while stakeholders disagree. There is one exception where both SEP employees and stakeholders agree that PSF’s CSP has not achieved its outcome of “First Nations, local communities and external parties participate in cooperative fisheries and watershed stewardship activities”. The opinion of survey respondents is summarized in Table 8.


16 Quinsam Review, 20 Jan 14. p.4

Table 8. Survey respondents’ opinion regarding the achievement of the PSF’s CSP outcomes

Table 8
Outcome SEP employee survey Stakeholder survey
First Nations, local communities and external parties participate in cooperative fisheries and watershed stewardship activities 71% not achieved 43% not achieved
Increased collaboration among community stakeholders in salmon enhancement, habitat conservation and restoration projects 61% not achieved 58% achieved
Public Awareness about salmon in Pacific Region is increased 53% not achieved 69% achieved

Source: Employee and Stakeholder surveys

TBSEF has contributed to the outcome of First Nation, local communities and external parties participate in cooperative fisheries and stewardship activities by accomplishing projects, such as OceanSMART Green Boating Guide, Green harbour planning and infrastructure projects, public awareness regarding pesticides and Capital Region District Liquid Water Management Plan.

Overall, SEP has been successful in engaging external groups, organizations and volunteers to participate in cooperative fisheries and stewardship activities. However, there are diverging opinions on PSF’s CSP achievement of its outcomes established in the contribution agreement.

To what extent has SEP achieved its two intermediate outcomes?

Finding: SEP has helped support the conservation, stewardship and rebuilding of salmon and their habitat through projects and activities delivered by the program and other organizations. Ensuring the public has access to harvest opportunities is beyond SEP’s control and was not assessed in this evaluation. Activities related to habitat restoration support salmon enhancement, however the evaluation found no clear strategy or vision within SEP related to the rebuilding of salmon habitat.


Intermediate outcome 1: The public has access to harvest opportunities

Recreational fishers wanting to fish salmon are required to purchase a license and stamp on an annual basis. Prior to 2013, only a portion of the fees collected through the recreational stamps were provided to PSF’s CSP. As of 2013, the fees collected are entirely provided to PSF’s CSP. Since 2008-09, the sales of recreational stamps have been consistent from year to year; on average, 213,452 stamps are sold annually allowing recreational fishers to catch salmon.

On the other hand, the number of salmon commercial licenses has decreased in the last six years from 2,066 licences to 1,752 licences.

Since 2008, DFO issued a total of 1,397 Food, Social and Ceremonial licences which are not exclusive to salmon. From 2011 to 2014, DFO issued 46 Excess to Spawning Requirements licences that are exclusive to salmon.

A majority of respondents in both surveys, as well as key informants, were of the opinion that this intermediate outcome is being achieved by SEP. Although survey and interviewee respondents were of the opinion that SEP is achieving this outcome, the evaluation did not assess the achievement of this outcome because it is addressing the accessibility that the public has to harvest opportunities, which is beyond SEP’s control. Decisions regarding harvest access are determined by DFO’s Fisheries Management. For example, it is Fisheries Management that issues the commercial and recreational licences as well as determining the opening and closing dates of fisheries. However, these decisions are largely based on the abundance of stocks, which is impacted by external factors such as extreme weather conditions, global warning, water levels, etc.

Intermediate outcome 2: The public supports the conservation, stewardship and rebuilding of salmon and their habitat

SEP engages different community groups with the goal of supporting the conservation, stewardship and rebuilding of salmon and their habitat. Specifically, through SEHAB, Pacific Streamkeepers Federation, as well as its contribution agreement with PSF’s CSP, SEP has helped the public support the conservation, stewardship and rebuilding of salmon and their habitat.

The Salmon Enhancement and Habitat Advisory Board met between one to four times per year since 2010-11.  Some records of decisions were produced for some meetings. According to the document review, in 2008-09 and in 2009-10 there was no meeting minutes.  In 2010-11 minutes were provided for one meeting and from 2011-12 to     2013-14 meeting minutes were provided for all meetings.  As described in SEHAB’s Terms of Reference, SEHAB is required to provide an annual advisory report to the responsible Regional Director, Ecosystems Management Branch.17 The evaluation team was unable to consult six annual advisory reports as only one report was produced in 2013, and did not feature advice offered by SEHAB to SEP. Nevertheless, a SEP manager did note an example demonstrating SEHAB support in the conservation, stewardship and rebuilding of salmon and their habitat. Specifically, SEHAB participated in workshops held by DFO prior to the implementation of Pacific Aquaculture Regulations and identified issues, challenges and options for addressing PAR changes for volunteer groups. As a result, SEHAB’s advice helped the understanding of insurance implications of licensing decisions for community organizations.

The Pacific Streamkeepers Federation offers a handbook and modules on protecting and restoring aquatic habitat, which allows individuals or community groups to support the conservation, stewardship and rebuilding of salmon and their habitat. Better documentation of this organisation’s contribution to SEP’s outcome would be beneficial to SEP.

SEP and its contribution agreement with PSF’s CSP, has restored close to 1.5 million m2 of habitat between 2008-09 to 2013-14 (Table 9) of which 92% was done by SEP. PSF’s CFP contribution represents only 8% of total habitat restored over the last five years. Although SEP and its contribution agreement are supporting salmon enhancement through habitat restoration, the evaluation found that SEP has no strategy or vision related to habitat restoration. Many comments provided in both surveys stress the importance of habitat restoration over fish production as a better means to achieve sustainability of resources in the longer term. Some employees are of the opinion that there is limited focus on habitat restoration and projects accomplished are not based on a strategic approach. In SEP’s logic model, there is no clear linkage demonstrating how habitat restoration activities contribute to an immediate outcome. In addition, habitat restoration is not defined as a specific expected result.

The majority of respondents in both surveys were of the opinion that PSF’s CSP was successful in improving salmon habitat. However, with the exception of 2012-13, the m2 of habitat restored by PSF’s CSP has consistently diminished. PSF’s CSP has no identified targets to reach every year as demonstrated in Table 9.


17 Salmon Enhancement and Habitat Advisory Board, Terms of Reference. p.5

Table 9. Habitat Restored by SEP and PSF’s CSP in Pacific Region 2008-09 to 2013-14

Table 9
Year Target
(# Projects)
Actual
(# Projects)
Target
(m2 habitat restored)
Actual (m2 habitat restored)
SEP:
2008-09 60 11 160,000 163,450
2009-10 60 57 160,000 87,712
2010-11 60 49 160,000 198,000
2011-12 60 25 160,000 79,320
2012-13 60 65 160,000 330,751
2013-14 60 49 160,000 516,731
PSF’s CSP:
2009-10       49,636
2010-11       *18,858
2011-12       15,047
2012-13       24,055
2013-14       **13,998
Grand total 1,497,558

Source: Figures provided by SEP and PSF.
*: In 2010-11, PSF participated in the in-stream fertilization project that covered 495,000 m2. However, because of the limited one year benefit to the habitat, this project did not qualify as habitat restored.
**: Figures for 2013-14 are not final.

In addition to habitat restoration, PSF’s CSP has contributed to this outcome through stock enhancement and stock assessment. PSF’s CSP volunteers have contributed more than 44,000 hours from 2009-10 to 2013-14 on stock enhancement and stock assessment projects. Key informants as well as both survey respondents agreed that through SEP the public is supporting the conservation, stewardship and rebuilding of salmon and their habitat.

Through the Pacific Streamkeepers Federation and the contribution agreement with PSF’s CSP, SEP has helped the public in supporting the conservation, stewardship and rebuilding of salmon and their habitat. Habitat restoration, stock assessment and stock enhancement activities were also undertaken and contributed to this outcome.  Improvement of the data collection on how SEP’s different components contribute to achieving this program’s outcome would be beneficial to SEP.

To what extent has SEP achieved its long-term outcome?

Finding: Overall, some evidence was available to suggest that SEP is making progress towards supporting healthy and diverse salmon populations by producing different salmon species, ensuring the genetic diversity of some vulnerable species, and by ensuring geographic diversity of salmon species throughout the Pacific region.  An economic analysis conducted in 2010 also demonstrates that the program contributes $89 million annually to British Columbia’s GDP.  However, there was no evidence to demonstrate how the program is contributing to the achievement of sustainable marine and freshwater ecosystems.


Long-term outcome: Healthy and diverse salmon populations and sustainable marine and freshwater ecosystems provide harvest opportunities and support a stronger Canadian economy

As previously mentioned, the evaluation found that the program has been working towards supporting healthy and diverse salmon stocks by producing different salmon species, ensuring the genetic diversity of some vulnerable species, and by ensuring geographic diversity of salmon species throughout the Pacific region. However, the evaluation would have expected that the program collect more extensive data in order to fully determine the extent to which vulnerable salmon species are supported, such as changes in the status for salmon listed as vulnerable species. The evaluation also found no evidence to demonstrate how the program is contributing to the achievement of sustainable marine and freshwater ecosystems. It is also important to note that this element is not reflected as an immediate or intermediate outcome in the program’s logic model, thus making it difficult to measure as a long-term outcome.

As for SEP’s impact to the economy, an economic analysis states that the program contributes approximately $89 million annually to British Columbia’s GDP. 18 In addition, most key informants and many survey respondents (51% of stakeholder survey respondents and 63% of employee survey respondents) consider that the program has achieved to a considerable or great extent its ultimate outcome of having healthy and diverse salmon populations and sustainable marine and freshwater ecosystems that provide harvest opportunities and support a stronger Canadian economy.  In addition, a few stakeholder survey respondents (17%) were of the opinion that the program was achieving its ultimate outcome only to a limited or very limited extent.

Overall, meaningful performance data was not available in order to clearly demonstrate the achievement of some immediate and intermediate outcomes. For this reason, the evaluation is limited in its capacity to conclude on the achievement of SEP’s long-term outcome, although some lines of evidence provided insight into the achievement of the long-term outcome and suggest that the program is making progress towards achieving its long-term goals. Table 10 summarizes SEP’s achievement of outcomes.


18 SEP Economic Contribution, Counterpoint Consulting, 2010

Table 10. Overview of SEP’s achievement of outcomes

Table 10: Overview
Immediate outcomes:   Extent of achievement Attribution challenges19
#1 Vulnerable salmon population are supported To some extent Limited evidence that vulnerable species are supported. E.g. no change in status of some salmon species listed as vulnerable species.
#2 Enhanced salmon provide harvest opportunities Achieved  
#3 First Nation, local communities and external parties participate in cooperative fisheries and stewardship activities Achieved  
Intermediate
Outcomes:
  Extent of achievement Attribution challenges
#1 The public has access to harvest opportunities Not assessed as out of SEP’s control Licences were issued however, ensuring the public has access to harvest opportunities is beyond SEP’s control
#2 The public supports the conservation, stewardship and rebuilding of salmon and their habitat Achieved  
Long-term outcome:      
#1 Healthy and diverse salmon populations and sustainable marine and freshwater ecosystems provide harvest opportunities and support a stronger Canadian economy The long-term outcome covers different aspects that were assessed separately:    
 
a)
Healthy and diverse salmon populations provide harvest opportunities
To some extent Limited evidence that vulnerable species are supported.
 
b)
Sustainable marine and freshwater ecosystems
No evidence on the achievement of sustainable marine and freshwater ecosystems.  
 
c)
Support a stronger Canadian economy
To some extent An economic analysis conducted in 2010 demonstrates that the program contributed $89 million annually to British Columbia’s GDP.  However, the study was limited to British Columbia and did not cover the Yukon and a more recent version does not exist.

19 Attribution challenges were described in section 3.4 of the methodology.

Are there any external factors and/or challenges that may have impacted SEP's results? Did SEP have any unintended results?

Finding: SEP faces several challenges, such as an aging work force and infrastructure, IM/IT challenges, extreme weather events, and staffing difficulties. The program must also deal with high public interest, which can create resistance to change.  On another note, Real Property and PAR have positively impacted program results.

SEP habitat restoration projects have helped some species at risk which were considered unintended outcomes. As well, the program’s high visibility has contributed to building strong interest and advocacy which has also limited the program’s ability to make needed adjustments.


External Factors and Challenges

SEP faces several challenges such as an aging workforce and infrastructure, information management and information technology (IM/IT) challenges (e.g. CITRIX, internet connections), extreme weather events, limited knowledge dissemination from more experienced employees to new employees and staffing difficulties. Specifically, according to the employee survey, 80% of respondents were of the opinion that IM/IT had negatively impacted SEP. In addition, other internal services such as financial (83% of respondents) and human resources (72% of respondents) services had also negatively impacted the program in respondents’ opinion. Examples of challenges faced by SEP due to internal services are some invoices not being paid in a timely manner, poor internet access connections in isolated hatcheries, and delays in the staffing process.

On the contrary, Real Property, Safety and Security were perceived as having positively impacted SEP’s results. Real Property has been excellent and very instrumental in assisting with infrastructure upgrades for many facilities. Funding for infrastructure issues has been provided through bundling of funding requests.  For example, several requests to upgrade water related infrastructure at many different hatcheries are bundled into one funding application. Also, Real Property’s support to resolve health and safety issues at facilities was also perceived as positive.

Some key informants along with comments provided in the employee survey noted that insufficient information and scientific knowledge, combined with many external factors beyond DFO’s control, make it difficult for SEP to understand interactions between enhanced and wild stocks. Because it is a shared responsibility between SEP and Ecosystems and Oceans Science Sector, the evaluation considers that even more linkages between both groups would be beneficial as SEP is dependent on other groups’ expertise to advance some of its activities.

The implementation of the Pacific Aquaculture Regulations has positively impacted SEP, specifically, it was mentioned that PAR brought a higher level of precision in production plans while formalising them, as well as improving compliance with production targets. In addition, PAR brought coherence to the practise of fish culture for SEP community facilities, formalized reporting requirements and fish culture protocols, and allows for scheduled inspections. Moreover, PAR keeps SEP accountable and aligned with aquaculture industry standards as well as increases transparency.

Unintended outcomes

A positive unintended outcome was observed during site visits of habitat restoration projects where improved salmon habitat also unintendedly contributed to supporting other species at risk, such as the Salish Sucker.

It was mentioned during the site visits and by some key informant interviews that the program’s high visibility, which has contributed to building strong interest and advocacy is also limiting the program’s ability to make needed adjustments.

4.3 EFFICIENCY

Are there any best practices and/or lessons learned from SEP?

Finding: SEP is successful in developing new and innovative projects which lead to the sharing of best practices and lessons learned. However, communication of these lessons with external stakeholders could be improved.


In 2014, SEP developed an Infrastructure Strategy which is viewed as an innovative practice within the department and mentioned as a best practice by several key informants.  The purpose of this strategy is to ensure SEP infrastructure is directed at meeting program’s objectives and DFO’s strategic outcomes and has taken into account the department’s long-term needs. The Infrastructure Strategy has been used by management to guide decision making regarding program assets.

SEP has also been involved in the development of guidelines to be used by CEDPs and other community involvement programs in order to ensure compliance with PAR.  The purpose of these guides is to provide regionally consistent approaches to monitoring fish production activities at facilities licensed under PAR. Stakeholders expressed their satisfaction with the guide and mentioned that as a result, they are more vigilant with respect to fish health protocols.

Innovative efficiencies have also been implemented in some major hatcheries, such as hydro powered micro-generation, cooperation with local industry, use of geo-thermal heat sources, high efficiency lighting, etc. The evaluation team was able to observe firsthand the impact of some of these upgrades at the Quinsam Hatchery which underwent a major refurbishment from 2009 to 2013 that resulted in 31% less energy consumption.

Although the evaluation found examples of implementation of innovative projects through the document review, 43% of stakeholders felt that the program was only sharing best practices and lessons learned to a limited extent.  Therefore, SEP could improve on communicating and sharing lessons learned with the larger stakeholder community.

Is there a performance reporting process/system in place? If so, is it being used for decision-making?

Finding: The program is successful in collecting and using data with respect to fish production. However, SEP’s data collection methods regarding community involvement activities are inconsistent across Areas and vary from year to year. As such, improved documentation and data collection highlighting how the different components of the program (e.g. Stream to Sea program) contribute to SEP’s outcome would be beneficial in demonstrating the program’s achievements.


Fish Production data

SEP collects large amounts of data regarding fish production.  Examples of data collected are production site, species produced, target release, actual release, release stage, release site, etc. The program has two electronic software systems to store and collect performance data related to salmon production: ENPRO and EPAD. Of the staff members who use this software, the majority said the databases meet their information needs to a fair or great extent.

Community involvement data

According to a review of PIP contracts, there is no systematic method of collecting performance data.  In addition, the review noted that PIP application forms vary by Area, by community advisor and by fiscal year.  In addition, PIPs are not required to provide reports upon project completion in order to report on performance and achievement of objectives.  The review of contracts did demonstrate that some PIPs did provide completion reports.  However, the program confirmed that this is not a requirement outlined in the contract agreements and is not a common practice.

With respect to the Stream to Sea Program, data is also inconsistently collected and reported across all Areas.  Each Community Advisor collects slightly different data and does not report in a consistent way in the program directory.  In 2010-11, the program started using a survey method in order to collect data on the Salmonids in the Classroom program.  However, participation is voluntary, thus not representative of the entire program.

Overall, data supporting stewardship and educational components of the program are collected in an inconsistent manner.

Performance Measurement Strategy

In accordance with Treasury Board’s Policy on Evaluation, the purpose of the performance measurement strategy is to ensure there is performance data available to program managers in order to continuously monitor and assess the results of their program and to ensure that data is available and relevant to evaluations. This information is needed to assist in demonstrating to elected officials, senior management and other interested Canadians whether a program is achieving its intended outcomes in support of a departmental strategic outcome.20

The 2009 Evaluation of SEP recommended that “the program finalize its logic model and performance measurement strategy with relevant and realistic performance indicators for all levels of outputs and outcomes (immediate, intermediate and long-term) to reflect the program’s results chain.”  As a result, a revised logic model and finalized performance strategy were developed it 2010. It was observed during the evaluation that SEP staff are very knowledgeable about the program logic model.  The document is well known by staff and used frequently. The documents have also been updated as part of a department wide initiative on development and implementation of performance measurement strategies for all DFO programs.

Despite these efforts to create, maintain and implement an effective performance measurement strategy, it became evident in the course of the evaluation that the logic model requires further adjustments, as do most of the performance indicators. For example, the public education and awareness component of the program is not captured as a program outcome, but rather as an output, making it difficult to measure through performance indicators.  Nonetheless, 70% of staff survey respondents felt they had sufficient performance information to adequately fulfil their job responsibilities.


20 http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024&section=text

Performance Reporting

The program has been collecting performance data through several means and for different reporting purposes, such as the Departmental Performance Report (DPR), Report on Plans and Priorities (RPP), performance measurement strategy, etc. Specific SEP information was only available in 2011-12 and 2012-13 DPRs.  Before 2011-12, SEP was a sub-program of Fisheries Management and was not reported as a standalone program activity in the DPR. As indicated in the 2011-12 DPR, the program used the SEP infrastructure strategy to guide major capital planning, prioritization, and decision-making about infrastructure (e.g., ceasing operations at a small number of facilities with low production values and significant concerns about site health and safety).

Contribution agreements

PSF’s CSP is required to provide an annual report at the end of each fiscal year in order to report on the achievements of the program.  PSF has collected data on the indicators required by DFO through the contribution agreement.  Nevertheless, these indicators may require updating by DFO, as previously noted in the 2009 evaluation, in order to ensure better alignment with SEP objectives.  For example, PSF’s CSP has been reporting on the number of square meters of restored habitat, as requested by SEP through the contribution agreement.  However, this indicator has proven to be an ineffective measure of habitat restoration, as smaller projects with critical habitat components may have a much larger effect than bigger projects with greater square meter coverage.

The TBSEF is also required to provide an annual report at the end of each fiscal year outlining the achievements of the Foundation.  Yet, the annual reports do not report on the indicators that were established in the contribution agreement. Instead, the program reports annually on the planned results which were set out at the beginning of the fiscal year through the Foundation’s work plan. It is important to note however that DFO does provide input into the annual work plan and provides approval.

To what extent are the program's activities, structures and processes appropriate to support the achievement of results?

Finding: There are many aspects of the program’s structure and processes that can be improved, such as SEP governance, budget allocation between areas, clarification of roles and responsibilities, and SEP design, delivery and priorities.


Program’s Structures and Processes

This section covers a broad range of topics, such as governance, resources, roles and responsibilities, design and delivery and priorities.

Governance

During the timeframe covered by the scope of the evaluation, the program was under an Area matrix management model which means that the regional management team does not functionally manage the delivery of the program in the Areas. This structure creates challenges in uniformity among Areas and in delivering on priorities that differ from one Area to another. On the other hand, the Area matrix model provides an opportunity to respond to Area-specific needs. Some employees through the survey suggested that Area-based management may work for some DFO sectors, but is not the best model for SEP. In their opinion, SEP should move away from an Area matrix model to a centralized delivery and reporting model for the entire program. The view of the majority of employees (86%) is that SEP’s organizational structure from April 2008 to March 2014 is not functioning adequately.

Of the SEP employee survey respondents, 70% believe that the program’s governance structures (decision-making processes, committees, etc.) are also not functioning adequately. Some employees mentioned that leadership is lacking and there is a need for a global direction to be communicated to all employees with clear directions to ensure consistency in the delivery of the program. Moreover, it was mentioned that Regional Headquarters is disconnected from the Areas which is problematic and that communication is also an issue. Finally, there is little connection with other SEP staff around the province and each Area tends to work independently.

SEP has established governance structures in order to ensure alignment with other DFO Sectors. For example, other Sectors and Directorates involved and consulted in SEP priority setting are Ecosystems and Oceans Science, Ecosystems and Fisheries Management, Real Property, Safety and Security and Aboriginal Strategies and Governance Directorate. Sixty-five percent (65%) of respondents in the employee survey mentioned that SEP has been efficient in planning and integration with other programs within DFO to a moderate or considerable extent. However, employees have divergent views on the issue of communications between SEP and other DFO Sectors: 36% of employees believe communications with other Sectors is functioning to a limited extent, 31% believe it is functioning to a moderate extent and 31% believe it is functioning to a considerable extent.

SEP recognizes the need for greater integration with other DFO sectors. In fact, in the 2013-14 Strategic Direction of program elements of SEP, it is mentioned that the program will develop priorities linked directly to fisheries management and stock assessment priorities and provide examples that demonstrate how this will be done.

Resources

SEP’s total annual budget is approximately $26M of which $11.7M is dedicated to operations and maintenance (O&M). During the period reviewed, the budget was allocated every year across Areas and Regional Headquarters. Table 11 shows SEP’s O&M budget by Areas from 2009 to 2014.

Table 11. SEP Operations and maintenance budget by Areas from 2009 to 2014

Table 11

Source: Program data

SEP dedicates $4.7 million of its O&M budget to the Community Involvement Program, which is allocated as follows: $3.2M to CEDPs, $260K to technical support contracts, $400K to Education (Stream to Sea), $560K to Public Involvement Projects (PIPs) and $280K for individual travel, vehicle mileage and maintenance, training, equipment, etc.

Many (64%) of employee survey respondents mentioned that the budget allocation across areas is not functioning adequately. The budget allocation within areas is also seen as not functioning adequately in the employees’ opinion (48%).

Some employees questioned the appropriateness of how the budget is allocated between the areas and suggested that funding should be based on program’s priorities instead of allocating the O&M budget consistently by area each year (see Table 10).

Clarification of Roles and Responsibilities

As part of the implementation of the Pacific Region Asset Class Strategy, Real Property, Safety and Security and SEP Regional staff will establish a document that identifies the respective roles, responsibilities and accountability of each group for the management and maintenance of SEP infrastructure sites.

Within Areas, the activities of SEP related to community involvement are delivered under the Community Involvement Program by Community Advisors (CA) and by Resource Restoration Units (RRU) where the latter comprise mostly engineers and biologists. Each CA supports a number of community projects as well as CEDPs. The evaluation was unable to find evidence of clear roles and responsibilities for CAs. Therefore, the way in which CAs deliver the program varies from one CA to another. Comments provided in the employee survey suggested the need to provide priorities and limits to the work CAs are able to undertake in a meaningful way. It was expressed in the stakeholder survey that respondents greatly appreciate the support and expertise provided by CAs. However, some stakeholders suggested that technical expertise to enhancement facilities should be provided by SEP major facilities staff instead of CAs. It is important to note that technical expertise and other support is provided in many cases from major facilities to PIPs and CEDPs which is usually coordinated by local CAs.

Roles and responsibilities regarding the provision of technical expertise between DFO (SEP staff) and PSF’s CSP need to be clarified in the contribution agreement. PSF’s CSP does not undertake the delivery of projects itself but requests proposals from other organizations who become ultimate recipients. PSF’s CSP is dependent on DFO for the provision of technical expertise. Within SEP, Community Advisors are often dedicated to work with community groups and organizations who are often ultimate recipients of the contribution agreement with PSF’s CSP.

Community Advisors often help ultimate recipients prepare their proposals. Systematically, SEP employees are involved in the selection process by reviewing all potential projects and providing advice on the feasibility of each project as well as alignment with SEP priorities. Finally, some employees continue to provide advice and technical expertise during the implementation of the projects.

Findings from the evaluation clearly demonstrate that DFO (SEP employees) is involved in the selection process as well as in the delivery of PSF’s CSP funded projects.  The evaluation also revealed that PSF’s CSP and ultimate recipients are dependent on DFO’s technical expertise to accomplish the projects.

Many of the respondents from the SEP employee survey (72%) and the stakeholder survey (69%) were of the opinion that projects funded through PSF’s CSP could not be accomplished without SEP’s technical expertise. 52% of stakeholder survey respondents and 89% of SEP employees mentioned that PSF’s CSP is unable to provide technical support and expertise. Among respondents that provided comments, it was mentioned that technical expertise exists outside DFO but it would increase costs significantly to use private contractors. It was also suggested that PSF could build its own in-house technical expertise.

Among SEP employee respondents involved in PSF’s CSP projects, 50% spent one to two days per year reviewing and/or assessing project proposals while 27% spent three to five days, 10% spent six to ten days and 13% spent more than eleven days per year reviewing PSF’s CFP project proposals.

During the implementation of projects and/or activities funded by PSF’s CSP, SEP employees responded through the survey that 37% of those involved spend on average less than two hours per month while 33% of them spend more than ten hours per month providing guidance, oversight and support to ultimate recipients. According to the stakeholder survey, when asked how much time on average SEP employees spend with their organization providing guidance, oversight and support on projects and/or activities funded by PSF’s CSP, 36% answered less than two hours per month and 22% answered more than ten hours per month.

Considering that as of April 2013, PSF is receiving the entire revenue produced by the salmon conservation stamp and that PSF’s CSP projects are likely to increase substantially, it’s even more important to clearly define the expected role and involvement of SEP staff in the delivery of such activities.

SEP Design, Delivery and Priorities

SEP priorities are established annually. However, according to survey results, 79% of employees believe SEP priorities are not communicated effectively to staff. Clear communication of SEP priorities would help ensure a consistent delivery of the program throughout all areas.

The program has existed for more than 35 years. Twenty nine percent (29%) of employees that responded to the survey have worked for SEP for 31 years or more and 21% have been with the program between 21 to 30 years. Many of the respondents have therefore been with the program for more than twenty years which shows stability and deep knowledge and expertise as well as building long term relationships with stakeholders. However, a large portion of employees are close to retirement and staffing is challenging. There is also limited knowledge dissemination from more experienced employees to new employees. Over time, the program’s objectives and priorities have changed and the budget is more and more limited considering inflation, aging infrastructure and the large amount dedicated to fixed costs.

The program’s visibility, popularity and long term relationships established with stakeholders create challenges when it is time to make changes. Due to financial constraints, difficulties in staffing, inconsistencies in the delivery, unclear roles and priorities, SEP should determine which activities are core versus non-core and adjust the delivery of its activities accordingly.

Many employees (53%) consider that SEP should make changes to its priorities or to its design and delivery. Examples provided through the employee survey highlighted the need to establish clear priorities in Resources Restoration Units, where some employees suggested that work should be centralized.  In addition, more strategic focus on habitat restoration projects was suggested by some survey respondents, as these projects have the potential to play a greater role in salmon enhancement over hatchery production. The importance of habitat restoration was also raised during the site visits.

Could the efficiency of SEP be improved?

Finding: A number of efficiencies have already been implemented by the program but there is still room for improvement. SEP could not achieve the same results without additional funding received through other sources.


On average, 65% of stakeholder survey respondents were of the opinion that SEP was efficient in the areas of fish production, public stewardship, partnerships, habitat restoration and habitat enhancement.  However, a few survey respondents (average of 14%) felt that SEP was not efficient with respect to the areas of habitat restoration and habitat enhancement.

According to the stakeholder survey, the majority of respondents thought that the provision of technical support and expertise to CEDPs, PIPs, Pacific Salmon Foundation and to organisations receiving funding from PSF’s CSP were accomplished in an efficient manner from a considerable to a great extent.

According to SEP employees, the program has been greatly efficient in fish production and public stewardship (education and awareness) activities.  Staff also believe that other program activities were moderately efficient, such as planning and integration with other programs within DFO, operational support (day-to-day operations) in the different area offices and/or facilities, partnerships, habitat restoration, provision of technical expertise to CEDPs, to PIPs, to the Pacific Salmon Foundation and to organizations receiving funding through PSF’s CSP.

A number of efficiencies have already been identified and initiated by the program.  Cost-saving initiatives have been implemented in several major facilities, such as system monitoring technology (allowing for reductions in on-site stand-by), hydro-powered micro-generation, use of geo-thermal heat, high efficiency lighting, etc.  The Kitimat Hatchery, for example, at one time was supplied by surplus heat generated by the neighbouring pulp mill to heat hatchery water.

Both stakeholder and employee survey respondents are of the opinion that SEP could improve on issues of communication. Areas for improvement include communication between Regional Headquarters and Areas (and vice versa), communication within areas, communication with other DFO Sectors, and communication between SEP and stakeholders.

As mentioned in the relevance section, there is significant funding dedicated to salmon in the Pacific Region. Although some discussions occur in different committees and the Areas are aware of the projects funded, a greater coordination could be beneficial as sometimes one project could be detrimental to the efforts made by other programs. For example, during the site visits, it was brought to the evaluators’ attention that one volunteer stewardship group received funding through a contribution agreement with another DFO program to remove a concrete dam from a small fish bearing stream. Further assessment of this initiative by SEP staff (Community Advisor and Resource Restoration biologist and engineer) revealed that removal of the dam would have resulted in destabilization of the watercourse upstream of the dam resulting in potential damage to fish habitat and private property.

Community advisors, restoration staff, and even hatcheries are highly successful at leveraging 3rd party funding from different sources. Examples of other funding sources that contribute to the achievement of SEP’s results mentioned in the employee survey are:

  • Collaborative agreements with stakeholders, both financial and in-kind contributions, B.C. Conservation Foundation, Fish and Wildlife Compensation Program, Georgia Basin Living Rivers, Northern and Southern Fund, compensation and other funds from railroads, highways, hydro and some private company, funding through a variety of community partners including BC Hydro, City Government, Provincial Government, US/Canada Pacific Salmon Treaty, Rotary Clubs, etc.

SEP relies on a large volunteer base to operate CEDP facilities and PIP hatcheries.  In return, these volunteer organizations are required to supplement their budgets by fund-raising or by applying for funding assistance through other programs.  For example, Seymour Hatchery secured matching funding from Metro Vancouver.

Moreover, SEP leverages funds through its major facilities. On average, SEP leveraged $7.7 million annually in years 2009-10, 2011-12 and 2012-13.21 For example, the Fish and Wildlife Conservation Fund provided funds to install circular rearing tanks at Rosewall Creek Hatchery allowing summer Chinook to be transported and ripen at the facility.  The Quinsam River Hatchery also established collaborative agreements with several organizations in exchange for services provided by the Hatchery.

The majority of employee survey respondents indicated that SEP could not achieve the same results without additional funding received through other sources.  Therefore leveraging is critical in achieving SEP’s intended outcomes.

SEP’s contribution agreements also rely on funding through additional sources.  In fact, it is a requirement of both contribution agreements with PSF’s CSP and with TBSEF that DFO funding be matched by other sources. For example, the evaluation found that both TBSEF and PSF’s CSP leverage DFO’s contribution by generating additional revenue through sponsorships, cash and in-kind donations, fundraising events, etc.


21 SEP Workplans of 2009-10, 2011-12 and 2012-13.

Table 12. DFO’s annual contribution agreement with PSF’s CSP

Table 12
Year DFO Cash Contribution
($)
10% Administrative Fee
($)
Total DFO Contribution
($)
2009-10 291,789** 32,421 324,210
2010-11 334,386** 37,154 371,540***
2011-12 316,531 35,170 351,702
2012-13 316,166 35,129 351,296***
2013-14* 1,184,755 118,475 1,303,230

Source: PSF’s CSP final reports.
*      The 2013-14 financial data are projected contribution figures.  It may vary from actual expenditures.
**    In 2009-10 and in 2010-11, PSF carried forward unused project funds to the following fiscal year although the contribution agreement stipulated the requirement to return unexpended funds to DFO. In early 2014, an amendment to the contribution agreement was made requiring the return of funds at the end of the term of the Agreement.
***  The evaluation noticed anomalies in financial reporting, as total DFO contribution for 2010-11 and 2012-13 reported in PSF annual reports are not consistent with figures provided by the Program (see Table 1).

Although several efficiencies have already been implemented by the program, the infrastructure strategy establishes ten next steps as part of its implementation, many of which are related to further or potential efficiencies.  The efficiencies identified are:

  • Completion of a decision making framework for resource restoration projects in order to allow for effective and responsible initiation and management of resource restoration projects;
  • Incorporation of a Sustainable Energy Management Plan into future maintenance upgrades and refits;
  • Exploration of the idea of “branding” as a way to identify, understand and build upon the unique, key production strengths of each SEP facility building upon what they are best known for.  Branding would not be intended to limit the activities of a particular facility, but rather a way to inform capital and maintenance investment planning for sites, and could help SEP management determine where more flexibility can or should be built into specific sites to ensure SEP can adapt and respond over time to evolving DFO priorities; and,
  • Exploration with partners of the introduction of the same or similar software-based preventative maintenance program to CEDP facilities.

Most program staff (69%) also consider that the use of new technologies in enhancement facilities could help improve SEP’s efficiency.  Examples of new or different technologies include: the use of high efficiency pumps with variable frequency drive, water recirculation systems (where applicable), circular fiberglass rearing containers with covers instead of open raceways and ponds, web applications on smart phones for data collection by volunteers, and DNA marking to enable more accurate estimates of enhanced contributions to the Wild Salmon Policy.

Both survey respondents and key informants also provided examples of adjustments to SEP’s program governance and design and delivery that could lead to improved efficiencies.  These suggestions included amalgamating or transferring fish production of some CEDPs to major facilities, revisiting the governance model employed for non-production aspects of the program, and keeping partners more regularly informed of program changes to increase the likelihood of stakeholder support and buy-in.

4.4 ECONOMY

Are there alternative, more cost-effective ways of achieving program goals?

Finding: A number of cost-effective alternatives were raised, such as identifying core versus non-core activities, determining the added value of some contracts and streamlining the process, and exploring cost-sharing opportunities for the production of Steelhead and Cutthroat trout species.


As previously noted, the program should consider identifying core and non-core activities in order to increase cost-effectiveness.  For example, SEP’s activities are currently structured under two major components: fish production and community involvement.  Although fish production is typically carried out by major facilities, other volunteer facilities, such as CEDPs, DPIs and PIPs also contribute to SEP’s fish production.  The contrary is also true; CEDPs, DPIs and PIPs focus mostly on community involvement, stewardship, education and awareness, yet some of SEP’s major facilities also contribute to these activities.  For example, the Quinsam major hatchery hosts a visitor education centre where patrons can learn more about salmon and their habitat. 

SEP staff time and effort is also questioned regarding fish production in community involvement facilities considering their minimal contribution to overall fish production.  SEP staff clearly concluded through their survey responses that CEDP facilities and PIP small hatcheries do not significantly contribute to fish production objectives.  The review of fish production plans showed that some CEDPs, DPIs and PIPs contribute to fish production for harvest opportunities.  In fact, CEDPs represent 4.6% of SEP’s total fish production, DPIs 2.8%, PIPs 0.9%, while the remainder of production (91.7%) is carried out in SEP major hatcheries.  A review of core and non-core activities with respect to fish production and community involvement could help the program focus on key priorities. This review would also allow the streamlining of the logic model, its activities and its outcomes.  For instance, the program should determine how CEDPs, DPIs and PIPs contribute to SEP’s outcomes.

The identification of core versus non-core activities may also be helpful for SEP in moving forward with Steelhead and Cutthroat trout production. In 2011, SEP produced Steelhead and Cutthroat species at seven of its major hatcheries. Two CEDP facilities and one DPI hatchery also produced Steelhead and Cutthroat in 2011.

The evaluation found possible cost-sharing opportunities for SEP for the production of Steelhead and Cutthroat species. The signed agreement between the Province of British Columbia and the Federal Government sets out funding responsibilities for the two levels of government. The evaluation found no evidence that there has been an exchange of funds for the production of Steelhead and Cutthroat trout species. The province of BC has undertaken the responsibility of the administration of freshwater sport fisheries. Specifically, the sport fishing of these species in non-tidal waters is managed by the province of BC. The province of BC receives $144 million annually from freshwater licensing and tax revenues.22  Considering this revenue, SEP could consider the opportunity for cost-sharing Steelhead and Cutthroat enhancement activities (estimated by the program at $400K annually).  This alternative was strongly supported by employee survey respondents, key informants and during site visits.

In addition, the education and awareness components of the program may be revisited considering that consistent efforts made over the past 35 years has contributed to salmon becoming an icon in BC. Moreover, education is not clearly linked to DFO’s priorities or mandate.

Streamlining of contracts may also help SEP achieve its goals in a more efficient manner.  For example, PIP contracts are currently awarded to community involvement groups through their local Community Advisor.  However, there application forms and follow-up vary by Area and by Community Advisor, making it difficult to determine the extent to which PIP contracts are contributing to SEP’s objectives. Community Advisors can also award contracts for the provision of technical expertise in community involvement facilities and for carrying out educational components in local schools. Given the expertise that already exists within SEP and the number of schools already engaged, the added value of these types of contracts is questioned.

Finally, during the period reviewed, the Salmon Enhancement and Habitat Advisory Board (SEHAB) was awarded $33,000 annually in order to provide SEP with advice from the volunteer community perspective on issues of program delivery and program policy.  As described in the Terms of Reference, SEHAB is required to provide an annual advisory report and one was provided as evidence. There is evidence that indicates SEHAB has provided advice to the program but this advice is not fully documented and did not come in the form of an annual report.


22 Economic Analysis, Freshwater Fisheries Society of BC, www.gofishbc.com

5. CONCLUSIONS AND RECOMMENDATIONS


Relevance

Overall, the evaluation found that there is a continued need for the Salmonid Enhancement Program through the Pacific Salmon Treaty and the Wild Salmon Policy.
SEP remains relevant as its activities also contribute to the Government of Canada outcomes and supports DFO’s strategic outcomes. The evaluation evidence demonstrated that funding towards Pacific salmon is widespread in British Columbia as there are several federal government programs and external organizations that also work towards protecting, restoring and/or enhancing Pacific salmon.

Previous studies determined that the program contributes approximately $89 million to British Columbia’s gross domestic product and generate direct employment of 592 jobs. SEP, through an agreement with the Province of British Columbia, is producing Steelhead and Cutthroat trout species at its hatcheries but the management of these species is shared between the federal government and the province of British Columbia governments.

Activities undertaken as part of the contribution agreement with the Pacific Salmon Foundation’s Community Salmon Program share similar objectives to SEP. As well, the contribution agreement with T. Buck Suzuki Environmental Foundation activities ultimately leads to an improved marine ecosystem, thus supporting healthy salmon stocks.  However linkage between TBSEF and SEP is weak.

Performance (Economy, Efficiency and Effectiveness)

SEP immediate, intermediate and long-term outcomes have been met to some extent. The evaluation faced attribution challenges as some outcomes are out of SEP’s control. Moreover, lack of evidence did not permit to conclude on the achievement of some outcomes, such as “vulnerable salmon species populations are supported”, “the public has access to harvest opportunities”, and “sustainable marine and freshwater ecosystems provide harvest opportunities”. Additional linkages with Ecosystems and Oceans Science Sector as well as Ecosystems and Fisheries Management Sector would be beneficial as SEP is dependent on other groups’ expertise to advance some of its activities such as stock assessment.

SEP provides harvest opportunities through enhancing salmon stocks and is successful in engaging stewards to participate in cooperative fisheries and stewardship activities. Through the Pacific Streamkeepers Federation, PSF’s CSP, habitat restoration, stock assessment and enhancement, SEP helps the public support the conservation, stewardship and rebuilding of salmon and their habitat, but there is room for opportunities of improvement. For example, the evaluation found that SEP has no strategy and/or vision related to habitat restoration and how the contribution of its delivery partners helps SEP to achieve its results is not clear.

SEP could not achieve the same results without additional funding from external sources, such as the Northern Boundary and Transboundary Rivers Restoration Enhancement Fund and the Southern Boundary and Transboundary Rivers Restoration Enhancement Fund. Additionally, SEP leverages funds through major facilities, CEDPs, PIPs, CAs, restoration projects and both contribution agreements.

Respondents from both surveys agree that PSF’s CSP has not achieved its outcome of “First Nations, local communities and external parties participate in cooperative fisheries and watershed stewardship activities” established in PSF’s CSP contribution agreement. In regard to the achievement of the other two outcomes established in the contribution agreement “Increased collaboration among community stakeholders in salmon enhancement, habitat conservation and restoration projects” and “Public Awareness about salmon in Pacific Region is increased”, the views of respondents are diverging. SEP employees are of the opinion that PSF’s CSP is not meeting those outcomes, while stakeholders disagree.

The program is successful in collecting and using data with respect to fish production. However, SEP’s data collection methods regarding community involvement activities are inconsistent across Areas and vary from year to year. Improved documentation and data collection on how the many components of the program (e.g. Stream to Sea program) contribute to SEP’s outcome would be beneficial to SEP as it could better demonstrate the program’s achievements. Also, a review of performance indicators and the logic model for SEP and its contribution agreements would be necessary in order to effectively report on program achievements.

The majority of employee survey respondents believe that changes are necessary in the design and delivery of SEP. The evaluation revealed that improvements can be made to SEP’s governance and leadership as well as the communication of priorities to employees. SEP would benefit from communicating clear directions to ensure consistency in delivery throughout Area offices. More specifically, there is a need to establish priorities and clear roles and responsibilities for Community Advisors as delivery of the program varies from one Area to another.

There is also a need to clarify the roles and responsibilities in the contribution agreement between DFO and PSF’s CSP related to SEP employees’ involvement in the project selection process as well as in the delivery of some PSF’s CSP funded projects. PSF’s CSP and ultimate recipients are dependent on DFO’s technical expertise and the large majority of respondents from both surveys are of the opinion that projects funded through PSF’s CSP could not be accomplished without SEP’s technical expertise.

Suggestions to improve SEP’s efficiency include the use of new technologies (e.g., high efficiency pumps with variable frequency drive) and amalgamating or transferring fish production of some CEDPs to major facilities, revisiting the governance model employed for non-production aspects of the program, and keeping partners more regularly apprised of program changes to increase the likelihood of stakeholder support and buy-in.

As for alternatives, the percentage of fish released from the community involvement facilities (CEDP, DPI and PIP) represent only 8% of total fish released and major facilities are not producing at maximum capacity. Reviewing the role and contribution of community involvement facilities could be helpful in improving SEP’s efficiency and economy.

The identification of core versus non-core activities may also help SEP identify priorities and allocate resources. Specifically, there could be cost-sharing opportunities between SEP and the province of British Columbia for Steelhead and Cutthroat trout species. Education and awareness were activities also raised as potential non-core responsibilities.

Finally, during the period reviewed, the Salmon Enhancement and Habitat Advisory Board (SEHAB) was awarded $33,000 annually in order to provide SEP with advice from the volunteer community perspective on issues of program delivery and program policy.  As described in the Terms of Reference, SEHAB is required to provide an annual advisory report and one was provided as evidence. There is evidence that indicates SEHAB has provided advice to the program but this advice is not fully documented and did not come in the form of an annual report.

Recommendations

With the above key findings in mind, the evaluation has made the following six recommendations.

Recommendation 1: Considering that data collection is inconsistent and in some instances no performance information was available to demonstrate SEP’s achievements, it is recommended that the Regional Director General (RDG), Pacific Region, review SEP’s Performance Measurement (PM) Strategy (logic model and performance indicators) to ensure that it appropriately reflects SEP’s performance story.  Any changes in the PM Strategy affecting the contribution agreements with the Pacific Salmon Foundation and with the T. Buck Suzuki Environmental Foundation should be reflected in their respective contribution agreements. Once the PM strategy is revised, ensure that data be collected and reported consistently across each area.

Recommendation 2: Roles and responsibilities in the contribution agreement between DFO and PSF’s CSP regarding SEP employees’ involvement in the project selection process as well as in the delivery of PSF’s CSP funded projects needs to be clarified, therefore it is recommended that the RDG, Pacific Region, ensure that the renewal of the contribution agreement with PSF’s CSP defines   the roles and responsibilities between DFO and PSF’s CSP, especially regarding the involvement of SEP employees in the selection and delivery of PSF’s CSP projects. Any changes to the contribution agreement should be communicated to all SEP employees.

Recommendation 3: An agreement between the province of British Columbia and the federal government set out funding responsibilities for the two levels of government. However, the evaluation found no evidence that there has been an exchange of funds for the production of Steelhead and Cutthroat trout species. So far, SEP supported all production costs for Steelhead and Cutthroat trout species. Because the responsibility of these species is shared between the federal and provincial government, it is recommended that the RDG, Pacific Region, advance the cost-sharing opportunities between DFO and the province of British Columbia with respect to the production of Steelhead and Cutthroat trout species.

Recommendation 4: Looking forward and taking into consideration the opportunities for improving the efficiency and economy of the program, it is recommended that the RDG, Pacific Region, develop a framework or strategy to identify SEP’s activities that are core to its mandate and to prioritize the non-core activities. The framework or strategy should include an action plan based on available resources, with specific timelines to implement necessary changes and re-focus SEP activities.

Recommendation 5: Considering the importance of habitat in achieving sustainability of resources in the longer term and that the evaluation found no clear strategy or vision related to the rebuilding of salmon habitat within SEP, it is recommended that the RDG, Pacific Region, develop a habitat restoration strategy for SEP.

Recommendation 6: SEP would benefit from clarifying the roles and responsibilities of the Salmon Enhancement and Habitat Advisory Board and its contribution to the achievement of SEP’s results. As such, it is recommended that the RDG, Pacific Region, review the roles and responsibilities of SEHAB and ensure that SEHAB’s contribution to the achievement of SEP’s outcomes is clearly documented.

ANNEX A: SEP EVALUATION MATRIX


annex a
Evaluation Questions/issues Indicators Data source
Relevance
1)
Is there a continued need for SEP and its contribution agreements?
  • Evidence of the needs of Canadians regarding SEP. Have they changed over time?
  • Evidence of the needs of Canadians regarding the contribution agreements.  Have they changed over time?
  • Evidence that SEP is unique, duplicates or is complementary to other programs
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
2)
Is SEP aligned with Government of Canada and DFO priorities? Are the contribution agreements aligned with SEP priorities?
  • Evidence of SEP’s objectives (including infrastructure) and priorities are aligned to federal objectives and priorities
  • Alignment of PSF’s CSP and TBSEF contribution agreements with SEP priorities.
  • Key informant respondents opinions
  • Document review
  • Key informant interviews
3)
Is SEP aligned with federal roles and responsibilities?
  • Evidence of alignment between SEP’s objectives and the legislated authorities (including the contribution agreement’s objectives)
  • Key informant respondents opinions
  • Document review
  • Key informant interviews
Performance
Effectiveness
Immediate outcomes
4.1
Vulnerable salmon populations are supported
  • Percentage of enhancement facility production groups in the Integrated Fisheries Management Plans (IFMP) where the objective of enhancement is conservation and rebuilding of vulnerable stocks
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
4.2
Enhanced salmon provide harvest opportunities
  • Number and percentage of total release of enhanced salmon by type of facilities and by species
  • Percentage of maximum target release for all major facilities from 2008-2012 for all salmon species except Steelhead and Cutthroat trout
  • Percentage of enhancement facility production groups in the Integrated Fisheries Management Plans (IFMP) where the objective of enhancement is harvest or stock assessment
  • Number of harvested enhanced Pacific salmon contribution to total catch
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
4.3
First Nation, local communities and external parties participate in cooperative fisheries and stewardship activities
  • Number of schools and incubators in the Stream to Sea Program per year
  • Evidence of how CEDPs and PIPs help community groups participate in cooperative fisheries and stewardship activities
  • Number of volunteer hours contributed by PSF’s CSP
  • Evidence of TBSEF’s contribution
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
  • Site visits
Intermediate outcomes
5.1
The public has access to harvest opportunities
  • Number of salmon commercial licenses
  • Number of Food, Social and Ceremonial licences
  • Number of Excess to Spawning Requirements licences
  • Key informant and survey respondents opinions
  • SEP databases
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
5.2
The public supports the conservation, stewardship and rebuilding of salmon and their habitat
  • M2 of habitat improved by SEP
  • M2 of habitat improved by the contribution agreement with PSF’s CSP
  • Number of volunteers hours contributed to stock enhancement and stock assessment by PSF-CSP projects
  • Evidence of SEHAB’s and Pacific Streamkeepers Federation  contribution to SEP
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
Long-term outcome
6)
Healthy and diverse salmon populations and sustainable marine and freshwater ecosystems provide harvest opportunities and support a stronger Canadian economy
  • Evidence of SEP’s contribution to the Canadian GDP through harvest opportunities
  • Evidence of SEP’s contribution to healthy and diverse salmon populations
  • Evidence of SEP’s contribution to the sustainability of the marine and freshwater ecosystems
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
7)
Are there any external factors and/or challenges that may have impacted the results of the program? Did the program have any unintended results?
  • Opinions of key informants and survey respondents
  • Examples from key informants and survey respondents
  • Evidence from documents
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
  • Site visits
Efficiency and Economy
8)
Are there best practices and lessons learned from SEP?
  • Examples from key informants, survey respondents and documents
  • Examples of successful facilities
  • Analysis of contextual and operational factors contributing to facilities success
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
  • Site visits
9)
Is there a performance reporting process/system in place? If yes, is it being used for decision making?
  • Evidence of a performance measurement strategy
  • Existence of a data collection system/database
  • Evidence of use of performance information
  • Availability and quality of performance information
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Key informant interviews
10)
To what extent are the program’s activities, structures and processes appropriate to support the achievement of results?
  • Evidence and opinions of key informants and survey respondents on the appropriateness and effectiveness of:
    • Program’s structure and processes
    • Governance
    • Clarification of roles and responsibilities
    • Resources
    • SEP design, delivery and priorities
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
  • Site visits
11)
Could the efficiency of SEP activities be improved?
  • Examples of possible further efficiency improvements
  • Examples of efficient and successful facilities
  • Evidence of additional funding related to SEP
  • Funds leveraged by SEP and by the contribution agreements
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
  • Site visits
12)
Are there alternatives more cost-effective ways of achieving program goals?
  • Opinions of key informants and survey respondents on ways the objectives, priorities and/or design of the program could or should be rethought to make it more economical
  • Cost-sharing opportunities
  • Alternative delivery models for SEP
  • Key informant and survey respondents opinions
  • Document review
  • Program staff survey
  • Stakeholder survey
  • Key informant interviews
  • Site visits

ANNEX B: MANAGEMENT ACTION PLAN


Management Action Plan
RECOMMENDATION
Rationale: Data collection is inconsistent and in some instances no performance information was available to demonstrate SEP’s achievements. An improved documentation and data collection on how many components of the program (e.g. Stream to Sea program) contribute to SEP’s outcome would be beneficial to SEP as it could better demonstrate the program’s achievements. In addition, a review of performance indicators and the logic model for SEP and its contribution agreements would be necessary in order to effectively report on program achievements.

Recommendation 1: It is recommended that the RDG, Pacific Region, review SEP’s Performance Measurement (PM) Strategy (logic model and performance indicators) to ensure that it appropriately reflects SEP’s performance story.  Any changes in the PM Strategy affecting the contribution agreements with PSF’s CSP and TBSEF, should be reflected in their respective contribution agreements. Once the PM strategy is revised, ensure that data be collected and reported consistently across each area.
STRATEGY
Strike a SEP working group to review and revise the SEP Performance Measurement (PM) Strategy (logic model and performance indicators).
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Examination of performance data collection sources and uses; validation of existing data sources and / or replacement with other more robust, consistent and relevant data sets November 2015    
Review Contribution Agreements with PSF - CSP and TBSEF to ensure alignment with new performance data collection and measurement needs. An Accountability Framework document will be appended to the new Contribution Agreement that will require DFO and the PSF to jointly establish a technical committee to examine and recommend appropriate performance measures that will assess program effectiveness over the long term. December 2015    
Seek RD EMB (and RDG Pacific) involvement and approval for revisions to the SEP program logic model, key delivery outputs and outcomes and detailed performance indicators March 2016    
Seek RD EMB approval to revise the Performance Measurement Strategy and update the Performance Measurement Framework for DPR/RPP planning and reporting purposes September 2016    
RECOMMENDATION
Rationale: Roles and responsibilities in the contribution agreement between DFO and PSF’s CSP regarding SEP employees’ involvement in the project selection process as well as in the delivery of PSF’s CSP funded projects needs to be defined. PSF’s CSP and ultimate recipients are dependent on DFO’s technical expertise and the large majority of respondents from both surveys are of the opinion that projects funded through PSF’s CSP could not be accomplished without SEP’s technical expertise.

Recommendation 2: It is recommended that the RDG, Pacific Region, ensure that the renewal of the contribution agreement with PSF’s CSP defines the roles and responsibilities between DFO and PSF’s CSP, especially regarding the involvement of SEP employees in the selection and delivery of PSF’s CSP projects. Any changes to the contribution agreement should be communicated to all SEP employees.
STRATEGY
Work bilaterally with PSF to define and document roles and responsibilities for DFO staff re: Community Salmon Program (CSP) project selection and support.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Specific actions to include:      
In the context of TB’s Policy on Transfer Payments, DFO SEP will collaborate with PSF to develop a formal Accountability Framework confirming that:
  • DFO staff will not sit on the CSP Selection Board
  • DFO has no decision making role in CSP project selection
  • DFO has no role in electing members to PSF’s CSP selection committee

The completed Accountability Framework will be vetted for approval through the PSF Board of Directors and DFO executive management.  Once supported it will then  be implemented and will formally limit the roles of DFO staff.

The approved Accountability Framework and any other relevant materials will be appended to the PSF Contribution Agreement.

April 2015    
Through the development of a a SEP Planning Framework for Restoration Projects and other program planning and guidance documentation, DFO will  define and communicate the nature and scope of work that can be undertaken by the Resource Restoration unit and its staff.  Specifically, the new guidance will:
  • define the activities of the restoration program;
  • provide restoration units with a structured criteria for prioritizing and selecting projects;
  • establish annual plans and reporting requirements;
  • communicate  objectives, guiding principles, and processes to Resource Restoration unit staff and external clients 
November 2015    
RECOMMENDATION
Rationale: The evaluation found possible cost-sharing opportunities for SEP for the production of Steelhead and Cutthroat species. The signed agreement between the province of British Columbia and the federal government sets out funding responsibilities for the two levels of government. The evaluation found no evidence that there has been an exchange of funds for the production of Steelhead and Cutthroat trout species.

Recommendation 3: It is recommended that the RDG, Pacific Region, advance the cost-sharing opportunities between DFO and the province of BC with respect to the production of Steelhead and Cutthroat species.
STRATEGY
Establish a bilateral working group with BC Ministry of Fisheries Lands and Natural Resources to develop a draft MOU governing Steelhead and Cutthroat production by DFO.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
MOU content to consider:
  • Accountabilities
  • Roles and responsibilities
  • Governance
  • Production Planning
  • Licensing
  • Communications
  • Exploration of cost sharing opportunities for DFO delivered Steelhead and Cutthroat culture
November 2015    
RECOMMENDATION
Rationale: Due to financial constraints, difficulties in staffing, inconsistencies in the delivery of the program, unclear roles and priorities and looking forward and taking into consideration the opportunities for improving the efficiency and economy of the program, SEP should determine which activities are core versus non-core and adjust the delivery of its activities accordingly. Specifically, a review of core and non-core activities with respect to fish production and community involvement could help the program focus on key priorities. Overall, the identification of core versus non-core activities may help SEP identify priorities and allocate resources.

Recommendation 4: It is recommended that the RDG, Pacific Region, develop a framework or strategy to identify SEP’s activities that are core to its mandate and to prioritize the non-core activities. The framework or strategy should include an action plan based on available resources, with specific timelines to implement necessary changes and re-focus SEP activities.
STRATEGY
Strike a SEP working group to assess current delivery activities against SEP core program mandate (as defined in a renewed SEP logic model )
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Undertake an exercise to establish SEP baseline SW and O&M budgets to inform development of an action plan to align resources to delivery of core mandate work. July 2015    
Develop core/non-core priority setting criteria and an assessment process to support allocation of SEP resources; the delivery of core  activities will be specifically captured in annual work plans October, 2015    
RECOMMENDATION
Rationale: Although SEP and its contribution agreement with PSF’s CSP are supporting salmon enhancement through habitat restoration and considering the importance of habitat in achieving sustainability of resources in the longer term, the evaluation found no clear strategy or vision related to the rebuilding of salmon habitat within SEP.

Recommendation 5: It is recommended that the RDG, Pacific Region, develop a habitat restoration strategy for SEP.
STRATEGY
Design program guidance documentation that outlines a clear strategic direction for SEP habitat restoration programming. This document will be integrated with other key SEP program guidance documents.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Develop and document strategic program direction that will:
  • establish guiding principles to set SEP restoration activities, assess risks and coordinate delivery of restoration works with other DFO programs
  • confirm key activities to be undertaken by the Resource Restoration program
  • outline resource restoration planning processes
  • support decision making and project monitoring and reporting
  • The outcomes of this strategic direction will be reflected as specific actions in annual RRU work plans.
January 2016    
RECOMMENDATION
Rationale: As described in the Salmon Enhancement and Habitat Advisory Board’s (SEHAB’s) Terms of Reference, SEHAB is required to provide an annual advisory report to the responsible Regional Director, Oceans, Ecosystems Management Branch23 and one report was produced in 2013 by SEHAB. There is evidence that indicates SEHAB has provided advice to the program but this advice is not fully documented and did not come in the form of an annual report. SEP would benefit from clarifying the roles and responsibilities of the Salmon Enhancement and Habitat Advisory Board and its contribution to the achievement of SEP’s results.

Recommendation 6: It is recommended that the RDG, Pacific Region, review the roles and responsibilities of the Salmon Enhancement and Habitat Advisory Board and ensure that SEHAB’s contribution to the achievement of SEP’s outcomes is clearly documented.
STRATEGY
Working bilaterally with the SEHAB, SEP will review and renew the terms of the Department’s contract with SEHAB.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Specific actions supporting renewal for the SEHAB contract will include:
  • more clearly defining SEHAB’s advisory role
  • confirming the specific nature and scope of the advice that SEHAB will provide DFO
  • confirming the administrative process by which SEHAB’s advice will be provided and its frequency.
  • establishing the DFO processes for documenting and following up on SEHAB advice to on to ensure necessary actions are taken
March 2016    

23 Salmon Enhancement and Habitat Advisory Board, Terms of Reference. p.5