Evaluation of the Science for Species at Risk Program

Project Number 6B144x
June 2011

Table of Contents

List of Acronyms

ADM
Assistant Deputy Minister
B.C.
British Columbia
CEAA
Canadian Environmental Assessment Act
CEC
Commission for Environmental Cooperation
CITES
Convention on International Trade in Endangered Species
COSEWIC
Committee on the Status of Endangered Wildlife in Canada
CSAS
Canadian Science Advisory Secretariat
DFO
Fisheries and Oceans Canada
DG
Director General
DND
Department of National Defence
DPR
Departmental Performance Report
EC
Environment Canada
EFM
Ecosystems and Fisheries Management
FRS
Fisheries Resources Science
FSCP
Fisheries Science Collaborative Program
FTE
Full-Time Equivalent
HAPAE
Health and Productive Aquatic Ecosystems
ICEE
International Conference on Environment and Energy
IRF
Interdepartmental Recovery Fund
NAFO
Northwest Atlantic Fisheries Organization
NHQ
National Headquarters
NGO
Non-governmental organization
NSERC
National Science and Engineering Research Council of Canada
O&M
Operations and Maintenance
PAA
Program Activity Architecture
PCA
Parks Canada Agency
PWGSC
Public Works Government Services of Canada
RDG
Regional Director General
RPA
Recovery Potential Assessment
RPP
Reports on Plans and Priorities
SARA
Species At Risk Act
SFA
Sustainable Fisheries Aquaculture
SSAR
Science for Species at Risk
TC
Transport Canada
UN
United Nations
U.S.
United States

Executive Summary

Introduction

The Species at Risk Act (SARA), 2003, was created to prevent wildlife species from becoming extinct; to provide for the recovery of threatened, endangered and extirpated species; and to manage species of special concern to prevent them from becoming endangered or threatened (SARA, S.6). Responsibility for the implementation of SARA is assigned to Environment Canada (EC), Fisheries and Oceans Canada (DFO) and Parks Canada Agency (PCA). Specifically, DFO has the authority for aquatic species under SARA.

The focus of this evaluation is specifically the sub-activity of Science for Species at Risk (SSAR) based on the 2010-11 Program Activity Architecture (PAA) within DFO.

Evaluation objective and scope

The main objective of this evaluation is to determine the extent to which the SSAR Program is relevant and has it achieved its results in an economical and efficient manner. As such, the evaluation examines the extent to which the SSAR Program demonstrates relevance and performance, including effectiveness, efficiency and economy based on the Treasury Board Policy on Evaluation (2009).

The timeframe for this evaluation, a first evaluation for the SSAR Program, covers the period from 2005-06 to 2010-11. The data collection and analysis evaluation was undertaken between September 2010 and January 2011, with report writing and approvals occurring from January to June 2011.

Program information

As DFO has the authority for aquatic species under SARA, the DFO SSAR program undertakes targeted research and monitoring of species at risk and their habitats, and  provides information to support assessments of the conservation status of the species (SARA s. 14-26, 128); departmental listing recommendations and government decisions in this regard (SARA 27(2)(b)); assessment of recovery feasibility; advice with regards to recovery goals and objectives; and identification of critical habitat and threats affecting the species and its habitat in the form of Recovery Potential Assessments (RPA). In addition, DFO SSAR also provides standards and protocols to help guide community based stewardship programs.

SSAR also provides data and expertise to the Committee on the Status of Endangered Wildlife in Canada’s (COSEWIC) assessment process, and works with provincial, territorial and Wildlife Management Board experts under the general umbrella of the National Accord for the Protection of Species at Risk, as well as the National Framework for Species at Risk Conservation.

SSAR also reports on the general Status of Aquatic Wildlife (SARA sec. 128) every five years.   In addition SSAR also reports on the Recovery Progress for Species at Risk (SARA sec. 46 and 55). The total SSAR program cost is approximately $3.3 million annually.  

In the 2010-11 PAA, the Program directly contributes to DFO’s strategic outcome Healthy and Productive Aquatic Ecosystems (HAPAE), by supporting programs such as Protection, Recovery and Monitoring and Evaluation of species at risk, Habitat Management and Oceans Management. The program also contributes to the strategic outcome of Sustainable Fisheries Aquaculture (SFA), by contributing to Fisheries Resource and Aquaculture Management.

SSAR strongly benefits from the Fisheries Resources Science (FRS) species research, monitoring, and stock assessment activities before being formally identified as a species at risk. In addition, both the FRS and SSAR share common activities and outputs. For example, FRS and SSAR have the same activities of research, monitoring and providing advice on the status of aquatic and fisheries resources, which include species at risk. They also have common outputs, for example, scientific advice to support sound policy development and implementation. Therefore, the data collection for the SSAR evaluation occurred concurrently with that of FRS to provide a more efficient approach to the evaluations. The FRS evaluation was approved by DFO’s Departmental Evaluation Committee on February 15, 2011.  

Methodology

For this evaluation a non-experimental design was used, in which measurements are taken after the program has been implemented. This model was chosen for a number of reasons.  First, the SSAR Program is a full coverage program, intended to be delivered across Canada and cannot be withheld in any region due to the legal obligations under SARA; therefore, a control group could not be established. Secondly, there were no measures in place before the SSAR Program was introduced so only post-test measurements were used. Although the non-experimental design lacks scientific rigour, the rigour of this design was increased through the use of multiple lines of evidence and triangulation across those lines of evidence to reduce the occurrence of bias and increase validity. This enabled evaluators to make linkages, and logically argue that results can be attributed to the program.

The analysis methods used for this evaluation were tailored to the nature and availability of the data to be gathered, which were in turn linked to the evaluation questions. Extensive use of triangulation was used as an analytical method. In the social sciences, triangulation means that more than two methods are used in a study in order to corroborate findings. Document review, interviews, survey and a case study were used as methodologies to support triangulation in this evaluation.

Document review

A review of documentation was conducted to assess most evaluation issues. This line of evidence was chosen because the strengths of documentation review is that it can provide historical insight into the development and delivery of a program, verify data from other methodologies and provides primary data on issues such as relevance. The following types of documents were reviewed: government-wide documents (e.g. federal legislation pertaining to fisheries, federal speeches from the Throne, federal budgets, etc.), and departmental-level documents (e.g. Departmental Performance Reports (DPRs), Reports on Plans and Priorities (RPPs), DFO policies, DFO and Canadian Science Advisory Secretariat (CSAS) website and publications, etc.).  The departmental documents also include administrative data from CSAS.

Interview

Key informant interviews were chosen as a line of enquiry to gather in-depth information for evaluation purposes, including views, explanations, examples, contextual and factual information that address the evaluation questions. Approximately 26 interviews with DFO staff including program staff (within science) and DFO client staff, such as individuals from the Species at Risk Management Program were conducted between November 2010 and January 2011. Key informant interviews were conducted to assess most evaluation issues.

Due to the small number of clients (e.g. Species at Risk Management Program) interviewed their responses are not statistically representative for generalizing the findings to all users of the program’s activities. In this context, both the client and key informant interviews are based on their extensive experiences.  As there is a likelihood of subjectivity, the qualitative evidence was used in combination with other lines of evidence to ensure validity for reporting purposes. 

Survey

The purpose of the survey was to determine the use of FRS and SSAR advice and information, the level of client satisfaction with the outputs that FRS and SSAR provides, and perceptions about the utility and influence of FRS and SSAR advice and information on their decision-making process.

The primary source of survey respondents were obtained from the Canadian Science Advisory Secretariat (CSAS) database on annual science requests for advice (for the year 2009-2010). However, since the database does not contain names of clients the evaluation team made a request to the regions to obtain names for each of the science requests. Regions provided these names between November 2010 and January 2011. Approximately 117 clients received an email to complete an online survey for FRS and SSAR. The final response rate for the survey was 53% (n=62). Of the 117 clients, 62 individuals responded to sections relevant to their situation and 21 of the 62 survey respondents, indicated that they requested information relating to species at risk. The information from these 21 respondents was used as evidence for the SSAR evaluation.

Case study

A case study was chosen as a line of evidence to provide in-depth focus on the contextual issues affecting the delivery and results of FRS and SSAR through a specific example. One case study, the Pacific Region killer whale (a species at risk) was undertaken in the FRS evaluation addresses the linkages between SSAR and FRS. Specifically, the case study was chosen to demonstrate the lessons learned and the challenges of collaboration between a species at risk (researched and monitored by SSAR) and its prey, which is not a species at risk and is researched and monitored by FRS.

The primary methods used in the case studies were key informant interviews, as well as a review of relevant documentation pertaining to the case. Interviews were conducted with regional FRS staff. Key informants were selected based on their knowledge of the subject matter. The case study was carried out between November 2010 and January 2011.

Constraints and Caveats

Over the course of the evaluation, the following limitations in the methodology were observed:

  • Organizational structure. SSAR is not an organizational entity within DFO Science (science sector), rather it is a sub-activity defined in DFO’s Program Activity Architecture (PAA). There is a challenge in identifying which of DFO Science’s resources, activities, outputs and outcomes are aligned with SSAR (e.g. many key informants did not know of SSAR; they were only aware of DFO Science). In addition, this evaluation occurred in combination with the FRS evaluation. As there is a likelihood of overlap between DFO Science in general and SSAR, the qualitative evidence was used in combination with other lines of evidence to ensure neutrality for reporting purposes. Therefore, the limitations of the organizational structure were mitigated by the use of multiple lines of evidence and therefore have not impacted the validity of the findings. 

  • Performance information. There was limited performance data available for the evaluation. To ensure the validity of the findings the evaluation supplemented for the lack of performance measurement by using additional lines of evidence and the use of triangulation.  It is not expected to have negatively impacted the validity of the findings.

  • Methodological limitations.

    • Document review: When reviewing the financial data of SSAR it did not include a breakdown of budget allocation by activities, outputs and outcomes, an analysis of research costs was not available and no comparative program was found. As such, this has limited the analysis of the efficiency and economy of the SSAR Program. To counter these limitations the evaluation team used alternate quantitative (e.g. survey, etc.) and qualitative measures (e.g. interviews, etc.) to assess the efficiency and potential impacts of the programs. The use of standardized tested tools to collect this data should ensure that these limitations have not negatively impacted the validity of the findings.

    • Interviews: Due to time and resource constraints, there was a limit in terms of how many key informants outside of DFO (i.e. external clients) could be interviewed and surveyed. In this context, both the client and key informant interviews were chosen based on their extensive experiences.  This has limited the number of perspectives and hence there is a risk of bias in their responses. As there is a likelihood of subjectivity, to ensure the validity of findings, the qualitative evidence was triangulated with other lines of evidence (e.g. document review, case study, etc.) to ensure neutrality for reporting purposes. However, the limited data from external interviews is not expected to negatively impact the validity of the findings. Given the wide range of external stakeholders that are particularly affected by DFO decisions and science advice/information, further study may be warranted to fully examine the views of external stakeholders and their opinions of DFO science efforts.

    • Survey: It should be noted that due to the small number of respondents (21), that responses are not representative of all potential users and findings from the survey are more suggestive in nature.  Readers are to use caution in interpreting results from surveys.

    • Case study: Due to the small number of individuals interviewed, responses are not statistically representative for generalizing the findings to all users of the program's activities. In this context, the key informants were chosen based on their extensive knowledge and experience with the case being examined.  As there is a likelihood of subjectivity, the qualitative evidence was used in combination with other lines of evidence to ensure validity for reporting purposes. 

Evaluation findings and conclusions

Relevance

The evaluation found that SSAR is consistent with the Government of Canada’s legal obligation to protect wildlife species under the SARA, and is consistent with DFO’s objectives of supporting a healthy and productive aquatic ecosystem. For example, SSAR supports DFO’s objectives to identify and protect threatened aquatic species, by undertaking research and monitoring activities on species at risk to determine their status, recovery potential, and critical habitats.

The program also addresses DFO’s responsibilities to produce sound science to support management decisions for aquatic species at risk. In addition, SSAR responds to Canada’s international commitments, such as to the United Nations Convention on Biodiversity, the Commission for Environmental Cooperation (CEC), and the Convention on International Trade in Endangered Species (CITES) that depend on sound biological understanding and principles.

There is a continued need for SSAR to address the needs of Canadians, specifically SSAR’s DFO clients. In particular in providing the science base to ensure the protection of aquatic species at risk, such as determining their recovery potential, critical habitats, and supporting their listing and development of recovery plans as part of the legal obligations under SARA.

Effectiveness

The types of outputs that both FRS and SSAR produce may include the following products and services:tables and figures;research publications;peer review meeting reports;advice;policy documents; reports to domestic and international organizations such as regional fisheries management groups and international forums; and presentations and communication products.

A number of these formal and informal outputs are not captured within the CSAS process or in science journals, especially those related to data management, administration, communication, and informal outputs that scientists provide to clients (both internal and external). Therefore, the amount of outputs achieved by SSAR goes beyond what has been quantifiably measured.

Specifically, SSAR completed 73% of the requests for science advice on aquatic species at risk, as specified in the 2009-10 DPR. The requests for science advice originate from the SAR Management program, which have been organized since 2010-11 through a ranking system to prioritize funding allocated to projects under the Species at Risk Management Program. The ranking system gives first priority to projects that assist DFO in meeting its numerous regulatory requirements under the SARA and to reduce backlogs. The evaluation found that while the advice provided is often perceived as high quality, as reliable and accessible, SSAR cannot meet the demand in terms of number of requests met, with lack of resources being a key factor. SSAR has legal obligations that must be met, and in some instances, assessments of some species may be delayed because of delays with data, as COSEWIC only meets twice a year, this could delay this assessment. In some specific cases, the information is not collected, or there is simply no expertise in the department in the request area. Although there are guidelines for the development Recovery Potential Assessments (RPAs), interview respondents indicated there are also inconsistencies in terms of the format of the RPAs. Many species at risk managers have indicated that they often complete the information provided by SSAR with those from other sources such as provincial governments, university and external contractors.

SSAR also provides advice to other jurisdictions, including the provinces/territories and municipalities; while Non-Governmental Organizations (NGO) also use the information for their own purposes.

Efficiency and Economy

As for many other DFO programs, governance of Species at Risk involves national and regional structures. The various functions under SSAR are distributed between National Headquarters (NHQ) and regional offices. The research teams work in regional offices under the direction of six Regional Directors General and Directors. The advice request system (including a central database) is managed at the national level by CSAS which receives a prioritized list of requests from the regional offices.

However, key informants identified concerns about governance in terms of coordination and roles and responsibilities. There are multiple offices and sectors within DFO (e.g. Ecosystems and Fisheries Management (EFM), Program Policy and Science sectors as well as regional offices) that are involved in the delivery of SARA programs and activities. Roles and responsibilities between these offices and sectors in relation to SSAR are not always clearly understood, such as for the approval of reports. Specifically, reviewing and approving the science contained in reports, for example, recovery strategies, action plans, and management plans, has proven to cause confusion.  This confusion did create situations where documents were not shared when it was needed or they were not shared with the correct individual, between regions and NHQ, resulting in inefficiencies especially for a program that operates under tight timelines. Species at risk issues often span over multiple regions/zones and the coordination among regions adds to the complexity of the governance structure.

As such, it is recommended that SSAR clarify and document its roles and responsibilities and approval processes between SSAR and Species at Risk Management Program, at NHQ and in the regions, and between SSAR at NHQ and Science in the regions.

Recommendation 1:
It is recommended that the Assistant Deputy Minister, Oceans and Science, clarify and document the SSAR roles and responsibilities and approval processes, between SSAR at the NHQ and in the regions as well as between SSAR and Species at Risk Management Program at the national headquarters and in the regions.

Currently, SSAR does not have a Performance Measurement Strategy for its sub-activity. Also, it is difficult to differentiate FRS and SSAR requests in the CSAS databases. A number of activities and outputs are not captured in the current system, including publications in peer reviewed science journals, non-CSAS advice and participation in consultations led by Ecosystems and Fisheries Management (EFM) and Species at Risk Management. This situation makes it difficult to fully assess effectiveness as a result of not having complete performance measures. For example, legal obligations under SARA in terms of scientific advice could form the basis of the performance measurement system.

SSAR does not have a Performance Measurement Strategy for its sub-activity. As of the 2011-2012 PAA, SSAR is no longer a program sub-activity, but is included in the Species at Risk Management Program - program activity. As such, activities, outputs and outcomes (where applicable) of SSAR should be included in the Performance Measurement Strategy of the Species at Risk Management Program - program activity to accurately reflect all the activities, outputs and outcomes within DFO, to support the implementation and delivery of the SARA. The Performance Measurement Strategy for the Species at Risk Management Program – program activity is scheduled to be completed in early 2011 as per DFO’s Performance Measurement Action Plan.

Recommendation 2:
It is recommended that the Assistant Deputy Minister, Oceans and Science ensure that Science for Species at Risk Program collaborate with the Species at Risk Management Program, in the development of the Performance Measurement Strategy for the Species at Risk Management Program in order to accurately reflect all the activities within the department (including Science activities) with regard to supporting the implementation and delivery of the SARA.

Evidence indicates that SSAR and the species at risk requests in relation to SARA have gradually increased (e.g. SSAR CSAS publications, CSAS scientific papers, etc.) since SARA’s implementation which has impacted the level of resources of FRS. For example, DFO is exploring ways of cost-sharing services, where species at risk programs have contributed to field costs (e.g. Canadian Coast Guard fees for vessel use) with their Operations and Maintenance (O&M) funds. In addition, there is evidence that some FRS Full Time Equivalent (FTEs) work on SSAR activities. Finally, regions lack expertise and data on freshwater species, but some have leveraged provincial expertise to help meet the species at risk demands.

1. Introduction

1.1 Context of the Evaluation

This report presents the results of the evaluation of the Science for Species at Risk (SSAR) Program. As identified by the Treasury Board Policy on Evaluation (2009), all direct program spending must be evaluated every five years. This evaluation was slated in the Department of Fisheries and Oceans (DFO) multi-year departmental evaluation plan and focuses on the core issues in assessing value for money: relevance and performance, including effectiveness, efficiency and economy.  

The timeframe for this evaluation, a first for the SSAR Program, covers the period from 2005-06 to 2010-11. The evaluation assesses the extent to which SSAR Program has achieved its outcomes stemming from activities as stated in its Logic Model (section 2.4). The evaluation is inclusive of the National Headquarters (NHQ) as well as DFO’s six regions: Maritimes, Quebec, Central and Arctic, Pacific, Gulf and Newfoundland and Labrador. The data collection and analysis was undertaken between September 2010 and January 2011, with report writing and approvals occurring from January to June 2011. The data collection for the SSAR evaluation occurred concurrently with that of Fisheries Resources Science (FRS) to provide a more efficient approach to the evaluations. The FRS evaluation was approved by DFO’s Departmental Evaluation Committee on February 15, 2011.

1.2 Objectives of the Evaluation

The main objective of this evaluation is to determine to what extent the SSAR Program is relevant, is managed effectively and efficiently and whether it had achieved its stated objectives. As such, the evaluation examines the extent to which the SSAR Program demonstrates relevance and performance (including effectiveness, efficiency and economy).  This evaluation assesses the extent to which the program demonstrates these elements through a set of evaluation issues and questions, the collection of data and analysis of the evidence in order to draw findings and conclusions, and finally develops recommendations to improve the program going forward.

2. Program Profile/Background

2.1 Program Profile

The Species at Risk Act (SARA), 2003, was created to prevent wildlife species from becoming extinct; to provide for the recovery of threatened1, endangered2 and extirpated3 species; and to manage species of special concern to prevent them from becoming endangered or threatened (SARA, S.6). Responsibility for the implementation of SARA is assigned to Environment Canada (EC), Fisheries and Oceans Canada (DFO) and Parks Canada Agency (PCA).

DFO has the authority for aquatic species under SARA. Within DFO there are multiple branches and sectors (e.g. Ecosystems and Fisheries Management (EFM), Program Policy and Science sectors as well as are regional offices) involved in the delivery of SARA. However, the focus of this evaluation is the science activities for species at risk within DFO4.

As such, DFO’s SSAR program undertakes targeted research and monitoring of species at risk and their habitats, and  provides information to support assessments of the conservation status of the species (SARA s. 14-26, 128); departmental listing recommendations and government decisions in this regard (SARA 27(2)(b)); assessment of recovery feasibility; advice with regards to recovery goals and objectives; and identification of critical habitat and threats affecting the species and its habitat in the form of Recovery Potential Assessments (RPA). In addition, DFO’s SSAR program provides standards and protocols to help guide community based stewardship programs.

SSAR also provides data and expertise to the Committee on the Status of Endangered Wildlife in Canada’s (COSEWIC) assessment process, and works with provincial, territorial and Wildlife Management  Board  experts  on  species  at  risk,  under  the  general umbrella of the National Accord for the Protection of Species at Risk5, as well as the National Framework for Species at Risk Conservation6.

SSAR also reports on the general Status of Aquatic Wildlife (SARA sec. 128) every five years and additionally reports on the Recovery Progress for Species at Risk (SARA sec. 46 and 55). Specifically, SARA sec. 46 indicates that “The competent minister must report on the implementation of the recovery strategy, and the progress towards meeting its objectives [comes into force summer of 2011], within five years after it is included in the public registry and in every subsequent five-year period, until its objectives have been achieved or the species’ recovery is no longer feasible…”

SSAR directly contributes to DFO’s 2010-11 strategic outcome Healthy and Productive Aquatic Ecosystems (HAPAE) and contributes to the strategic outcome of Sustainable Fisheries Aquaculture (SFA) as demonstrated below in the description of the program activities.

The SSAR program is an internal program that primarily supports other DFO Programs with sound science advice; however, the outputs of the program also ensure that the public has access to science information and advice that guide decision making with the intent to share knowledge.  Specifically, the DFO SSAR Program contributes to the departmental Species at Risk Management sub-activity, including the protection, recovery and monitoring and evaluation activities (see 1, 2, 3 below) and to several other Program Activity Architecture (PAA) program sub-activities (see 4, 5, 6, 7 below) by sharing knowledge. For example, these include:

  1. Protection of Species at Risk (under the strategic outcome of HAPAE): Through production of RPA which are departmental pre-requisite to listing recommendations (Protection of Species at Risk) and recovery planning.

  2. Recovery of species at risk (under the strategic outcome of HAPAE): conducting critical habitat studies to allow identification of critical habitat in recovery strategies or action plans.

  3. Monitoring and Evaluation (of species at risk) (under the strategic outcome of HAPAE): monitoring species at risk populations and their habitats in order to report on progress towards meeting strategy objectives every 5 years as required under SARA.

  4. Fisheries Resource (under the strategic outcome of SFA): SSAR strongly benefits from the FRS species research, monitoring, stock assessment activities before species are formally identified as being at risk. In addition, both the FRS and SSAR share common activities and outputs. For example, FRS and SSAR have the same activities of research, monitoring and providing advice on the status of aquatic and fisheries resources, which include species at risk. They have common outputs, for example scientific advice to support sound policy development and implementation.

  5. Habitat Management (under the strategic outcome of HAPAE): when SSAR gathers data and produces advice pertaining to critical habitat identification and threats affecting critical habitat for species at risk, it supports the Habitat Management sub-activity.

  6. Ocean Management (under the strategic outcome of HAPAE): data that is gathered by SSAR for marine species, for example, areas of concentrated whale activities, etc. contribute to the Oceans Management sub-activity.

  7. Aquaculture Management (under the strategic outcome of SFA): when environmental assessments are carried out, species at risk are taken in consideration, which indirectly contributes to the Aquaculture management sub-activity.   

2.2 Expected Results

The primary results and indicators articulated for SSAR are in the DFO PAA. The following tables describe the main expected results identified for the program for 2009-2010 as well as for 2010-11.

Table 1: Expected Results of SSAR
Expected Result/Output Performance Target Result Achieved

Increased knowledge and information on aquatic species at risk for decision-makers

Number of requests for science advice on aquatic species at risk

Maintain 4-year average

Completed 73% of the requests for science advice on aquatic species at risk

Number of Canadian Science Advisory Secretariat (CSAS) publications on aquatic species at risk that are posted on the DFO website

(NEW) Maintain 5 year average

43 publications on aquatic species at risk were posted on the DFO website (39 in 2008-09) in response to requests submitted to CSAS for science advice on aquatic species at risk

Number of publicly available products on aquatic species at risk completed by DFO Science

(NEW) Maintain 5 year average

19 peer-reviewed publications on aquatic species at risk were completed and made publicly available

Source: DFO 2009-10 Departmental Performance Report (DPR)

Table 2: Expected Results of SSAR 7:
Expected Result Performance Indicator Target

Decision-makers have science information to manage species at risk

Percentage of requests for science advice on species at risk, approved by senior management, that are completed

90%

Percentage of publications for each completed DPR request on species at risk that is posted on the CSAS website

90%

The public has access to science information and advice on species at risk

Number of documents on species at risk downloaded from the CSAS web site

Maintain 5 year average (2006-11)

Science advice on Species at Risk

# of species at risk for which peer reviewed science or science advice is provided

Maintain 5 year average (2006-11)

# of CSAS publications on species at risk that are posted on the DFO Science website

Maintain 5 year average (2006-11)

Number of publicly available products on species at risk completed by DFO Science (other than CSAS publications)

Maintain 5 year average (2006-11)

Source: DFO 2010-2011 Reports on Plans and Priorities (RPP)

Prior to 2009-10, SSAR was under the Program Activity of Science for Sustainable Fisheries and Aquaculture (i.e. within the same Program Activity as FRS). In 2010-11 SSAR was moved a sub-activity under the Science for HAPAE Program Activity. In DFO’s 2011-12 PAA, FRS and SSAR are included in the Integrated Fisheries Resource Management and the Species at Risk Management program activity and are placed under new strategic outcomes, as demonstrated in table 3 below.

Table 3: Changes of DFO’s Strategic Outcomes 2009-10 to 2011-12
Strategic Outcomes
2009-10 2010-11 2011-12
Sustainable Fisheries Aquaculture

FRS and SSAR

FRS

Healthy and Productive Aquaculture Ecosystems

SSAR

Economically Prosperous Maritime Sectors and Fisheries

Integrated
Fisheries
Resource
Management

Sustainable Aquatic Ecosystems

Species at Risk Management

2.3 Governance and Resources of the Program

SSAR reports to the Ecosystems Science Director General within the Oceans and Science Sector of DFO. The SSAR Program has personnel both at the NHQ and in all of the DFO regions (Central and Arctic, Maritimes, Gulf, Newfoundland and Labrador, Quebec, Pacific).

The following table summarizes SSAR financial information from 2007-08 to 2010-118:

Table 4: SSAR Program (Sub-activity) Total Cost
Total Program Cost ($000)
Spending Profile 2007-2008 2008-2009 2009-2010 2010-2011 Total

Total

2,759.60

2,493.70

3,119.60

3,328.05

11,700.95

Source: Main Estimates

2.4 Logic Model of the Program

Due to the output and activity based nature of the results identified for FRS and SSAR, a logic model was developed in December 2010 to capture both aspects of science in a more outcomes-based manner. The following diagram is the logic model developed for both FRS and SSAR.

Graph 

3. Methodology and Evaluation Design

3.1 Evaluation Design

For this evaluation a non-experimental design was used, in which measurements are taken after the program has been implemented. This model was chosen for a number of reasons.  First, the SSAR Program is a full coverage program, intended to be delivered across Canada and cannot be withheld in any region due to the legal obligations under SARA; therefore, a control group could not be established. Secondly, there were no measures in place before the SSAR Program was introduced so only post-test measurements were used.

Although the non-experimental design lacks scientific rigour, the rigour of this design was increased through the use of multiple lines of evidence and triangulation across those lines of evidence to reduce the occurrence of bias and increase validity. This enabled evaluators to make linkages, and logically argue that results can be attributed to the program.

3.2 Methodology

A multiple-lines-of-enquiry approach was used to study the issues and questions that were the focus of this evaluation. This approach allows for findings and conclusions to be drawn from more than one source of information as a means to ensure the reliability of the information. The evaluation questions for SSAR are addressed through the following lines of evidence: document review; key informant interviews; focus groups; survey of users of science for species at risk; and a case study. This section outlines the scope and methods of our approach, evaluation questions, the methodological approach, analytical methods as well as the constraints and caveats of the evaluation.

As previously explained in section 2.1, SSAR and FRS have similar activities, outputs and outcomes. As such, the same lines of evidence were applied in both evaluations and field work was undertaken concurrently, where possible.

Specifically, the methodologies for assessing the relevant evaluation questions posed for SSAR are primarily key informant interviews and focus groups. Some survey questions were asked of DFO clients to determine how satisfied they are with SSAR during the FRS survey. One case study, the Pacific Region killer whale science undertaken in the FRS evaluation addresses the linkages between SSAR and FRS.

Document Review
A review of documentation was conducted to assess most evaluation issues. This line of evidence was chosen because the strengths of documentation review is that it can provide historical insight into the development and delivery of a program, verify data from other methodologies and provides primary data on issues such as relevance. The evaluation team reviewed all relevant documentation and administrative files available. The following types of documents were reviewed: government-wide documents (e.g. federal legislation pertaining to fisheries, federal speeches from the Throne, federal budgets, etc.), and departmental-level documents (e.g. Departmental Performance Reports (DPRs), Reports on Plans and Priorities (RPPs), DFO policies, DFO and CSAS website and publications, etc.).  The departmental documents also include administrative data from CSAS.

Interviews
Key informant interviews were chosen as a line of enquiry to gather in-depth information for evaluation purposes, including views, explanations, examples, contextual and factual information that address the evaluation questions. Most interviews in the NHQ, Newfoundland/Labrador region, the Gulf and Maritimes regions, and the Pacific region were conducted in person. Interviews for other regions were conducted by telephone. Approximately 26 interviews with DFO staff including program staff (within science) and DFO client staff, such as individuals from the Species at Risk Management Program were conducted between November 2010 and January 2011. Key informant interviews were conducted to assess most evaluation issues. Key informants were selected based on their knowledge of the subject matter.

Survey
A survey of clients who use FRS advice and information was conducted to gather information for evaluation purposes, including views that address the evaluation questions to gather a deeper insight within a target population. Because FRS and SSAR have similar clients, SSAR questions were asked during the FRS survey. The purpose of the survey was to determine the use of FRS and SSAR advice and information, the level of client satisfaction with the outputs that FRS and SSAR provides, and perceptions about the utility and influence of FRS and SSAR advice and information on their decision-making process.

The primary source of survey respondents was obtained from the CSAS database on annual science requests for advice (for the year 2009-10). Since, the database does not contain names of clients, the evaluation team made a request to the regions to obtain names for each of the requests. Regions provided names between November 2010 and January 2011. Approximately 117 clients received an email to complete an online survey. The final response rate for the survey was 53% (n=62). The number of unique email addresses was used as some respondents received more than one email message because they had made more than one request for science. Each respondent was asked a core set of questions as well as specific questions for up to two requests they were involved in. Of the 117 clients surveyed for FRS, 62 individuals responded to sections relevant to their situations and 21 of the 62 survey respondents, indicated that they requested information relating to species at risk. The information from these 21 respondents was used as evidence for the SSAR evaluation.

Case study
For this evaluation, case studies were chosen to focus more in-depth on the contextual issues affecting the delivery and results of FRS and SSAR through a specific example. Case studies were conducted for the FRS evaluation to provide a better understanding of the background, as well as the factors supporting the achievement (or non-achievement) of results for the program. One case study, the Pacific Region killer whale (a species at risk) was undertaken in the FRS evaluation addresses the linkages between SSAR and FRS. Specifically, the case study was chosen to demonstrate the linkages between two science programs that operate and collaborate in order to accomplish assessment/research on species, including species at risk.  For example, the killer whale (a species at risk) has an impact on how fisheries are managed because of their ecosystem linkages, in particular, their predator-prey dietary relationship with certain species. The resident killer whale targets primarily Chinook salmon, even though other forms of salmon are available. As such, the case study demonstrates the lessons learned and the challenges associated with a species at risk, which is researched and monitored by SSAR and its prey, which is not a species at risk and is researched and monitored by FRS science program. In addition, the case study informed the evaluation with: background information about the fishery and the specific challenges surrounding it; FRS and SSAR program activities, processes, and budget/costs related to the fishery; FRS and SSAR results achieved to date and their impacts on decisions about the fishery; and best practices and lessons learned.

The primary methods used in the case studies were key informant interviews, as well as a review of relevant documentation pertaining to each case. Interviews were conducted with regional FRS staff. Key informants were selected based on their knowledge on the subject matter. The case studies were carried out between November 2010 and January 2011.

3.3 Evaluation Questions

The evaluation questions were determined on the basis of the Treasury Board Policy on Evaluation (2009), by reviewing documents, and the results of the planning phase interviews with key program contacts. Annex A features an evaluation matrix organized by evaluation issue (relevance and performance including effectiveness, efficiency and economy) and outlines the evaluation questions.

3.4 Analytical Methods

The analysis methods used for this evaluation were tailored to the nature and availability of the data to be gathered, which were in turn linked to the evaluation questions. Extensive use of triangulation was used as an analytical method. In the social sciences, triangulation means that more than two methods are used in a study in order to corroborate findings. Document review, interviews, a survey and a case study were used as methodologies to support triangulation in this evaluation.

3.5 Methodological Constraints and Caveats

Over the course of the study some methodological constraints were observed in the evaluation. In order to minimize the impact on the results of the evaluation, employed information was collected from a variety of data sources, such as document review, interviews, a survey and a case study. In other words, various data sources were used to arrive at the same conclusions, thereby reinforcing the assessment as to their validity. These constraints did not significantly affect the validity or accuracy of the evaluation results.

  • Organizational structure. SSAR is not an organizational entity within DFO Science (science sector), rather it is a sub-activity defined in DFO’s Program Activity Architecture (PAA). There is a challenge in identifying which of DFO Science’s resources, activities, outputs and outcomes are aligned with SSAR (e.g. many key informants did not know of SSAR; they were only aware of DFO Science). In addition, this evaluation occurred in combination with the FRS evaluation. As there is a likelihood of overlap between DFO Science in general and SSAR, the qualitative evidence was used in combination with other lines of evidence to ensure neutrality for reporting purposes. Therefore, the limitations of the organizational structure were mitigated by the use of multiple lines of evidence and therefore have not impacted the validity of the findings.

  • Performance information. There was limited performance data available for the evaluation. To ensure the validity of the findings the evaluation supplemented for the lack of performance measurement by using additional lines of evidence and the use of triangulation.  It is not expected to have negatively impacted the validity of the findings.

  • Methodological limitations:

    • Document review: When reviewing the financial data of SSAR it did not include a breakdown of budget allocation by activities, outputs and outcomes, an analysis of research costs was not available and no comparative program was found. As such, this has limited the analysis of the efficiency and economy of the SSAR Program. To counter these limitations the evaluation team used alternate quantitative (e.g. survey, etc.) and qualitative measures (e.g. interviews, etc.) to assess the efficiency and potential impacts of the programs. The use of standardized tested tools to collect this data should ensure that these limitations have not negatively impacted the validity of the findings.

    • Interviews: Due to time and resource constraints, there was a limit in terms of how many key informants outside of DFO (i.e. external clients) could be interviewed and surveyed. In this context, both the client and key informant interviews were chosen based on their extensive experiences.  This has limited the number of perspectives and hence there is a risk of bias in their responses. As there is a likelihood of subjectivity, to ensure the validity of findings, the qualitative evidence was triangulated with other lines of evidence (e.g. document review, case study, etc.) to ensure neutrality for reporting purposes. However, the limited data from external interviews is not expected to negatively impact the validity of the findings. Given the wide range of external stakeholders that are particularly affected by DFO decisions and science advice/information, further study may be warranted to fully examine the views of external stakeholders and their opinions of DFO science efforts.

    • Survey: It should be noted that due to the small number of respondents (21), that responses are not representative of all potential users and findings from the survey are more suggestive in nature.  Readers are to use caution in interpreting results from surveys.

    • Case study: Due to the small number of individuals interviewed, responses are not statistically representative for generalizing the findings to all users of the program's activities. In this context, the key informants were chosen based on their extensive knowledge and experience with the case being examined.  As there is a likelihood of subjectivity, the qualitative evidence was used in combination with other lines of evidence to ensure validity for reporting purposes. 

4. Major Findings

4.1 Relevance

Relevance Question #1: To what extent are SSAR mandate and activities aligned with Government and Department priorities and objectives?

Key findingSSAR is aligned with Government of Canada and DFO priorities and objectives to protect wildlife species under the Species at Risk Act (SARA).

Findings
SSAR is highly consistent with the Government of Canada’s legal obligation to protect wildlife species under the SARA, and is consistent with DFO’s objectives of supporting a healthy and productive aquatic ecosystem (HAPAE). For example, SSAR supports DFO’s objectives to identify and protect threatened aquatic species, by undertaking research and monitoring activities on species at risk to determine their status, recovery potential, and critical habitats.

It also addresses DFO’s responsibilities to produce sound science to support management decisions for aquatic species at risk. In addition, SSAR responds to Canada’s international commitments, such as to the United Nations Convention on Biodiversity, the Commission for Environmental Cooperation (CEC), and the Convention on International Trade in Endangered Species (CITES)9 that depend on sound biological understanding and principles.

Evidence
Globally, SSAR is aligned with DFO’s 2010-11 strategic objective HAPAE. It also links into the DFO’s sustainable development and fisheries mandate. SSAR is aligned with DFO’s legal requirements as the Department responsible for implementing the SARA with respect to aquatic species. SSAR supports DFO’s objectives to identify and protect threatened aquatic species, by undertaking research and monitoring activities on species at risk to determine their status, recovery potential, and critical habitats. SSAR provides standards and protocols to help guide community based stewardship programs, and supports DFO requirements to report on SARA obligations on an ongoing basis.

A majority of respondents interviewed indicated that SSAR was consistent with DFO’s HAPAE strategic objective, and consistent with the overall sustainable development and conservation mandate of the Department. A majority of respondents also mention that the Government of Canada’s obligations under the SARA is predicated on good science. Evidence from the case study also indicates that “proper science is neceSSARy”. For example, in the case study found that scientific information was necessary to identify critical habitat of the resident killer whale. SSAR also responds to Canada’s international commitments, such as to the UN Convention on Biodiversity and the CEC. The respondent group also confirmed that science is also an important contributing factor in meeting Canada’s international obligations, including those under the CITES, that depend on sound biological understanding and principles.

Relevance Question #2: Does SSAR continue to address the needs of Canadians?

Key finding:  There is a continued need for SSAR to address the needs of Canadians, specifically its DFO clients. In particular SSAR provides the science information to support the Species at Risk Management Program Activity in delivering the legal obligations under SARA.

Findings
There is a continued need for SSAR because the program addresses the needs of Canadians, specifically its DFO clients. In particular SSAR addresses the needs of it’s clients in providing the science base to ensure the protection of aquatic species at risk, determining the recovery potential, critical habitats, action and management plans, monitoring, and supporting listing decisions and development of recovery strategies as part of the legal obligations under SARA and in support of the Species at Risk Management Program Activity.

Evidence
A majority of SSAR respondents (which include respondents from SSAR and the Species at Risk Management Program) indicated that there was a need for SSAR. DFO has legal obligations under SARA to protect listed species, develop recovery strategies, etc. and these measures are science-based. Before listing, SSAR will provide science data and information to COSEWIC, as listing recommendations and decisions are science-based. DFO conducts these assessments through SSAR prior to COSEWIC’s work on specific species. DFO Science plays a key role in assessing populations, and in understanding the ecosystem and the linkages between various factors and population levels.

The internal clients of SSAR (DFO managers) also confirmed the need for SSAR. More concretely, one example provided explained that the entire Species at Risk Management Program Activity at DFO is based on the advice provided by SSAR. The need for data is extensive: Many species are of concern, including marine and freshwater commercial and non-commercial species, marine mammals and other aquatic species living in various habitats. The Species at Risk Management Program Activity depends on scientific data to monitor the status of various species and identify critical habitats. The available science information also allows DFO to conduct “Pre-COSEWIC” assessments. COSEWIC relies on DFO data (among other sources, including the best available scientific information as well as community knowledge) to produce status reports for aquatic species and make recommendations. It was explained that following COSEWIC status recommendations, DFO needs to assess the potential for recovery, and the expectations on scientific information are high for recovery potential assessments purposes. Science is also required to answer questions about recovery targets, critical habitats, and provide scope for activities that may harm species at risk (e.g. help determine also if some fishing permits are issued or not). Senior level decision-makers at DFO also rely on science for listing decisions. The science information and advice are also key for DFO staff that are participating on committees or teams that are involved with recovery efforts. DFO also works with provincial and territorial authorities and relying on science and science-based advice to meet their commitments towards these other levels of government. All federal departments could use the science information products from SSAR, if they have activities that may impact species at risk. For instance, Environment Canada (EC), Transport Canada (TC), Department of National Defense (DND), Public Works Government Services of Canada (PWGSC), etc. may have activities or land that includes species at risk. In this case they use the science information produced by SSAR to respect SARA as well as for any study on impact. Federal departments also use the science information produced by SSAR when they submit projects for the Interdepartmental Recovery Fund (IRF), a program under SARA

Relevance Question #3: Is SSAR consistent with federal and DFO roles and responsibilities?

Key findingSSAR is consistent with federal and DFO roles and responsibilities with respect to protecting species at risk.

Findings
SSAR is consistent with the federal and DFO roles and responsibilities with regards to SARA. DFO science provides the science information and advice to support DFO’s responsibilities towards aquatic species at risk.

Evidence
DFO’s responsibilities are defined under the SARA, which was developed to protect wildlife species from becoming extinct, to provide for the recovery of threatened, endangered and extirpated species, and to manage species of special concern to prevent them from becoming endangered or threatened10. Internally SSAR works with other programs such as the Species at Risk Management Program, FRS, Habitat Management, Ocean Management and Aquaculture Management as well as with regional offices.

DFO Science provides data and expertise to the COSEWIC11 assessment process, and works with provincial, territorial and Wildlife Management Boards experts on species at risk, under the general umbrella of the National Accord for the Protection of Species at Risk, as well as the National Framework for Species at Risk Conservation. Science information and advice is also provided in relation to activities likely to be in violation of SARA’s prohibitions before considering issuing permits, or as part of the Canadian Environmental Assessment Act (CEAA) process.

The following graph visually demonstrates the relationships between SSAR and other programs internal and external to DFO that supports SARA, which is a federal responsibility and for which DFO is responsible for aquatic species.

Graph

4.2 Effectiveness


Effectiveness Question #4: To what extent is SSAR achieving its results? And to what extent have the intended outputs been produced?

Key finding: SSAR science advice and information has been used as intended (i.e. inform decision-making) to a large extent. However, overall, science requests are exceeding the capacity of SSAR to meet them, and there is expressed dissatisfaction about timeliness and the extent of support provided.

Findings
The types of outputs that both FRS and SSAR produce may include the following products and services:tables and figures;research publications;peer review meeting reports;advice;policy documents; reports to domestic and international organizations such as regional fisheries management groups and international forums; and presentations and communication products.

A number of these formal and informal outputs are not captured within the CSAS process or in science journals, especially those related to data management, administration, communication, and informal outputs that scientists provide to clients (both internal and external). Therefore, the amount of outputs achieved by SSAR goes beyond what can currently be quantifiably measured.

SSAR play key roles in providing information on species, development of RPAs, recovery planning, identification of critical habitat, monitoring, and input for listing decisions. The science information is used and plays a key role in developing the RPAs. According to interview respondents, the RPAs by species is a key deliverable from the Science Branch.

Overall, the DFO managers (clients) of SSAR were satisfied with the quality of science information and advice provided. They confirmed that science was used for listing decisions and recovery planning and implementation, as well as for many other reasons, including: ranking species at risk for management purposes; identifying thresholds for habitat protection and restriction areas; identifying other recovery actions; determining mitigation and protection measures; developing public educational materials; determining how to monitor catches and activities which might impact negatively on species at risk; and developing better/new gear to protect other species (e.g., traps and nets).

However, there is evidence that SSAR is not meeting the demand for advice and science information. While the advice provided is often of high quality and is perceived as reliable and accessible, SSAR cannot respond to all requests made. Specifically, SSAR completed 73% of the requests for science advice on aquatic species at risk, as specified in the 2009-10 DPR. The requests for science advice originate from the Species at Risk Management program, which have been organized since 2010-11 through a ranking system to prioritize funding allocated to projects under the Species at Risk Management Program which gives first priority to projects that assist DFO in meeting its numerous regulatory requirements under the SARA and to reduce backlogs.

The lack of resources is a key factor for SSAR not meeting the number of requests for science advice. SSAR has legal obligations to meet, and in some instances, some assessments of species may be delayed because of delays with data or lack of data. As well, as COSEWIC only meets twice a year, this could delay their assessments. In some cases, the information is not collected, or there is simply no expertise within the department in the request area.

There are also inconsistencies in terms of the format of the RPA reports. For example, variable recovery targets and population levels which cannot be breached are not always indicated. Many Species at Risk managers say that they sometimes complement (for additional information) the information provided by SSAR with those from other sources, such as with information from provincial governments, universities and contractors.

Evidence

Outputs
Quantitative data on the number of science requests and publications was obtained from DFO Science to determine if FRS and SSAR achieved expected outputs. One of the indicators that measures the expected result “science advice on Species at Risk” (as indicated in Table 2) is the number of species at risk for which peer reviewed science or science advice is provided. As such, table 5 below describes the number and percentage of science advice and peer review requests received by the CSAS 12 in 2009-10, the first year in which DPR established its database of science requests13. The data shows that for SSAR, 47 requests 20% of the total number were submitted, of which only 53% (25) were undertaken.

For SSAR requests undertaken, NHQ, Quebec Region and Gulf Region again undertook all their SSAR related queries (100%14), followed by the Pacific Region (83%), Maritimes Region (71%), and Newfoundland/Labrador Region (50%), Central and Arctic Region (26%).

Table 5: 2009-2010 peer review requests 15
Region Total peer review requests16 Planned Science requests related to SSAR17 Planned
ON OFF MAYBE18

 

ON OFF MAYBE
NHQ

27

15

6

6

1

1

0

0

Pacific

49

25

22

2

6

5

1

0

Central and Arctic

77

19

38

20

34

9

13

12

Quebec

14

14

0

0

3

3

0

0

Gulf

11

11

0

0

1

1

0

0

Newfoundland and Labrador

21

11

7

3

2

1

1

0

Maritimes

42

23

5

14

7

5

1

1

Total

241

118

78

45

47

25

16

13

Percentage

100%

49%

32%

19%

100%

53%

34%

28%

Source: CSAS

The 2009-10 DPR indicates that there was 19 peer-reviewed publications on aquatic species at risk were completed and made publicly available.

Table 6 below described the number of CSAS publications produced from 2004 to 2008 provided by CSAS (including science advisory reports, research documents, proceedings, and special response reports). Disregarding preliminary data for 2008, the analysis reveals that CSAS publications species at risk have increased from 40 to 51. It should also be noted that the second largest number of CSAS publications produced in any one year are for species at risk (13 to 23%). This information is used to report on the expected result of the SSAR Program, specifically that decision-makers have the science information needed to manage species at risk and science advice on species at risk.

Table 6: Number and Percent of CSAS Publications by PAA
PAA 2008 (preliminary) 2007 2006 2005 2004
Fisheries Resources 112 62.6% 115 51.1% 109 54.2% 128 65.0% 148 59.9%
Species at Risk 24 13.4% 51 22.7% 36 17.9% 30 15.2% 40 16.2%
Aquatic Invasive Species 5 2.8% 9 4.0% 2 1.0% 3 1.5% 5 2.0%
Aquatic Animal Health   0.0% 2 0.9% 1 0.5% 1 0.5% 2 0.8%
Sustainable Aquaculture Science 4 2.2% 1 0.4% 6 3.0% 6 3.0% 2 0.8%
Genomics and Biotechnology   0.0% 0 0.0% 2 1.0% 1 0.5% 1 0.4%
Science Renewal   0.0% 0 0.0% 0 0.0% 0 0.0% 0 0.0%
Fish Habitat 13 7.3% 14 6.2% 15 7.5% 5 2.5% 10 4.0%
Aquatic Ecosystems 7 3.9% 19 8.4% 18 9.0% 12 6.1% 20 8.1%
Ocean Climate 14 7.8% 14 6.2% 12 6.0% 11 5.6% 19 7.7%
Total 179   225   201   197   247  

Source: CSAS

Table 7 shows two particular types of fish population science papers that have been produced by DFO internally (CSAS): those related to stock assessments and those related to species at risk19. Data for DFO fish population science papers that are related to species at risk, indicate that DFO has produced 113 CSAS papers from 1998 to 2008. This has increased from one publication in 1999 to a high of 30 in 2007.  Performance information on the number of science papers that have been produced by CSAS on species at risk is needed to measure the science advice on species at risk expected result. In addition, the 2009-10 CSAS indicates that 43 publications on aquatic species at risk were posted on the DFO website (39 in 2008-09) in response to requests submitted to CSAS for science advice on aquatic species at risk.20

Table 7: Total annual number of DFO CSAS scientific papers pertaining to Species at Risk and Stock Assessment (subset of Fish Population) – 1998-2008
Topic / keywords 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 1998-
2008
Species at Risk

 

1

2

 

3

11

24

8

15

30

19

113

Stock Assessment

103

160

119

124

96

99

93

91

72

79

82

1,118

Source: CSAS

According to the interview respondents, the types of outputs that both FRS and SSAR produce may include the following products and services:data reports (tables and figures);scientific and technical reports;scientific literature (research publications);peer review meeting reports;advice (formal reports);question and answers (informal);policy documents; reports to international organizations such as regional fisheries management groups (e.g. Northwest Atlantic Fisheries Organization (NAFO)) and domestic and international forums (e.g. International Conference on Energy and the Environment (ICEE)); and presentations and communication products.

Overall, as identified in the 2009-10 DPR, SSAR completed 73% of the requests for science advice on aquatic species at risk. Performance information from previous years on this particular indicator was not being measured in the previous DPRs. The requests for science advice originate from the Species at Risk Management program, which have been organized since 2010-11 through a ranking system to prioritize funding allocated to projects under the Species at Risk Program. For example, first priority is given to projects that assist DFO in meeting its numerous regulatory requirements under SARA and to reduce backlogs. The ranking system also takes into consideration the ever-growing number of species being added for consideration. Thus a larger proportion of the total national funding envelope is set aside to meet increasing and immediate legal obligations. The priority scheme leaves some flexibility to address backlogged species while allowing for some progress with new species and avoiding future delays.

A number of these formal and informal outputs are not captured within the CSAS process or in science journals, especially those related to data management, administration, communication, and informal outputs that scientists provide to clients (both internal and external). Therefore, the number of outputs achieved by FRS and SSAR go beyond what can be currently quantifiably measured.

Outcomes

The SSAR is used for various purposes, mostly to develop RPAs following COSEWIC status recommendations and to guide listing or de-listing decisions. RPAs include various impact analyses and other items; this and other information guide the listing decision. The SSAR information is used, and plays a key role, in developing the RPAs. According to interview respondents, the RPAs are key deliverables from the Science sector.

Some respondents explained that SSAR and FRS overlap only to a certain degree. For example, science about commercial species (including stock assessments) may or may not be used for work on species at risk. There is also a qualitative difference in the sense that the advisory data is different in terms of content; while fisheries management seeks information and advice on harvest limitations, species at risk information is based on, and responds to, COSEWIC assessments, which may be considerably different in terms of scope (local vs. regional state of stocks) and criteria involved. The complementarities between the SSAR and FRS programs can be further observed in the case study on killer whale science. SSAR looks at science information on the killer whale (such as critical habitat), but FRS assesses the amount of Chinook salmon that resident killer whales consume currently and might need in 10 years, as well as identifying the stocks of origin that may be important to resident killer whales.

SSAR was noted as extremely valuable as there is resistance from various stakeholders to have species listed, especially commercial species. Developing science information and projections for the purposes of Species at Risk Management is challenging, according to FRS and SSAR respondents. A decision to list or de-list a species (listing decisions) is impacted by the science advice, but also by other inputs such as socio-economic studies. Some FRS and SSAR interview respondents expressed some frustration in this regard. A few respondents noted that research teams have expressed cynicism in the process as many listing decisions do not solely reflect the science. That is, other considerations, such as mentioned above, have impacted the final listing decision.  

Overall, the DFO managers (clients) of SSAR were satisfied with the science information and advice provided. They confirmed that science was used for RPAs and listing decisions purposes, but also for many other reasons, including: ranking species at risk for management purposes; identifying thresholds for habitat protection and restriction areas; identifying other recovery actions; determining mitigation and protection measures; developing public educational materials; determining how to monitor catches and activities which might impact negatively on species at risk; and developing better/new gear to protect other species (e.g., traps and nets).

SSAR is also used to provide advice to other jurisdictions, including the provinces/territories and municipalities. Non-Governmental Organizations (NGOs) also use the information for their own purposes. The case study demonstrated through the example of the killer whale that SSAR has worked in collaboration with other science programs in DFO, as well as externally with institutions and organizations to undertake the research that they need. For example, SSAR worked in collaboration with the salmon science team (which is within another directorate within FRS) to undertake an assessment of the amount of Chinook salmon that resident killer whales need. DFO Science also collaborated with a number of academic institutions and organizations on killer whale research, including the National Marine Fisheries Service in the United States (US), NGOs, independent researchers in British Columbia (B.C.), and with First Nations groups.  

Considering all of the above and interview respondents, the type and range of information requests are therefore significantly different than those for fisheries management (stock assessments).

Most interview respondents expressed satisfaction with the science products and advice provided overall, even though some researchers do not yet realize the importance of species at risk issues and the legal obligations of DFO in the area. A number of managers expressed dissatisfaction in some areas such as timelines and expertise. For example, a number of respondents mentioned that the timeliness of the information and advice is crucial and that the impacts are considerable when the information is delivered late. While there is a general understanding that SSAR is operating on limited resources, some respondents mentioned that there are legal obligations to meet, and that in some instances, some assessments of species may be delayed because of delays with data, which in turn could delay COSEWIC assessments. However, while COSEWIC may delay the assessment for a certain period, it will eventually go forward with the assessment regardless. In some cases, the information is not collected, or there is simply no expertise in the department in the area for which advice is being requested.

In terms of quality of information, some respondents noted that the format and quality of the RPAs are inconsistent, although document review indicated that there are guidelines and templates available for use. It was mentioned in the interviews that RPAs are not all equal, in that there is no standardized way of undertaking them, and that the information in the RPAs depends on the individuals preparing them (e.g. variable recovery targets, etc). Many Species at Risk managers say that they sometimes complement (for additional information) the information provided by SSAR with those from other sources, such as with information from provincial governments, universities and contractors.

Some advice is considered too vague and generic to be useful. This has created problems as the information is shared with other jurisdictions. It was explained by a few respondents that the DFO science capacity was largely developed for stock assessment purposes, and that the research teams have not yet been reconfigured to meet species at risk requirements. Some further noted that there are competing priorities and resources for species at risk and stock assessment science advice for fisheries management purposes.

Of the approximately 117 clients who received the FRS survey, 21 indicated that they requested information or advice from DFO Science relating to species at risk. When asked whether they received the advice or information asked for, 48% (n=10) clients responded “yes, entirely”, 24% (n=5) responded “yes, in part”, while the remaining 29% (n=6) clients responded that they did not receive the requested information.

Impacts
In terms of actual impact, many interview respondents said that they were satisfied with the quality of the science advice provided and that the information is used (daily, in some cases). Some mentioned that the contribution of SSAR to listing decisions can be considerable. For example, a directed fishery or by-catch species being listed could shut down an entire fishery and the decisions made by DFO can have a huge impact on many communities. The ability of Government to make sound conservation and policy decisions also depends on science advice.  For these reasons, the expectations on the level of precision of the science information can be very high.

However, a few respondents mentioned that science advice in species at risk is newer, and that it is difficult to comment on its usefulness at this point, though impacts in terms of protection and recovery are expected to be significant. For example, DFO scientists have worked on a model that tends to show how controlling disposal at sea could lead to net benefits for killer whales. They have also examined other ecosystem linkages related to killer whales. Evidence from the case study provides an example on the predator-prey relationships between resident killer whales (a species at risk) and Chinook salmon (species monitored and researched by FRS) and further provides an example of the science linkages and the collaborations between species at risk and fisheries resources, ensure that species at risk are protected. Specifically, this example (described below) demonstrates how the critical habitat of a species at risk, including its prey, is important to its recovery, and that the collaboration between DFOs internal programs is crucial to this recovery.

Ecosystem science linkages between species at risk and fisheries resources:
The case of the B.C. resident killer whale and Chinook salmon

There are two types of resident killer whales in the Pacific region, northern residents (usually seen between mid-Vancouver Island and southeastern Alaska), and southern residents (usually range from northern British Columbia to central California). Both have been assessed by COSEWIC in 2001 as either “threatened” (northern residents), or “endangered” (southern residents).

DFO Science has been able to determine that resident killer whales, while apex predators eat primarily Chinook salmon as their prey (almost to the exclusion of other prey, salmon or otherwise). Science has also been able to correlate the decline of resident killer whale populations to the rising and falling abundance of Chinook salmon over the last few decades. And, although challenging, Science was also able to identify critical habitat areas for resident killer whales, particularly during summer months when they feed on migrating Chinook salmon.

As part of DFO’s legal responsibility to protect resident killer whales, it is also required to make sure that there is adequate prey for resident killer whales in critical habitat areas. DFO management has obtained science advice in 2009 around the needs of resident killer whales in terms of how much prey they require for their survival. DFO has also mentioned the importance of Chinook abundance to resident killer whale survival in the 2010 salmon integrated fisheries management plan as a first step.

4.3 Efficiency and Economy


Efficiency Question #5: To what extent is the SSAR governance structure well designed and implemented?

Key finding:  The governance structure (e.g. roles and responsibilities) and the approval processes are not always clearly understood for SSAR.

Findings
The SSAR program and researchers interviewed identified concerns about SSAR governance. There are multiple branches and sectors involved in the delivery of SARA (e.g. EFM, Program Policy and Science sectors as well as regional offices). Roles and responsibilities between these branches and sectors in relation to SSAR are not always clearly understood, such as for the approval of reports. Specifically, reviewing and approving the science contained in reports such as recovery strategies, action plans, and management plans, has proven to cause confusion. As well, species at risk issues often span multiple regions and which has also been the cause of governance issues.

Evidence
According to interview findings (DFO researchers and SSAR managers) SSAR is maturing within DFO a number of issues have been uncovered, including governance and resourcing issues. Many DFO researchers mentioned that at the regional level it is the same research teams that handle advisory requests from both fisheries management and from SSAR programs. While budget lines are different for FRS and SSAR, the work is divided between various specialists (mostly by species) that deal with both stock assessments and species at risk issues; therefore, science requests are accomplished based on science expertise and are not limited to organizational structure (i.e. between FRS and SSAR).

Many respondents expressed concerns, and are supported by the document review, about governance in terms of coordination and roles and responsibilities. At NHQ, the Director General (DG) for Ecosystems Science is responsible for SSAR, and is supported by advisors in SSAR. At the regional level, science teams (e.g. FRS and SSAR) are integrated. However, regional science staff report to the Regional Director General (RDG) (a situation that is not unusual at DFO). According to interview respondents, NHQ Science reports to the Assistant Deputy Minister (ADM), Oceans and Science (O&S), but regional science directorates report to the RDGs of each region. On the Species at Risk Management Program side, two sectors are involved, including EFM sector (e.g. Ecosystem Management branch) and Program Policy sector (e.g. Ecosystem Programs Policy branch) (both present at national and regional levels), as well as external bodies, such as COSEWIC, all of which are separate from the Oceans and Science sector.

As well, roles and responsibilities between the various branches (i.e. Ecosystems Science, Ecosystem Management and Ecosystem Program Policy) of the different sectors (i.e. Oceans and Science, EFM and Program Policy) were unclear in some regions. Some regions spent considerable time in clarifying the different processes, roles and responsibilities across the different branches and breaking down the roles and responsibilities to the individual positions. Some interview respondents mentioned that the relationship between Species at Risk Management Program at NHQ and Science in the regions is not clear and interactions are routed through the regional Species at Risk Management Program. This creates confusion for activities such as reviewing and approving the science contained in recovery strategies, action plans, and management plans. For example, this confusion could have created situations where documents were not shared when it was needed or they were not shared with the correct individual, between regions and NHQ; an inefficient process, especially for a program operating under tight timelines. Roles are perceived as “unclear” by some respondents. A better understanding among DFO staff of the different programs would help resolve misunderstandings about roles and responsibilities and the relationships between various branches and regions. DFO managers expressed similar views to those of the scientist about the governance structure. Although the FRS and SSAR activities and outputs are similar, DFO managers agree that the products of SSAR are distinct from science products for fisheries management, even though many use the same information sources.

Species at risk activities often span across multiple regions and this has also caused governance issues. Species at Risk Management Program is present at the regional level, but the issues it deals with can be inter-regional/zonal, because of wide ranging species present in more than one region.  Coordination among regions adds to the complexity of science delivery and governance. Each region works differently and manages different sources of data, and it was mentioned that data from each region is not necessarily equivalent, which makes delivery of science information to the client problematic. For example, Species at Risk Management Program needs to work with data from multiple FRS research teams (from different regions) that are not always in a consistent format.

Efficiency Question #6: Is there a performance measurement strategy and a reporting process/system in place to communicate the results of the program?

Key findingSSAR activities, outputs and outcomes (where applicable), should be included in the Performance Measurement Strategy of the Species at Risk Management Program Activity to be in line with the 2011-2012 PAA.

Findings
Currently, SSAR does not have a Performance Measurement Strategy. As well, it is difficult to differentiate FRS and SSAR requests in the CSAS databases and a number of activities and outputs are not captured in the current system, including advisory and consultation work outside of CSAS. This situation makes it difficult to assess effectiveness as a result of not having appropriate or complete performance measures. It was suggested by interviewees that the legal obligations under SARA in terms of scientific advice could be the basis of the performance measurement system.

As such, activities, outputs and outcomes (where applicable) of SSAR should be included in the Performance Measurement Strategy of the Species at Risk Management Program - program activity (in line with the 2011-2012 PAA)  to accurately reflect all the activities, outputs and outcomes within DFO which support the implementation and delivery of SARA.

Evidence
Staff members working on SSAR enter output and advisory information in the same information system as FRS. Unfortunately, it is difficult to differentiate FRS and SSAR requests in the CSAS databases. A number of activities (and outputs) are not captured in the current system, including advisory and consultation work outside CSAS.

A few SSAR interview respondents mentioned that the program is still dealing with governance issues and is busy meeting the demand of science requests. It was noted that there are legal obligations (regarding SARA) to meet in terms of scientific advice, and that this could be the basis of the performance measurement system. The turnaround time in meeting the requests should also be monitored as there are legal requirements to this as well.

SSAR does not have a Performance Measurement Strategy for its sub-activity. As of the 2011-2012 PAA, SSAR is no longer a program sub-activity, but is included in the Species at Risk Management Program – program activity. As such, activities, outputs and outcomes (where applicable) of SSAR should be included in the Performance Measurement Strategy of the Species at Risk Management Program – program activity to accurately reflect all the activities, outputs and outcomes within DFO, to support the implementation and delivery of the SARA. The Performance Measurement Strategy for the Species at Risk Management Program – program activity is scheduled to be completed in early 2011 as per DFO’s Performance Measurement Action Plan.

Economy Question #7: Did the SSAR resource utilization and activities optimally produce expected levels of outputs? How could the efficiency of SSAR activities be improved?

Key finding: SSAR has impacted the level of resources of FRS since SARA’s implementation; and the workload for both SSAR and FRS has increased.

Findings
Evidence indicates that SSAR and the species at risk requests in relation to SARA have increased since SARA’s implementation which has impacted the level of resources of FRS. For example, DFO is exploring ways of cost-sharing services, where species at risk programs have contributed to field costs (e.g. Canadian Coast Guard fees for vessel use) with their Operations and Maintenance (O&M) funds. In addition, there is evidence that some FRS Full Time Equivalents (FTEs) work on SSAR activities. As well, evidence found that regions lack expertise and data on freshwater species, but to mitigate this some have leveraged provincial expertise to help meet the species at risk demands.

Evidence
It should be mentioned that the vast majority of respondents were of the opinion that SSAR lacked resources to accomplish what is expected of the program. The workload has increased since SARA was implemented and the research teams (staff levels) have not significantly changed since 2004 thereby putting a strain on the current level of resources (i.e. staff) according to interviewees. Despite efforts, SSAR is not meeting the demand of science requests; however, there are few options to cut costs without harming science capacity. For example, it was mentioned by a respondent that DFO is exploring ways of cost-sharing FRS services with other DFO branches. This is happening in some regions where species at risk programs have contributed to field costs (e.g. Canadian Coast Guard fees for vessel use) with their O&M funds. However, evidence suggests that these cost-sharing practices are inconsistent between regions.

Examples of challenges regarding lack of resources were also identified in the case study. Specifically, the case study highlighted that the availability of ship time has declined in the last couple of years, however, the Cetacean program need ship time to assess killer whale occurrence in outer coastal waters, which are important to determine winter distribution and their prey resources. There is also another type of challenge that has been raised for the killer whale, which is the establishment of their critical habitat given their high mobility (e.g. move up to 100 miles a day). The decline in ship time will also pose challenges to other species such as fin whales and blue whales, which are also required under SARA.

It was mentioned by key informants that the funding identified in the main estimates for the SSAR sub-activity does not include the funding from other programs that contribute financially to the SSAR activities. For example, O&M funds of the Species at Risk Management Program Activity transfers funding for projects from headquarters directly to Science in the regions without it being accounted for in the SSAR program costs.21

According to the 2010-11 RPP, the 2010-11 planned budget allocated to SSAR is $3.3 million, with approximately 33 FTEs associated to the sub-activity. However, it was mentioned by SSAR management that in addition approximately 40 FTEs work for SSAR but are funded through FRS. The 2008-09 RPP indicated that the planned FTEs for SSAR in 2008-2009, 2009-2010 and 2010-2011 were approximately 63FTEs annually.

In addition, the case study demonstrated that DFO Science, specifically science for killer whales, faces resource limitations in undertaking its research. To address this they have worked in collaboration with other science programs in DFO, as well as externally with institutions and organizations to undertake the research that they need. For example, SSAR worked in collaboration with the salmon science team part of FRS to gather information on stocks of Chinook salmon. DFO Science also collaborated with a number of academic institutions and organizations on killer whale research, including the National Marine Fisheries Service in the US. Science has also worked with NGOs and independent researchers in B.C. looking at killer whale occurrence, and with First Nations groups for photo-identification and developing survey protocols. This provides examples of how SSAR is going beyond the department to gather science information on species.

Finally, a gap in skills was noted in the area of freshwater species, as they were not a core priority for DFO prior to SARA. Interviewees noted that regions lack expertise and data, but some have leveraged provincial expertise to help meet the species at risk demands. An analysis of the costs of leveraging has not been accomplished, but respondent’s mentioned that the advantages of leveraging may vary from project to project. However, one of the main reasons why SSAR needs to go outside of DFO is to leverage expertise and knowledge. In addition, it was indicated that in certain cases, external researchers may have access to funds from the National Sciences and Engineering Research Council of Canada (NSERC).

5. Conclusions and Recommendations

5.1 Relevance

SSAR is consistent with the Government of Canada’s legal obligation to protect wildlife species under the SARA, and is consistent with DFO’s objectives of supporting a healthy and productive aquatic ecosystem. For example, SSAR supports DFO’s objectives to identify and protect threatened aquatic species, by undertaking research and monitoring activities on species at risk to determine their status, recovery potential, and critical habitats.

It also addresses DFO’s responsibilities to produce sound science to support management decisions for aquatic species at risk. In addition, SSAR responds to Canada’s international commitments, such as to the UN Convention on Biodiversity, the CEC, and the CITES that depend on sound biological understanding and principles.

There is a continued need for SSAR to address the needs of Canadians, specifically SSAR’s DFO clients. In particular in providing the science base to ensure the protection of aquatic species at risk such as determining their recovery potential assessments, critical habitats, and supporting their listing and development of recovery plans which are part of the legal obligations under SARA.

5.2 Effectiveness

SSAR plays a key role in providing information on species, development of RPAs, recovery planning, identification of critical habitat, monitoring, and input for listing decisions. The science information is used and plays a key role in developing the RPAs. The evidence demonstrates that the RPAs by species is a key deliverable from the Science branch. However, there is evidence that SSAR is not meeting the demand for advice and science information. While the advice provided is often of high quality and  perceived as reliable and accessible, SSAR cannot meet the demand in terms of number of requests met with lack of resources being a key factor.

The types of outputs that both FRS and SSAR produce may include the following products and services:tables and figures;research publications;peer review meeting reports;advice;policy documents; reports to domestic and international organizations such as regional fisheries management groups and international forums; and presentations and communication products.

A number of these formal and informal outputs are not captured within the CSAS process or in science journals, especially those related to data management, administration, communication, and informal outputs that scientists provide to clients (both internal and external). Therefore, the amount of outputs achieved by SSAR goes beyond what has been quantifiably measured.

There is evidence that SSAR is not meeting the demand for advice and science information. While the advice provided is often of high quality, perceived as reliable and accessible, SSAR cannot meet the demand in terms of number of requests met. Specifically, SSAR completed 73% of the requests for science advice on aquatic species at risk, as specified in the 2009-10 DPR. The requests for science advice originate from the SAR Management program, which have been organized since 2010-11 through a ranking system to prioritize funding allocated to projects under the Species at Risk Management Program, which gives, first priority to projects that assist DFO in meeting its numerous regulatory requirements under the SARA and to reduce backlogs. The lack of resources is a key factor. SSAR has legal obligations to meet, and that in some instances, some assessments of species may be delayed because of delays with data, which could delay COSEWIC assessments. In some cases, the information is not collected, or there is simply no expertise in the department in the request area.

Although there are guidelines for the development of RPAs, interview respondents indicated that there are also inconsistencies in terms of the format of the RPAs reports for example variable recovery targets, population levels which cannot be breached not always indicated, etc). Evidence has indicated that they sometimes complement (for additional information) the information provided by SSAR with those from other sources, such as with information from provincial governments, universities and contractor.

SSAR is also used to provide advice to other jurisdictions, including the provinces/territories, and municipalities. NGOs also use the information for their own purpose.

5.3 Efficiency and Economy

The governance of SSAR involves both national and regional structures. The various functions under SSAR are distributed between NHQ and the regional offices. The research teams work in regional offices under the direction of Directors and RDGs. The advice request system (including central database) is managed at the national level by CSAS that receives prioritized lists of requests from the regional offices.

However, the SSAR program and researchers identified concerns about governance. There are multiple branches and sectors (e.g. EFM, Program Policy and Science sectors as well as regional offices) involved in the delivery of SARA. Roles and responsibilities are not always clearly understood, such as the approval of reports. Species at risk issues often span over multiple regions and this also causes governance issues.

As such, it is recommended that SSAR clarify and document its roles and responsibilities and approval processes between SSAR and Species at Risk Management Program at NHQ and in the regions, and between SSAR at NHQ and in the regions.

Recommendation 1:
It is recommended that the Assistant Deputy Minister, Oceans and Science, clarify and document the SSAR roles and responsibilities and approval processes, between SSAR at the NHQ and in the regions as well as between SSAR and Species at Risk Management Program at the national headquarters and in the regions.

Currently, SSAR does not have a Performance Measurement Strategy. Also, it is difficult to differentiate FRS and SSAR requests in the CSAS database. A number of activities (and outputs) are not captured in the current system, including advisory and consultation work outside CSAS. This situation makes it difficult to assess effectiveness as a result of not having appropriate performance measures. For example, as it was mentioned by interviewees that legal obligations under SARA in terms of scientific advice could be the basis of the performance measurement system.

SSAR does not have a Performance Measurement Strategy for its sub-activity. As of the 2011-2012 PAA, SSAR is no longer a program sub-activity, but is included in the Species at Risk Management Program - program activity. As such, activities, outputs and outcomes (where applicable) of SSAR should be included in the Performance Measurement Strategy of the Species at Risk Management Program - program activity to accurately reflect all the activities, outputs and outcomes within DFO, to support the implementation and delivery of the SARA. The Performance Measurement Strategy for the Species at Risk Management Program – program activity is scheduled to be completed in early 2011 as per DFO’s Performance Measurement Action Plan.

Recommendation 2:
It is recommended that the Assistant Deputy Minister, Oceans and Science ensure that Science for Species at Risk Program collaborate with the Species at Risk Management Program, in the development of the Performance Measurement Strategy for the Species at Risk Management Program in order to accurately reflect all the activities within the department (including Science activities) with regard to supporting the implementation and delivery of the SARA.

Evidence indicates that SSAR and the Species at Risk Management Program requests in relation to SARA have gradually increased since SARA’s implementation which has impacted the level of resources of FRS. For example, DFO is exploring ways of cost-sharing services, where species at risk programs have contributed to field costs (e.g. Canadian Coast Guard fees for vessel use) with their O&M funds. In addition, there is evidence that some FRS FTEs work on SSAR activities. Regions lack expertise and data on freshwater species, but some have leveraged provincial expertise to help meet the species at risk demands.

 Annex A: Evaluation Matrix

Science for Species at Risk (SSAR) Evaluation Matrix
QUESTIONS INDICATORS METHODOLOGY
RELEVANCE

1. To what extent are the SSAR mandate and activities aligned with Government and Department priorities and objectives?

1.1 SSAR mandate and activities relate directly to Departmental strategic outcomes for fisheries and science, to the Government of Canada in fulfilling its management responsibilities nationally and internationally.

  • Document review
  • Interviews with DFO staff and DFO clients
  • Case study

2. Does SSAR continue to address the needs of Canadians?

2.1 Informed opinion by various stakeholders that there is an ongoing need for the SSAR program and activities.

  • Interviews with DFO staff, DFO clients, and external clients

3. Is SSAR consistent with federal and DFO roles and responsibilities?

3.1 Views from stakeholders that the federal government/DFO have legitimate roles in providing science-based advice and information for fisheries management

3.2 Evidence that there is  consistency between SSAR and federal/DFO roles and responsibilities

  • Document review
EFFECTIVENESS

4. To what extent is SSAR achieving its results? To what extent have the intended outputs been produced?

4.1 DFO/Govt clients are aware of SSAR and are satisfied with the information and advice it provides.

4.2 DFO/Govt clients are using the information and advice from SSAR.

4.3 External clients are satisfied with the information and advice SSAR provides.

4.4 External clients are using the information and advice from SSAR.

4.5 Evidence that SSAR reports are generated and published as planned and on time.

  • Interviews with DFO staff, DFO clients, and external clients

  • Focus groups with FRS staff and DFO clients

  • Survey of DFO clients

  • Case study
EFFICIENCY/ECONOMY

5. To what extent is the SSAR governance structure well designed and implemented?

5.1 SSAR governance structure is well-articulated, with roles, responsibilities and accountabilities clearly described.

5.2 Evidence of science advisory process in place, including prioritization of science requests and management of risks that may include issues related to species, personnel, etc.

5.3 Evidence that SSAR overall considers issues such as adequate and timely response to science requests, transparent and timely publication of research, and justification on what requests could and could not be met.

  • Interviews with DFO staff and DFO clients

  • Document review

  • Focus groups with FRS staff and DFO clients

6. Is there a performance measurement strategy and a reporting process/system in place to communicate the results of the program?

6.1 Evidence that results/outcomes are well-articulated, and a performance measurement strategy and reporting process/system have been developed for SSAR.

  • Interviews with DFO staff, DFO clients, and external clients
  • Document review

7. Did the SSAR resource utilization and activities optimally produce expected levels of outputs? How could the efficiency of SSAR activities be improved?

7.1 Views from stakeholders on how SSAR can be more efficient in using its allocated resources.

7.2 SSAR and DFO are satisfied with the amount of resources allocated to meet demands.

  • Interviews with DFO staff and DFO clients
  • Case study

 Annex B: Descriptive Flow Chart of relationship between Species at Risk Management Program, SSAR and COSEWIC

graph

  1. DFO Science provides all available science information to COSEWIC to inform the status report.  This can take the form of a pre-COSEWIC meeting or coordination between the research scientists, DFO jurisdictional members and COSEWIC.

  2. DFO has three opportunities (draft, 6-month and 2-month versions of the status reports) to review and comment on the COSEWIC species status reports.  All DFO sectors are given the opportunity to comment, though Science provides the largest amount of input.

  3. DFO Science completes the Recovery Potential Assessment (RPA) meetings.  There is input from other sectors here.  This is the science advice that feeds into the listing decisions.

  4. DFO Science feeds into the Recovery Strategies, Actions Plans and Management Plans.  Identification of Critical Habitat is a very important Science role at this stage.

  5. Science continues to provide information, advice and review of documents and plans forward.  Required reporting on monitoring and recovery progress comes into force summer 2011.  Science will have/does play a significant role in the monitoring of species (continued data collection, research projects, etc.).  The guidelines and expectations around these requirements have not been defined yet, but Science will have a significant role to play. 

DFO Science (SSAR) also feeds into many other aspects of the Species at Risk Management Program such as commenting/reviewing documents, contributing to the development of guidelines, coordination of information from COSEWIC to DFO (which sets the timelines and work requirements for work planning), etc.

 Annex C: Management Action Plan

Management Action Plan – Science for Species at Risk
Recommendations Management Action Plan Status Report Update
Actions Completed Actions Outstanding Target Date

Recommendation 1:

It is recommended that the Assistant Deputy Minister, Oceans and Science, clarify and document the SSAR roles and responsibilities and approval processes, between SSAR at the NHQ and in the regions as well as between SSAR and Species at Risk Management Program at the national headquarters and in the regions.

The National Oceans & Science Director Committee will review and propose Science roles and responsibilities with regards to the approval processes of SARA documents in consultation with Species at Risk Management Program at the national headquarters and in the regions. Arrangement to be approved by Departmental Management Board, or the Sustainable Aquatic Ecosystem Strategic Outcome Committee, as directed.

Establishment of the Species at Risk Science Network whose mandate is to “Foster a coherent and harmonized approach in science delivery in support of DFO species at risk program implementation.”

Discussion between HQ Fish Population Science and HQ SARA Management of the draft document: “Review & Approval Process for SARA Recovery Strategies and Management Plans.”

Discussion between HQ Fish Population Science and Regional Science on responsibilities with regards to the review of draft SARA Recovery Strategies, and more specifically Critical Habitat Schedule of Studies – development of “challenge questions” for Science review.

Discussion in many regions on roles and responsibilities of Science with Species at Risk Management regional counterparts.

Discussion between Science in the regions and HQ Science regarding roles and responsibilities in the review and approval of recovery strategies, action plans, management plans, and SARA listing framework documents.

Discussion between Science in the regions and regional Species at Risk Management regarding roles and responsibilities in the review and approval of recovery strategies, action plans, management plans, and SARA listing framework documents.

Document describing Science role in the review and approval of recovery strategies, action plans, management plans, and SARA listing framework documents

August 2011


October 2011

March 2012

Recommendation 2:

It is recommended that the Assistant Deputy Minister, Oceans and Science ensure that Science for Species at Risk Program collaborate with the Species at Risk Management Program, in the development of the Performance Measurement Strategy for the Species at Risk Management Program in order to accurately reflect all the activities within the department (including Science activities) with regard to supporting the implementation and delivery of the SARA.

The National Oceans & Science Director Committee, through its Species at Risk Science Network, will collaborate with  the Species at Risk Management Program in the development of the Performance Measurement Strategy (PMS) of the Species at Risk Management Program so that the PMS properly reflect Science priority activities in support of SARA implementation

The draft Performance Measurement Strategy for the Fisheries Resource program will be useful in the development of the Species at Risk PMS.

The Program profile, logic model and key activities and output within the SSAR evaluation report will provide a good starting point to account for science contribution in the PMS for Species at Risk Management

The first version of the Science for Species at Risk Evaluation report was provided to Species at Risk Management Program in February 2011

The final version of the Science for Species at Risk Evaluation report  with the Management Action Plan provided to Species at Risk Management

Draft Species at Risk Measurement Strategy to be sent to Science HQ and Regional Science for review and comment by Species at Risk Management

Science HQ and Regional Science provide input to Species at Risk Management on the draft PMS

June 2011


September, 2011

September  15, 2011


1 “Threatened species” means a wildlife species that is likely to become an endangered species if nothing is done to reverse the factors leading to its extirpation or extinction. Source: http://laws-lois.justice.gc.ca/eng/S-15.3/page-1.html

2 “Endangered species” means a wildlife species that is facing imminent extirpation or extinction. Source: http://laws-lois.justice.gc.ca/eng/S-15.3/page-1.html

3 “Extirpated species” means a wildlife species that no longer exists in the wild in Canada, but exists elsewhere in the wild. Source: http://laws-lois.justice.gc.ca/eng/S-15.3/page-1.html

4 Annex B provides a description and a flow chart of the working relationships between the SAR Management Program, Science for species at risk and COSEWIC

5 The Accord lays out a number of commitments to protect species at risk. By its terms, governments recognize that intergovernmental cooperation is crucial to the conservation and protection of species at risk, that governments must play a leadership role, and that complementary federal and provincial/territorial legislation, regulations, policies and programs are essential to protecting species at risk. http://www.ec.gc.ca/media_archive/press/2001/010919_b_e.htm

6 The purpose of the National Framework for Species at Risk Conservation (NFSARC) is to support the implementation of the 1996 Accord for the Protection of Species at Risk by providing a set of common principles, objectives and overarching approaches for species at risk conservation that all participants can share and work toward in a collaborative way.

7 Results achieved were not available for 2010-11, at the time of the evaluation.

8 Although the scope of the evaluation covers a period of five years, the financial information for the SSAR is available for a period of four years, because SSAR was included in another sub-activity.

9 CITES is a multinational agreement that entered into force in 1975 to prevent species from becoming endangered or extinct because of international trade.  The 175 Parties to CITES include countries from Africa, Asia, Central and South America and the Caribbean, Europe,  Oceania and North America (including Canada).The goal of the treaty is to ensure that any trade in protected plant and animal species is sustainable, based on sound biological understanding and principles.

10 SARA section 6.

11 SARA section 14 to 26.

12 April 2009. Data on peer review requests and risk scores approved by the Science Management board for 2009-2010.

13 DPR mentioned that while it started formally recording science requests nationally in 2007-08, the requests were not compiled into a formal excel database until 2009. However, the science advisory process itself has been occurring in the regions for more than a decade.

14 QC and Gulf are only compiling request that they have informally agreed with clients that they can deliver on in the next fiscal year – they control the demand and then they respond to 100% of the mutually agreed on demands.

15 Requests are categorized in the following headings: fishing gear impacts, aquaculture, impact of activities, ecosystem (including frameworks; overviews), stock assessment, recovery potential assessment, marine protected areas, water flow requirements, aquatic invasive species, critical habitat information, fishery management plans, fish passage, pre-COSEWIC review, oceans.

16 Requests’ wording has been refined between client and Science; these are “finalized” requests with some type of peer review meeting attached to them (e.g. advisory meeting, workshop, etc).

17 Only requests categorized as recovery potential assessment, critical habitat information, and pre-COSEWIC reviews were considered.

18 A portion of the planned “maybe” may have been undertaken during the year.

19 Based on keyword searches.

20 While not shown in the tables, it should be noted that DFO papers on stock assessments and papers related to species at risk are not often indexed in Scopus.

21 SAR Management Program funding is not part of the SSAR Evaluation scope. The horizontal evaluation of programs and activities of SARA is being undertaken in 2011-2012.