Archived – Evaluation of the Fisheries Resources Science Program

Archived information

The Standard on Web Usability replaces this content. This content is archived because Common Look and Feel 2.0 Standards have been rescinded.

Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Project Number 6B139
February 2011

Table of Contents

List of Acronyms

ADM
Associate Deputy Minister
CCG
Canadian Coast Guard
CSA
Regional Centres for Science Advice
CSAS
Canadian Science Advisory Secretariat
DFO
Fisheries and Oceans Canada
DPR
Departmental Performance Report
FTE
Full-time equivalent
GoC
Government of Canada
ICCAT
International Commission for the Conservation of Atlantic Tunas
IGS
International Governance and Science Program
MSC
Marine Stewardship Council
NAFO
Northwest Atlantic Fisheries Organisation
NHQ
National Headquarters
NSDC
National Science Directors Committee
OECD
Organisation for Economic Cooperation and Development
PAA
Program Activity Architecture
RFMO
Regional Fisheries Management Organisations
RBAF
Risk-Based Audit Framework
RMAF
Results-Based Management and Accountability Framework
RPP
Report on Plans and Priorities
SAC
Science Advisory Council
SARA
Species At Risk Act
SMB
Science Management Board
SSAR
Science for Species at Risk
UNCLOS
United Nations Convention on the Law of the Sea

 

Executive Summary

Introduction

This evaluation report presents the results of the evaluation of Fisheries Resources Science (FRS).  FRS is a program activity on the 2010-11 DFO Program Activity Architecture (PAA). This evaluation was slated in the Department of Fisheries and Oceans (DFO) multi-year departmental evaluation plan and focuses on the core issues in assessing value for money as defined by the Treasury Board Policy on Evaluation (2009).  The main objective of this evaluation is to determine to what extent the FRS is managed effectively and efficiently and whether it has achieved its stated objectives and results.  The timeframe for this evaluation covers the period from 2005/06 to 2010/11. The evaluation was conducted by the DFO’s Evaluation Directorate and was undertaken between September 2010 and January 2011.

Program Profile

FRS aims to provide a sound and credible scientific basis for the sustainable management of marine and inland fisheries, to contribute to the overall economy of Canada while supporting a stable and safe food supply. The program addresses all domestic fisheries (Aboriginal, commercial and recreational) as well as specific international fisheries, and provides key scientific advice and information for decision-makers to support the sustainable management of fisheries resources. FRS centers on four primary and interrelated functions: monitoring, research, data and information management, and science advice. It also provides much of the scientific information base for other science programs in DFO.

The table below describes the financial resources for Fisheries Resources Science:

Total Program Cost ($000)

Spending Profile

2006-2007

2007-2008

2008-2009

2009-2010

Salary

53,414.6

48,793.5

51,266.9

50,198.9

Non-Salary O&M

13,676.0

16,051.7

29,242.7

28,634.6

Total

67,090.6

64,845.2

80,509.6

78,833.5

Methodology

The evaluation employed a non-experimental design in which measurements are taken after the program has been implemented with no control group. The evaluation addressed issues through document review, key informant interviews, focus groups, a survey of DFO clients, and case studies.  Evidence drawn from these methods was triangulated to arrive at valid findings and conclusions.

Findings and Conclusions

Relevance

According to the evaluation evidence, FRS is consistent with the Government of Canada’s priorities for economic development, DFO’s priorities for sustainable fisheries management to support the wealth of Canadians, directly supports the Strategic Outcome of Sustainable Fisheries and Aquaculture, as well as Canada’s commitment to international conventions and agreements. It also addresses DFO’s responsibilities, by virtue of the Constitution Act, the Fisheries Act, the Oceans Act, and the Species at Risk Act to produce sound science to support management decisions for fisheries management.

FRS continues to address the needs of Canadians by providing scientific information and advice to ensure the sustainability of aquatic resources.  DFO managers use science advisory reports most, followed by research documents, proceedings documents and finally Science Response Processes documents. Science information guides regulatory decisions with respect to setting quotas for example. 

Performance: Effectiveness

FRS produces a significant amount of advice and information (i.e. publications), but this has been decreasing over time.  Data demonstrates that FRS addresses the majority of science requests (note that there are regional differences);   however, between 2004 and 2007, the number of publications produced has decreased. In addition to publications, FRS program personnel provide a variety of products and services that include data reports, scientific and technical reports, consultation meetings, and formal advice, to name a few.

DFO users of FRS science information are generally satisfied with the quality of science information and advice.  There is evidence that in many instances, DFO managers as well as external stakeholders use FRS information for decision-making, planning and engaging in talks with industry. Quotas and other regulatory decisions are guided by the science information and advice from researchers. For example, science information and advice has played key roles in successfully managing lobster and crab resources. In short, FRS is producing publications as well as providing other products and services, and is achieving its immediate, intermediate and long term outcomes. 

Additionally, the new ecosystems approach has had an impact on FRS. This approach expands the research and advisory scope for DFO scientists from single species research and monitoring, to multiple species.

Recommendation #1
FRS needs to identify options to address the challenges with moving towards a complex eco-systems based approach.

Performance: Efficiency and Economy

Various aspects of efficiency and economy were assessed as part of this evaluation, including governance, performance monitoring and resource utilization.

Governance of FRS involves national and regional structures similar to that of other DFO programs. The various functions under FRS are distributed between national headquarters (NHQ) and regional offices. The research teams work in regional offices under the direction of Directors and Regional Director Generals. The advice request system (including a central database) is managed at the national level by the Canadian Science Advisory Secretariat (CSAS) that receives prioritized lists of requests from the regional offices. There were no major issues on governance with respect to FRS in general.

Views about the performance measurement system were also gathered. Currently, performance measurement is focused on publications and the advisory services, including number of requests completed. A number of activities (and outputs) are not captured in the current system, including advisory and consultation work outside CSAS and participation in consultations led by Ecosystems and Fisheries Management (EFM).

While it is acknowledged that the current performance measurement system can account for the core outputs of the program, a review of the system is recommended to allow FRS to capture and report a more complete picture of its performance. FRS is also encouraged to develop a formal feedback system after the delivery of each advisory report to learn lessons as well as to capture the impacts of the advisory services.

Recommendation #2
The ADM of Oceans and Science should develop a Performance Measurement Strategy with the intent of informing decision making. The Performance Measurement Strategy will clearly define indicators that align with the outputs and expected outcomes of the program and would include a program profile, a logic model, a performance measurement framework, an evaluation and reporting strategy, as well as a data collection strategy.

Qualitative evidence indicates that FRS is generally efficient based on the mix and skills of research staff and the type of technologies used to gather scientific information. At the current time, however, there is a skills shortage among staff in the quantitative/modeling areas.

While a true cost-effectiveness or cost-benefit analysis was deemed unfeasible, the evaluation assessed the potential benefits of the FRS using a specific case (snow crab). According to the evidence, FRS actively contributed to quota decisions regarding snow crab harvesting activities. Based on evidence, it is highly likely that quota decisions have contributed to maintaining a sustainable snow crab harvesting industry, which generates approximately $500 million in revenues on an annual basis. If it is assumed that the absence of quotas would threaten the sustainability of the resource, and that FRS plays a key role in supporting quota decisions, it can be concluded that FRS contributes to extensive benefits for the Canadian economy that exceed FRS investments.

Lessons learned include the need for, and benefits of, nationally consistent approaches and improved data management. Best practices include upfront preparation to ensure clear, timely requests and advice.

Recommendation #3
FRS should develop options to address gaps in skills and achieve further efficiencies for example, through further leveraging of resources from other sources (e.g. industry, universities, etc.). To support this, FRS should organize a dialogue between FRS and its clients to re-assess research and advisory priorities and to discuss alternatives.

 1.0  Introduction

This report presents the results of an evaluation of the Fisheries Resources Science (FRS) Program. FRS is a sub-activity of Fisheries and Oceans Canada (DFO), captured within the program activity of Science for Sustainable Fisheries and Aquaculture on the 2010-2011 Program Activity Architecture (PAA).

This evaluation was slated in the DFO multi-year departmental evaluation plan and focuses on the core issues in assessing value for money as defined by the Treasury Board Evaluation Policy (2009). These core issues include: relevance and performance, including effectiveness, efficiency and economy.

The timeframe for this evaluation covers the period from 2005/06 to 2010/11. The evaluation assesses the extent to which FRS has achieved its outcomes stemming from the activities outlined in the Program Logic Model (Figure 1). The evaluation is inclusive of the National Capital Region as well as in all regions that DFO operates. The evaluation was undertaken between September 2010 and January 2011.

2.0  Program Profile

2.1  Mandate

DFO has one of the most comprehensive and complex science programs in the federal government, and relies on science to provide qualitative and quantitative information and analysis and advice to support policy and program decisions regarding marine and freshwater environments and systems across Canada as well, as in the international context. As DFO is a highly decentralized federal department, the science sector is also decentralized across DFO’s seven regions (Pacific, Central and Arctic, National Capital, Quebec, Gulf, Maritimes, and Newfoundland/Labrador). DFO Science supports a variety of client sectors, including fisheries, aquaculture, oceans, species at risk, habitat management and maritime safety, both nationally and regionally. The department employs approximately 1,700 scientists, technicians and hydrographers, and has fifteen research facilities across Canada. DFO Science is delivered through five key functions:

  • Monitoring of physical, chemical, and biological conditions needed to understand changes in aquatic environments and resources;
  • Data and information management collected and generated by DFO Science each year;
  • Research to create new knowledge and methods that will support the development of better advice required for policy and decision making.
  • Scientific advice in support of sound policy development and informed decision-making, derived from series of complex and systematic steps involving data collection, compilation, synthesis and analysis, and subjected to a critical review and peer review process; and
  • Products and Services, including hydrographic charts, nautical publications, and other oceanographic products.

FRS aims to provide a sound and credible scientific basis for the sustainable management of marine and inland fisheries and to contribute to the overall economy of Canada while supporting a stable and safe food supply. The program addresses all domestic fisheries (Aboriginal, commercial and recreational) as well as specific international fisheries, and provides key scientific advice and information for decision-makers to support the sustainable management of fisheries resources.

Ecosystem-Based Management Approach

DFO is currently undergoing a fundamental transition from the traditional single-species fisheries management approach to an ecosystem-based management approach. This has affected how science programs, such as FRS, are being delivered in that scientific advice and information are being asked for not just the abundance levels of specific stocks, but also the ecosystem context in which those stocks operate and are affected by. This includes understanding the habitat linkages, climate change variability, invasive species and disease, impacts of human activity, predator-prey relationships, and other areas that add to the complexity and increased demand for scientific research and advice.

FRS Science Research and Advisory Process

DFO scientists generally conduct research in pursuit of improving the knowledge base around aquatic resources and environments, as well as anticipate potential request for scientific advice that may emerge. When DFO as well as international fisheries management organizations such as the Northwest Atlantic Fisheries Organization (NAFO), requires scientific evidence to support decisions around fisheries resource management and other issues, it formulates a request that is expressed to DFO Science to undertake data collection, analysis, and produce research and/or advisory reports. Requests may also be derived from needs expressed by other federal government departments and formal advisory boards.

DFO’s Science Advisory process is the main, formal process for obtaining science advice to inform policy and management decisions for all DFO priorities, including fisheries management. Developing the science peer review schedule is an annual process whereby various DFO client sectors submit formal requests to obtain science around specific issues or questions, the result of which are peer reviewed science products that emerge to respond to such requests.

The main DFO Science Advisory Process is coordinated by the Canadian Science Advisory Secretariat (CSAS). In line with the decentralized nature of the department, CSAS has regional Centres for Science Advice (CSAs), who conduct their resource assessment reviews independently, tailored to regional characteristics and stakeholder needs.

The table below describes the financial resources for Fisheries Resources Science:

Table 1: Resourcing for FRS from 2006-07 to 2010-11

Total Program Cost ($000)

Spending Profile

2006-2007

2007-2008

2008-2009

2009-2010

Salary

53,414.6

48,793.5

51,266.9

50,198.9

Non-Salary O&M

13,676.0

16,051.7

29,242.7

28,634.6

Total

67,090.6

64,845.2

80,509.6

78,833.5

      
According to the 2010-11 Report on Plans and Priorities (RPP), the 2010-11 planned financial resources allocated to the FRS (under the program activity of Science for Sustainable Fisheries and Aquaculture) is $80.2M, with the engagement of 569 full-time equivalents (FTEs).

2.2  Expected Results

Due to the output and activity based nature of the results identified for FRS, DFO Science drafted a preliminary logic model in 2010 to capture both aspects of science in a more outcomes-based manner. The following diagram is the draft preliminary logic model (Figure 1) developed:

Figure 1: Draft Logic Model of FRS and Science for Species at Risk (SSAR)1

Figure 1: Draft Logic Model of FRS  and Science for Species at Risk (SSAR)

3.0  Methodology

A multiple-lines-of-enquiry approach was used to study the evaluation issues and questions that were the focus of this evaluation. This section outlines the scope and methods of our approach, the evaluation design, evaluation questions, the methodological approach, analytical methods as well as the limitations of the evaluation.

3.1  Evaluation Design

A non-experimental design was used for this evaluation, in which measurements are taken after the program has been implemented with no control group. This model was chosen because FRS is a full coverage program and it is intended to be delivered across Canada and cannot be withheld from any area or region.  The evaluation employed a variety of methods (outlined in section 3.3) where the evidence drawn from these methods was triangulated to arrive at valid findings and conclusions.

3.2  Evaluation Questions

The evaluation questions were determined on the basis of the Policy on Evaluation (2009) and by reviewing documents and the results of the planning phase interviews with key contacts. Annex A features an evaluation matrix organized by topic: relevance and performance (effectiveness, efficiency and economy) and includes the evaluation questions. 

3.3  Data Sources

The evaluation questions of FRS are addressed through the following lines of evidence: document review, key informant interviews, focus groups, survey of users, and case studies.

Document Review

A review of documentation was undertaken to assess most evaluation issues. The evaluation team reviewed all relevant documentation and administrative files available. The following types of documents were reviewed: government-wide documents, and departmental-level documents.  The departmental documents also include administrative data from CSAS.

Interviews

Key informant interviews are used to gather in-depth information for evaluation purposes, including views, explanations, examples, contextual and factual information that address the evaluation questions. Most interviews in the National Capital Region, Newfoundland/Labrador region, the Gulf and Maritimes regions, and the Pacific region were conducted in person. Interviews for other regions were conducted by telephone. Approximately 35 interviews were conducted with DFO employees and external stakeholders between November 2010 and January 2011.

Survey

A survey of clients who use FRS advice and information was conducted. The purpose of the survey was to determine the use of FRS advice and information, the level of client satisfaction with the outputs that FRS provides, and perceptions about the utility and influence of FRS advice and information on their decision-making process.

The primary source of survey respondents was obtained from the CSAS database on annual science requests for advice (for the year 2009-2010). Approximately 117 clients received an email to complete an online survey. Each respondent was asked a core set of questions as well as specific questions for up to two requests they were involved in. The final response rate for the survey was 63% (n=74).

Case studies

Three case studies were conducted to provide a better understanding of the background, as well as the factors supporting the achievement (or non-achievement) of results for the program. Case studies were conducted on the following themes:
Science for Northern shrimp in the Newfoundland/Labrador region;
Science for Western Atlantic Bluefin Tuna fishery; and
Ecosystems science considerations between Killer Whales and fisheries resources in the Pacific.

The main source of information to document the issues related to the cases were key informant interviews, as well as review of relevant documentation pertaining to each case. Interviews were conducted with regional FRS staff, DFO clients, and external clients to the extent possible.

3.4  Analytical Methods

The analysis methods used for this evaluation were tailored to the nature and availability of the data to be gathered, which were in turn linked to the evaluation questions. Extensive use of triangulation was used as an analytical method. In the social sciences, triangulation means that more than two methods are used in a study with a view to double (or triple) checking results using multiple lines of enquiry to corroborate findings.

3.5  Methodological Limitations and Constraints

Over the course of the study shortcomings in the methodology were observed. In order to minimize their impact on the results of the evaluation, we employed information collected from a variety of sources using a variety of methods. In other words, we combined the various methods discussed earlier to arrive at the same conclusions, thereby reinforcing our assessment as to their validity. None of these shortcomings significantly jeopardized the validity or accuracy of the evaluation results. The shortcomings were:

  1. Non Experimental Design. When using this model, it is difficult to clearly measure the net effects of the FRS. Since there are no measurements before the program began, nor a comparison group against which to assess other plausible causes for the outcome, it is difficult to attribute impacts to the program. In other words, it is difficult to draw a clear conclusion about the incremental or net effects of the program. Although this model lacks scientific rigour, the rigour of this design was increased by describing activities, outputs and outcomes through a logic model, enabling evaluators to make causal linkages, and logically argue that results can be attributed to the program (and increase the internal validity).

  2. Data from interviews. Due to time and resource constraints, there was a limit in terms of how many key informants outside of DFO (i.e. external clients) could be interviewed. In light of this, and where possible, their responses were corroborated with other lines of evidence to validate the statements they presented as evidence. Given the wide range of external stakeholders that are particularly affected by DFO decisions and science advice and information, further study may be warranted to examine the views of external stakeholders and their opinions of DFO science efforts.

  3. Cost-effectiveness and cost-benefit analyses. These analyses were not conducted due to the nature of the program is such that it is not feasible to estimate conventional incremental outcomes. In this context, the evaluation team assessed the efficiency and economy using alternate quantitative and qualitative measures to assess the efficiency and potential impacts of the programs.

4.0  Findings

4.1  Relevance

Key Findings:
The evaluation found that FRS aligns with Government of Canada and DFO priorities and objectives, and is consistent with federal and DFO roles and responsibilities with respect to sustainable fisheries management and there is a clear demonstration of a continual need for FRS.

Question 1: To what extent is the FRS mandate and activities aligned with Government and Department priorities and objectives?

The Government of Canada priorities and objectives are informed by the Speech from the Throne and the Budget, the most recent of which have focused on economic stimulation and growth. The Minister of DFO has indicated that one of DFO’s aims is to “improve the economic viability of Canada’s fisheries”2, which is aligned with Canada’s overall objectives of economic development. FRS is the largest science activity of DFO, and supports DFO’s strategic outcome of Sustainable Fisheries and Aquaculture, which mentions contributing to the “wealth” of Canadians as a priority: “Delivering an integrated fisheries and aquaculture program that is credible, science-based, affordable, and effective and contributes to the sustainable wealth for Canadians while respecting Aboriginal and treaty rights.”3.

A review of the literature found that these priorities are grounded by the important contribution of the fishery to the economy, where for example, Canada’s capture fish production4 comprised US$1.755 billion in 2007, while comparatively, Australia was US$1.191 billion, Spain was US$2.245 billion, and the UK was $1.15 billion for the same year5. Canada employed about approximately 48,239 people in 20076 in commercial harvesting 7, compared to Australia (9,735), Spain (33,069), and the UK (12,729) for the same year. Canada’s recreational fisheries are also an emerging economy, receiving approximately CAD$2.6 billion in direct investments in 2005.

Question 2: Is FRS consistent with federal and DFO roles and responsibilities?

The federal government has a role in the management of sea coast and inland fisheries according to the Constitution Act Article 91 (12). The Fisheries Act outlines the responsibilities of the Minister to manage fisheries, habitat and aquaculture. The Oceans Act Article 4 (1) provides the Minister with responsibilities to lead integrated oceans management and hydrographic services. The FRS program key informants explained that fisheries research supports DFO’s mandate, which is to develop and implement policies and programs in support of Canada’s scientific, ecological, social and economic interests in oceans and fresh waters. The role of research is to provide scientific information to ensure the sustainability of resources, which meets the ecological, social and economic interests.

FRS is linked to Canada’s international commitments by virtue that only sovereign states may engage in such agreements.  For example, Canada has ratified the United Nations Convention on the Law of the Sea (UNCLOS)9, in which Canada commits to protecting and preserving the marine environment, and to carrying out marine scientific research in its sovereign territories.  By virtue of the 2001 United Nations Fish Stock Agreement, Canada also has a leadership role in the management of international fisheries and its commitment to implement sustainable and precautionary principles.

Question 3: Does FRC Continue to address the needs of Canadians?

Evidence from the survey and interviews indicated that there is a need for FRS, specifically related to the sustainable fisheries resources and achieving the delicate balance between maximizing harvesting activity while preserving the resources in a long-term perspective. Science-based evidence and advice about stocks and ecosystems is the best info available to enlighten DFO decisions about programs, policies and regulations.

The survey results illustrate the most used sources of science-based evidence and advice, which reflects client needs as consumers.  Sixty-three percent (n=39) of respondents regularly used science advisory reports, while a little over a quarter use them occasionally. About 40% (n=25) of the respondents use the research documents on a regular basis (the same number uses them occasionally), while less than one-quarter regularly use the proceedings documents. Approximatley 48% (n=29) never or rarely use the Science Special Response Processes10 documents.

Figure 2
Source: Survey of DFO managers
Source: Survey of DFO managers

The DFO managers who use FRS information and advice also mentioned the need for science information to guide regulatory decisions. In particular, there is a need to develop good reference points pertaining to the precautionary approach for stocks, as resources are, in many cases, scarce. Some respondents provided concrete examples of the need for this type of information, including the need to assess the stocks of various species, and the need to assess the impacts and potentials risks of aquaculture on wild species (e.g., impacts of cultured salmon on wild trout population). 

Some FRS program respondents also explained that without science and science-based advice, decisions such as those related to quotas would not reflect the state of resources and could lead to high-risk situations such as over-fishing. An external stakeholder mentioned that science has had a direct input into specific requirements related to the Pacific Wild Salmon Policy11. The implementation of the policy relies on science information to ensure the best decisions to support conservation of the resource.

Evidence from the survey (Figure 3) indicates that FRS provides a unique service. As indicated, 73.2% (n=41) of respondents did not think that there are alternatives to FRS research and about 76% (n=38) of respondents agreed that FRS provides advice which sets them apart from other thinks tanks or research institutes. Further, 73.2% (n=41) agree that they could easily find the information they required on the FRS website, and that the FRS research publications meet or exceed standards of most peer-reviewed journals (73.7%, n=28).

Figure 3
Source: Survey of DFO managers
Source: Survey of DFO managers

In addition, there is need for FRS to help Canada meet its international obligations. Scientific information about fisheries is needed by Canada to meet international commitments (e.g., United Nations Convention on Biodiversity), which places expectations on Canada to manage resources wisely and sustainably. Some respondents mentioned that the only way to meet expectations set forth within international agreements is through a strong science base.

Finally, FRS program key informants note that there are also newly emerging needs arising from industry: many large buyers (e.g., large food chains and suppliers) are requiring labels indicating which products are "sustainable products". This requires scientific information to confirm the state of the stock and to develop science-based rebuilding plans for depleted stocks. Clients of the Marine Stewardship Council (MSC) are drawn from the fishing industry for the purpose of enhancing market opportunities through eco-certification12.  DFO may assist industry stakeholders in the provision of information required to demonstrate compliance with MSC certification criteria – thus industry may pressure DFO to support their aspirations to achieve eco-certification – but also this is part of DFO mandate under the Oceans to Plate initiative13.

External stakeholders also mentioned the need for information on a variety of issues, such as habitat, impacts of aquaculture and water quality issues. On the industry side, it is expected that DFO will provide science information that will be consistent with the approaches the department is advocating, such as the “ecosystem management approach”. Industry also expects relevant information that will also serve their interests.

4.2  Performance: Effectiveness

Key Finding:
The evaluation found that there is a general level of satisfaction with the quality of the science advice and information produced. The science advice and information has also been used as intended (i.e. inform decision-making) to a large extent. However, overall, science requests are exceeding the capacity of FRS to meet them, and there is expressed dissatisfaction about timeliness. FRS has been significantly impacted by the new ecosystems approach, which expands research from a single to a multiple-species approach.

Question 4: To what extent have the intended outputs been produced?

Quantitative data on the number of science requests and publications obtained from DFO Science determined whether FRS produced expected outputs.

According to advice and peer review requests received by the CSAS14 in 2009-2010, the first year in which CSAS established its database of science requests15, the data shows that DFO Science received 241 requests overall that year, of which 45% (n=109) of the total number were for advice on fisheries resources. Approximately 59% of FRS requests were addressed16. In terms of total completion rates, Central and Arctic had the lowest amount of requests met (25%), followed by Pacific, Newfoundland/Labrador, Maritimes, and NHQ (all between 51-55%), then Quebec and Gulf (100%17).

Analysis of the number of CSAS publications produced from 2004 to 2008 (including science advisory reports, research documents, proceedings, and special response reports) reveals that the number of CSAS publications related to fisheries resources have decreased from 148 in 2004 to 115 in 2007.  It should also be noted that the largest number of CSAS publications produced in any one year is for Fisheries Resources issues (51 to 65%), followed by Species at Risk (13 to 23%). Other issues comprise less than 10% of all CSAS publications.

A second set of data obtained from DFO Science with respect to a forthcoming bibliometric study18 of DFO science publications is captured in Scopus19 as well as the CSAS database and website. This data compares the amount of DFO science publications to that produced by other countries around the world in the field of fish population science.  From 1998 to 2008, Canada was second only to the USA in producing published papers in fish population science captured in Scopus (4,512 papers compared to 13,676 from the U.S.A.). Out of 4,512 publications captured in Scopus, DFO produced the most fish population science papers in Canada (1,486 papers or 32.9%), and was also ranked second world-wide in terms of the institution producing most fish population science papers (in first place was the National Oceanic and Atmospheric Administration in the U.S.A at 2,282). In addition, an overall trend indicates that the number of publications that Canada (and DFO) produces in fish population science increased from 1998 to 2008.

It was found that the amount of outputs achieved by FRS go beyond what has been quantifiably measured. According to the FRS program respondents, the types of outputs that FRS produces may include the following products and services: Data reports (tables and figures); Scientific and technical reports; Scientific literature (research publications); Consultation meetings; Peer review meeting reports; Advice (formal reports); Question and answers (informal); Policy documents; Reports to international organizations such as regional fisheries management groups (e.g. Northwest Atlantic Fisheries Organization or NAFO) and international forums (e.g. International Conference for the Exploration of the Sea); and, Presentations and communication products. A number of these formal and informal outputs are not captured within the CSAS process or in science journals, especially those related to data management, administration, communication, and informal outputs that scientists provide to clients (both internal and external).

Question 5: To what extent is FRS achieving its results?

Many DFO Managers expressed satisfaction with the quality of some of the FRS information, but felt that they could use information that is broader in scope. Some mentioned that FRS met only half of their requests. As discussed ealier, data indicates that in 2009-2010, 59% of FRS requests (CSAS) were addressed.

Views were mixed in terms of the reliability of the information depending on the species and area. It is generally understood that FRS can identify population trends, but not precise time-specific information due to limited data. Respondents mentioned that apart from resource issues, variability in quality may increase as senior FRS staff retire.

Most external stakeholders were pleased with the quality and reliability of the science produced. The FRS is subject to peer review, similar to academia and the CSAS reports were qualified as “very good”. Some respondents mentioned that the information is accessible, at least for those who are active in the sector. The peer review process is seen as a “challenge” for FRS staff to present the results of their analyses in a clear, accessible and succinct fashion.

Survey respondents indicated the extent to which they agree with a number of impact-related statements (Figure 4). Overall, survey respondents felt that FRS provides adequate science-based evidence for the management of fisheries.  Approximately 45% (n=9) agreed that FRS provides adequate science based evidence for the management of international fisheries, and approximately 60.4% (n=29) for the management of Canadian fisheries.  Respondents agreed the quantity of the FRS information provided by DFO is satisfactory (38.6%, n=22) and 40.7% (n=24) said that their information needs in the areas of fisheries resources are being met.  Sixty-three percent (63.3%, n=38) of respondents were satisfied with the types of FRS information and advice produced by DFO. However respondents were split with regard to having to supplement the information they obtained from DFO with other sources such as non-governmental organizations (35.6%, n=21 agreed that they often needed to do so, while 39.0%, n=23 indicated they do not).

Figure 4
Source: Survey of DFO managers
Source: Survey of DFO managers

Comments about communications were fairly positive. DFO Managers interviewed discussed the clarity of the information, mainly saying that the scientists are very helpful in making science information accessible by using plain language, although new DFO managers may still find it difficult to understand the science without adequate support.  Some respondents perceived a disconnect between FRS and the rest of the department. It was noted that a more formalized process to create linkages may be helpful.

Figure 5 presents survey respondents’ ratings on the quality of various aspects of the advice they received. Respondents indicated that the advice received provided new perspectives or insights (55.6% rated this as high in quality n=15). Sixty-four percent of those who responded to the questions rated the timeliness of the advice high (n=18). Seventy-one percent rated the level of decisiveness of the advice as high (71.4%, n=20).

Figure 5
Source: Survey of DFO managers
Source: Survey of DFO managers

In terms of timeliness, while many expressed satisfaction, others expressed dissatisfaction and said that the surveys were not happening often enough, depending on the species (up to eight years), and others mentioned that information has arrived late for some species (in terms of the seasonal harvesting cycle). DFO managers and many interview respondents, including clients noted that FRS is often faced with having to address multiple complex requests and cited that the lack of human resources and competing priorities sometimes prevent FRS staff from providing the service in a timely manner. Many respondents said that they complete the FRS science information and advice with those from other sources, including academia and external consultants.

DFO managers said that the science information and advice is extremely useful for making various decisions, including planning decisions. It is also used to engage talks with industry about various limitations, such as quotas and other restrictive mechanisms. 

Figure 6 presents respondents’ ratings on the quality of various aspects of the advice they received. Respondents were very positive about the availability of researchers to provide clarifications (92.6% rated this as high in quality, n=25) as well as the rigour of the analysis supporting the advice (92.3% rated it as high, n=24) and the objectivity of the information supporting the advice (also at 92.3%, n=24). Also highly rated were the relevance and overall quality of the advice (both at 89.3%, n=22).

Figure 6
Source: Survey of DFO managers
Source: Survey of DFO managers

Figure 7 presents respondents’ ratings on the their perceptions of the science advice provided and the peer review20 process. Most respondents felt that science advice was important for decision making purposes (74% to 94%).

Figure 7
Source: Survey of DFO managers
Source: Survey of DFO managers

On the international side, one key informant mentioned that the International Governance Strategy (IGS) has been helpful in providing focus on specific research that would help Canada in negotiations in Regional Fisheries Management Organizations (RFMOs). The case study of Atlantic Bluefin Tuna exemplifies the support provided by the IGS in strengthening Canada’s ability to negotiate on the international stage, as outlined below.

Fisheries Resource Science through International Research and Quotas: The Case of the Atlantic Bluefin Tuna

The Atlantic bluefin tuna consist of two fish stocks, Eastern and Western Atlantic bluefin tuna (although mixing of the stocks is highly likely). Total landed value of bluefin tuna (Atlantic Canada) was estimated at $7,178,945 in 2009. Quotas on bluefin tuna are determined by the International Commission for the Conservation of Atlantic Tunas. Despite quotas, the Eastern/Mediterranean bluefin population has diminished by 15 or 18 per cent of what it was in the 1970s.

Through the International Governance Strategy, DFO has invested considerably in research on bluefin tuna. This has afforded Canada a leadership role in the Standing Committee on Research and Statistics (SCRS), the science component of International Commission for the Conservation of the Atlantic Tuna (ICCAT). Canada has also contributed to studies on the movement and migrations of bluefin tuna using satellite archival tags. Most recently, Canada has led a ground-breaking study of the post-release survival of bluefin tuna caught in a simulated catch and release recreational fishery. According to findings, FRS research has informed both domestic and international regulations, as well as national policy and program decisions. The research conducted by DFO is directly fed into Canada’s ICCAT position.

On the east coast, prime examples where FRS made a positive contribution to DFO decisions include two shellfish: snow crab and lobster. DFO regulates snow crab fishing through quotas at local levels (in tons). DFO has successfully implemented quotas on multiple occasions (and areas) over the last decade and both scientists and DFO managers agree that these limitations have helped maintain a sustainable resource. The snow crab is a prime example where DFO-industry decisions have been very close to the advice provided by scientists.

For example, quotas for crab in the Saint-Lawrence Gulf (CFA16- QC North shore) were reduced from 4,000 to 2,200 tons in 2003 as scientists noted a significant decline in stocks. Though industry accepted the decision reluctantly, the snow crab population has since increased as a result and quotas went back to previous levels. This is an example of how science has helped decision-makers to ensure sustainable fishing.

The Snow Crab (Northwestern Atlantic)
Snow crabs live on muddy bottoms in cold waters.  Commercial fishing of the snow crab began in the mid-1960s in Canada. The main fishing grounds range from Quebec to western Newfoundland, as well as the lower Gulf and around Nova Scotia. This is a male-only fishery with the minimum size limit set at 9.5 cm in carapace width. The fishery is also managed by fishing season, number of traps, and quota. The total landings in Atlantic Canada were at 90,000 tonnes in 2007. In 1995, the landed value of snow crab reached a peak of $134.7 million.  [Source: DFO Website]

Other scientists mentioned that lobster is another example where science has helped ensure a sustainable resource (through various limitations, including trap sizes, carapace21 sizes of captured lobster, etc.). Lobster represent a major source of revenue for industry; it is Canada’s most valuable seafood export, representing as much as $1 billion in export sales annually to more than 59 countries. In 1995, the landed value of lobster reached a high of $168.7 million.

Unintended Outcomes

A number of respondents mentioned how the new ecosystems approach has had an impact on FRS. This new approach expands the research scope for DFO scientists from single (and mostly commercial) species research and monitoring (stock-assessments), to multiple species (including non-commercial and mammals) as well as research about the various factors that affect these species, such as environmental and human factors.

Some ongoing changes noted in document review mentioned that FRS is broadening the basis for science advice to include the regime status of the environment, and understanding how stock productivity is expected to change with population demographics and environmental conditions. Assessing status and trends of some non-commercial species such as species taken as by-catch in fisheries and benthic ecosystem components (e.g. that provide habitat or food for other species) are also part of the ecosystem approach. 22

Previous assessments 23 indicate that the move towards ecosystem-based management has resulted in more complex requests for science advice and information. This has included advice to support the management of new fisheries that were not previously conducted, more science (in terms of information and frequency), more comprehensive evaluations of populations in an ecosystem context, more support for international fisheries negotiations, more monitoring of marine mammals and linkages to fisheries resources, more information on species at risk, and more information on human impacts on habitat and ocean environments. Documentation suggests that FRS still faces challenges in moving towards an ecosystem approach.

4.3  Performance: Efficiency and Economy

Key finding for Efficiency and Economy section

  • There were no major governance issues identified for FRS.
  • A review of the performance measurement system was completed and the development of a performance measurement strategy is recommended to allow FRS to capture and report a more complete picture of its performance.
  • However, a gap in certain skills (namely quantitative) was identified as well as the need to continue to leverage resources from other resources.
  • A number of lessons learned are identified and still remain to be implemented, including the need for and benefits of nationally consistent approaches and improved data management. Best practices include upfront preparation to ensure clear, timely requests and advice. 

4.3.1  Governance and Delivery

Question 6: To what extent is the Fisheries Resources Science governance structure well designed and implemented?

No issues with governance from interviews and surveys were identified. FRS is a highly decentralized program with research staff working in multiple labs, including labs in the Atlantic, Quebec, Ontario and British Columbia with FRS senior management located at NHQ.

With respect to DFO Science in general, documentation indicates that the governance of Science is decentralized to the regions, with strategic linkages to the NHQ level. The Assistant Deputy Minister (ADM) of Oceans and Science is the overall sector head, whose fisheries responsibilities include ecosystem science, aquaculture science, environment and biodiversity science, fish population science, oceanography and hydrography, and genomics, etc. The ADM works closely with the Director Generals for Science Renewal, Integrated Business Management, Ecosystem Science, and Oceans Sciences–Canadian Hydrographic Service, as well as with the Executive Director of Strategic Science Outreach. A National Science Directors Committee (NSDC) provides overall direction on the operations of DFO Science at large, and consists of the aforementioned managers as well as six Regional Directors of Science. The NSDC is chaired by the Assistant Deputy Minister.

Regionally, the Regional Directors of Science are responsible for the operations of the 15 science institutes, laboratories and experimental centres. DFO Science also has a series of mostly virtual Centres of Expertise (CoE) where partnerships between internal and external researchers occur to achieve research results serving the national science agenda. Regional Directors of Science report to their Regional Director Generals, but also receive guidance from NHQ’s Oceans and Science branch. The organization of each DFO region’s science branch is distinct and unique to the context of the region. The Regional Directors of Science report to their Regional Director General who report to the Deputy Minister.

It should be mentioned that at NHQ, a Science Management Board (SMB) identifies science-related issues of importance to achieving DFO mandated objectives, establishes priorities in need of science support, setting strategic priorities and direction on work planning for DFO Science, and provides guidance on the alignment of resources. The SMB is chaired by the Deputy Minister and includes the Assistant Deputy Minister of each major sector, two senior scientists, the Chair of the Science Advisory Council and a Regional Director from both the east and west coast. A Science Advisory Council (SAC) provides advice to DFO Science management on strategic matters pertaining to science and technology. They consist of about 15 external experts, and four ex-officio members from within DFO Science.

4.3.2 Performance Management

Question 7: Is there a performance measurement strategy and a reporting process/system in place to communicate the results of the program?

From an analysis of expected results and indicators described in the DFO RPP and Departmental Performance Reports (DPR), indicators have moved from achievement of planned activities (e.g. in 2005-06), to specific scientific products emerging on fish stocks. The most recent expected results and indicators have focused not only on assessing the number of science publications emerging for fish stocks, but also on comparisons of the demand for science (e.g. percentage of science requests completed), the products of science (e.g. publications produced), and the availability and access to science (e.g. publications posted on website, download rates, etc). Since new indicators were only introduced in 2010-2011, performance data for these have yet to be captured.

Many FRS staff members are of the opinion that the performance management system should be reviewed. As mentioned above, CSAS currently stores data on the advisory requests and monitors the provision of the advice through that system. Regional offices also have databases of advice requests and some store the actual advice in databases as well, but these regional databases are not connected with NHQ.

Most respondents agreed that the performance measurement could be improved. It noted that there is no Results Measurement Accountability Framework (RMAF) or Results-Based Accountability Framework (RBAF) for FRS. Other respondents noted that the number of publications produced, while useful, is not the best indicator of the performance of FRS. While publications, a decade ago, would have been an appropriate indicator for the performance of FRS, it does not reflect today’s performance and success of the advisory work. As a result, much of FRS work has not been appropriately captured since.  FRS now provides policy documents, frameworks, and technical reports, which are not being counted by current systems.

Some respondents noted how it would be challenging to monitor the actual impact of FRS on the ultimate results, including the number of species . FRS only provides advice and cannot be accountable for management decisions taken after the science information has been provided. Nevertheless, some respondents called for a review of the performance management system and above all, for the development of a broader business plan.

4.3.3 Resource Utilization

Question 8: Did the FRS resource utilization and activities optimally produce expected levels of outputs? How could the efficiency of FRS activities be improved?
Question 9: To what extent is FRS incorporating measures to minimize costs in achieving its objectives?

Based on key informant interview findings, the evaluation assessed the extent to which FRS is efficient was based on: 1) the extent to which FRS had the right mix of staff to conduct research; and 2) the extent to which FRS was using the latest technologies.  General discussions about efficiency were also conducted with senior managers, including potential improvements and leveraging resources from other sources.

A review of documents suggests that some efforts have been taken by FRS to improve efficiencies in its activities. For example, in implementing the recommendations from the Report from the DFO Science Monitoring Implementation Team, time on Canadian Coast Guard (CCG) ships to conduct science research has now been maximized to collect a spectrum of data through multi-disciplinary surveys, not just stock status information. An example is standard groundfish surveys now collect a wide range of information on the ecosystem such as data on oceanographic conditions (e.g. phytoplankton, zooplankton, oxygen levels, etc.), fish condition and marine mammal observations.

Other activities mentioned in documentation include the implementation of the risk-based prioritization framework to efficiently streamline the science advisory process, a potential new initiative to investigate new monitoring and assessment approaches to stock assessment, and use of new technology (e.g. electronic sensing, video camera) to allow for more collection of diverse data related to ecosystems. However, data management and analysis was mentioned as an impediment to being more efficient.

Mix of staff

Many respondents provided comments about the extent to which FRS teams had the right mix of skills to achieve the expected FRS products and services. Many respondents mentioned that there is one area where there is a significant skills shortage: quantitative analysis, specifically modeling skills (knowledge in interpreting models). This prevents FRS teams from producing population projections efficiently.

A number of respondents also mentioned that the effectiveness of the teams will be affected by the retirement of FRS staff. Many regional representatives indicated that there is no succession planning in their region with respect to FRS.

Technology

Apart from human resources, the other major productivity factor in science is the research equipment and technology used; however, no respondents felt that DFO was using outdated methods or technologies.

Leveraging

A number of respondents mentioned that DFO was leveraging resources from other sources. Examples included the use of CCG ships to gather data (cost-shared); partnering with universities and provincial experts to acquire information and data; partnering with the private sector and First Nations (e.g., work closely with wildlife boards as mandate under modern treaties agreement) to obtain data; obtaining funds from industry associations to conduct research.

4.3.4 Economy

Question 10: To what extent is Fisheries Resources Science economical?

The potential benefits of FRS were assessed using the specific case of snow crab. According to the evidence, FRS actively contributed to quota decisions regarding snow crab harvesting activities. Based on evidence, it is highly likely that quota decisions have contributed to maintaining a sustainable snow crab harvesting industry, which generates approximately $500 million in revenues on an annual basis. If it is assumed that the absence of quotas would threaten the sustainability of the resource, and that FRS plays a key role in supporting quota decisions, it can be concluded that FRS contributes to extensive benefits for the Canadian economy that exceed FRS investments.

In addition to the above findings, the economy of the program was examined by assessing the potential benefits of FRS using various data sources. A true cost-benefit analysis of FRS would be difficult to assess considering the unknowns surrounding the incremental impact of the program. As explained earlier, FRS provides stock assessments and other science information to support DFO decision-making about quotas and other protective measures to ensure sustainable harvesting. The science information is only one element considered for quota decisions; as well, the exact relationship between quotas and actual population levels is not always obvious, considering the various environmental factors that affect fish populations.

Despite these limitations, the evaluation team assessed the scope of the FRS costs and the potential benefits associated with quotas. As such, the evaluation cannot draw a direct linkage with the benefits, but will explore various ratios to assess the potential scope of the benefits.

Potential Benefits: the Case of the Snow Crab

Many respondents cited the case of the snow crab as being a highly successful case where DFO quotas played a key role in maintaining a sustainable resource. As mentioned in section 4.2, DFO regulates snow crab fishing through quotas at local levels (in tons). DFO has successfully implemented quotas on multiple occasions (and areas) over the last decade and both scientists and DFO managers agree that these limitations have helped maintain a sustainable resource. According to evidence, snow crab is a prime example where DFO-industry decisions have been very close to the advice provided by scientists. According to DFO, the snow crab market is significant. Exports of Canadian snow crab product reached a record $678 million in 2002. Various assumptions could be made as to the actual contribution of FRS in maintaining this market. The evaluation did not examine a cost-benefit analysis based on various assumptions. However, if it is assumed that science plays a key role in determining quotas (this has been demonstrated by evidence presented in section 4.2 in this report), and that quotas have played a key role in maintaining a sustainable snow crab resource, it can be stated that FRS plays a key role in maintaining a key sector of the fisheries industry, a sector generating approximately $500 million in revenues on an annual basis. These figures alone demonstrate how FRS can bring significant value to Canadians.

4.3.5 Best Practices and Lessons Learned

Are there best practices and lessons learned from FRS?

From the documents reviewed, a number of lessons learned have been mentioned in past studies and assessments that should be implemented. Most of these lessons learned have some level of implementation, so work remains to be accomplished by the Program for the full implementation. These include the following:

  • A nationally-consistent approach to dealing with the emerging issues of eco-certification in the fishing industry would support prioritization of science efforts (not yet implemented);
  • Continue replacing ageing DFO research fleets (CCG and other DFO vessels) and strategically invest in technology to maintain the ability of Science to undertake its monitoring and research activities (ongoing; more work needs to be done);
  • Greater coordination with other jurisdictions involved in freshwater, diadromous and anadromous species monitoring and research is needed (ongoing; more work needs to be done);
  • Work closer (through formal and nationally consistent mechanisms) with clients when requests are not prioritized, to determine what mitigation actions should be taken or possible adjustments to the program that should be made (ongoing; more work needs to be done);
  • Develop a more efficient managing and reporting strategy, adapted to the context of integrated activities financed by multiple sources, to allow scientific staff to dedicate more time on scientific activities. This includes performance indicators to appropriately measure and record progress (ongoing; more work needs to be done);
  • Improve data management (e.g. more integrated databases, efficient exchanges between databases among programs and regions) (ongoing; more work needs to be done); and
  • Support for more balanced and nationally consistent contribution from external experts and stakeholders in the advisory process (ongoing; more work needs to be done).

The key informant interview respondents also identified many best practices and lessons learned that should be implemented. Most of these lessons learned have some level of implementation, so work remains to be accomplished by the Program for the full implementation:

  • Research needs to start before any type of advice is provided. Foundational level information that is monitored on routine basis is needed to answer questions (ongoing; needs continued support).
  • There is a need for FRS and its clients to work together, to share information where program demands will be in the future (ongoing; needs continued support).
  • Ensuring that requests and scientific questions are clear – this includes taking time to understand requests to make sure that this is what clients are really looking for (ongoing; needs continued support).
  • Ensuring that the advice is clear (by following the guidelines of framework more strictly) (ongoing; needs continued support);
  • Managing the peer review meetings to avoid them from becoming “free-for-all forum to voice concerns”. Overcrowded meetings disrupt the process and discourage external scientists from attending. It involves paring down the number of participants to effectively manage the meetings (ongoing; needs continued support);
  • Some low risk requests can be dropped from year to year, but may become higher risk later on (ongoing);
  • Dealing with a group of researchers instead of one provides a balanced view (ongoing); and
  • The earlier notice that can be provided the better – this way expertise can be sought in the region or elsewhere (ongoing; needs continued support).

5.0  Conclusions

Relevance

According to the evaluation evidence, FRS is consistent with the Government of Canada’s priorities for economic development, DFO’s priorities for sustainable fisheries management to support the wealth of Canadians, directly supports the Strategic Outcome of Sustainable Fisheries and Aquaculture, as well as Canada’s commitment to international conventions and agreements. It also addresses DFO’s responsibilities, by virtue of the Constitution Act, the Fisheries Act and the Oceans Act, to produce sound science to support management decisions for fisheries management.

FRS continues to address the needs of Canadians by providing scientific information and advice to ensure the sustainability of aquatic resources.  DFO managers use science advisory reports most, followed by research documents, proceedings documents and finally Science Response Processes documents. Science information guides regulatory decisions with respect to setting quotas for example. 

Performance: Effectiveness

FRS produces a significant amount of advice and information (i.e. publications), but this has been decreasing over time.  Data demonstrates that FRS addresses the majority of science requests (there are regional differences); however, the number of publications produced from 2004 to 2007 has decreased. In addition to publications, FRS program personnel provide a variety of products and services that include data reports, scientific and technical reports, consultation meeting and formal advice, to name a few.

DFO users of FRS science information are generally satisfied with the quality of science information and advice.  There is evidence that in many instances, DFO managers as well as external stakeholders use FRS information for decision-making, planning and engaging in talks with industry. Quotas and other regulatory decisions are guided by the science information and advice from researchers. For example, science information and advice has played key roles in successfully managing lobster and crab resources.

In short, FRS is producing publications as well as providing other products and services, and is achieving its immediate, intermediate and long term outcomes. 

Also, the new ecosystems approach has had an impact on FRS. This approach expands the research scope for DFO scientists from single species research and monitoring, to multiple species.

Recommendation #1
FRS needs to identify options to address the challenges with moving towards a complex eco-systems based approach.

Performance: Efficiency and Economy

Various aspects of efficiency and economy were assessed as part of this evaluation, including governance, performance monitoring and resource utilization.

As for many other DFO programs, governance of FRS involves national and regional structures. The various functions under FRS are distributed between NHQ and regional offices. The research teams work in regional offices under the direction of Directors and Regional Director Generals. The advice request system (including central database) is managed at the national level by CSAS that receives prioritized lists of requests from the regional offices. There were no major issues on governance with respect to FRS in general.

The evaluation team also gathered views about the performance measurement system. Currently, performance measurement is focused on publications and the advisory services, including number of requests made and met. A number of activities (and outputs) are not captured in the current system, including advisory and consultation work outside CSAS.

While it is acknowledged that the current performance measurement system can account for the core outputs of the program, a review of the system is recommended to allow FRS to capture and report a more complete picture of its performance. FRS is also encouraged to develop a formal feedback system after the delivery of each advisory report to learn lessons as well as to capture the impacts of the advisory services.

Recommendation #2
The ADM of Oceans and Science should develop a Performance Measurement Strategy with the intent of informing decision making. The Performance Measurement Strategy will clearly defined indicators that align with the outputs and expected outcomes of the program and would include a program profile, a logic model, a performance measurement framework, an evaluation and reporting strategy, as well as a data collection strategy.

Qualitative evidence indicates that FRS is generally efficient based on the mix and skills of research staff and the type of technologies used to gather scientific information. At the current time, however, there is a skills shortage among staff in the quantitative/modeling areas.

While a true cost-effectiveness or cost-benefit analysis was deemed unfeasible, the evaluation team assessed the potential benefits of the FRS using a specific case (snow crab). According to the evidence, FRS actively contributed to quota decisions regarding snow crab harvesting activities. Based on evidence, it is highly likely that quota decisions have contributed to maintaining a sustainable snow crab harvesting industry, which generates approximately $500 million in revenues on an annual basis. If it is assumed that the absence of quotas would threaten the sustainability of the resource, and that FRS plays a key role in supporting quota decisions, it can be concluded that FRS contributes to extensive benefits for the Canadian economy that exceed FRS investments.

Lessons learned include the need for and benefits of nationally consistent approaches and improved data management. Best practices include upfront preparation to ensure clear, timely requests and advice.

Recommendation #3:
FRS should develop options to address gaps in skills and achieve further efficiencies for example, through further leveraging of resources from other sources (e.g. industry, universities, etc.). To support this, FRS should organize a dialogue between FRS and its clients to re-assess research and advisory priorities and to discuss alternatives.

6.0  Management Action Plan

Recommen-
dations

Management Action Plan

Status Report Update

Actions
Completed

Actions
Outstanding

Target Date

Recommen-
dation #1
FRS needs to identify options to address the challenges with moving towards a complex eco-systems based approach.

The National Oceans & Science Director Committee (NOSDC) will, after discussion with its clients, identify and implement actions to increase matching up capacity with demands. Delivery of multi-year stock assessment, one of the building blocks of the ecosystem approach, combined with better integration of ecosystem-related demands, will be a key strategy to respond to this recommendation.  Approach to be approved by Departmental Management Board, or the Sustainable Aquatic Ecosystem Strategic Outcome Committee, as directed.

Initial discussion with EFM to further expand the multi-year stock assessments.

 

 

 

 

 

 

First draft of schedule for multi-year assessments

Second draft of schedule for multi-year assessments

Final schedule for multi-year assessments

Develop process for integration of questions necessitating science advice re ecosystem approach:
- Evaluate current CSAS process and prioritization framework and consider feasibility to include eco-systems criteria

Develop a client operational strategy to match up demands with capacity24:
- Discussion with client sectors (EFM, SAR Management, Habitat Management, Oceans)
- Develop first draft of strategy
- Develop final draft of strategy

April 2011

 

March 2012

 

September 2012

 

 

 

December 2011

 

 

 

 

December 2011

 

 

March 2012

 

June 2012

Recommen-
dation #2
The ADM of Oceans and Science should develop a Performance Measurement Strategy with the intent of informing decision making. The Performance Measurement Strategy will clearly define indicators that align with the outputs and expected outcomes of the program and would include a program profile, a logic model, a performance measurement framework, an evaluation and reporting strategy as well as a data collection strategy.

The current draft FRS Performance Measurement Strategy, with all of its subcomponents, will be revised, after discussion with the clients, to better account for the breadth of activities and products delivered by DFO Science.

First draft of Performance Measurement Strategy, including a program profile, logic model, CSAS performance indicators

Modify FRS PMS to align with the content of the FRS evaluation report

Inventory all non-CSAS science activities or products not accounted for with the current indicators

Identify draft indicators/
performance evaluation method for these non-CSAS science activities

Engage discussions with clients and regions re how science activities will be portrayed in Management PMSs (new PAA)

Modify existing draft of FRS PMS based on discussion with clients and regions and integrating non-CSAS indicators

Finalize FRS PMS after second round of feedback from clients and regions

June 2011

 

 

September 2011

 

 

December 2011

 

 

 

March 2011 and on-going

 

 

December 2011

 

 

 

March 2012

Recommen-
dation 3
FRS should develop options to address gaps in skills and achieve further efficiencies, for example through further leveraging of resources from other sources (e.g. industry, universities, etc.).  To support this, FRS should organize a dialogue between FRS and its clients to re-assess research and advisory priorities and to discuss alternatives.

The FRS evaluation’s findings included gaps in the areas of quantitative analysis and modeling and also pertaining to freshwater.

The evaluation also demonstrated that although a majority of management demands were addressed, a significant number had to be postponed due to insufficient internal capacity. Science needs to respond to priority needs from DFO management. Formalization of the later is therefore a key step here.

Identification of gaps and strength in skills, present and future, first need to be clearly identified in each science node, and then possibilities for inter-regional support investigated. Actions proposed here are building on recent actions completed (e.g. establishment of TESA and Centres of Excellence). The Departmental policy on the role of public vs private science in fisheries management will also have direct implications on the science capacity gap analyses. The long deadlines for actions identified in response to recommendation #3 will allow for the policy to be completed first. Discussion with other research authorities in Canada to explain our science priorities and understanding their own priorities will help prevent overlap in science activities and identify areas where leveraging is possible. Finally, as capacity to conduct stock assessment and all related science requirements will be directly influenced by the plan for multi-year stock assessment (see actions proposed to respond to recommendation #1), the later needs to be developed first if capacity gap analyses are to be useful.

For areas of specific DFO Science responsibilities where gaps in specific skills are identified, the existing HR Successional Plan will be complemented by strategic hiring plans that will be implemented according to financial resources available, drawing from sources of expertise such as participants to the NSERC Capture Fisheries Research Network.

The Technical Expertise in Stock Assessment (TESA) program was specifically established to maintain a critical mass of expertise, or fill gaps, in stock assessment, more specifically in relation with quantitative/modeling analysis.

Centers of Excellence were established to connect existing expertise in key areas and make it available where it was needed.

The NSERC Capture Fisheries Research Network is a 5 years initiative established in 2009 with the participation of DFO Science and with the aim of:

  • generating new knowledge/ technology with the strong potential to strengthen Canada’s industrial base, generate wealth, create employment and/or influence Canadian public policy;;
  • increasing the number of highly qualified personnel in the Capture Fisheries area;
  • foster the increased participation of Canadian-based companies and/or government organizations in academic research;
  • result in the transfer of knowledge/technology and expertise to Canadian-based companies that are well positioned to apply the results for economic gain, or to government organizations to strengthen public policy.

An initial report was completed by the DFO taxonomy working group to investigate the taxonomy needs of the Department.  The initial report is complete, but has not yet been presented to NOSDC. 

TESA to compile global gaps/needs in the Oceans and Science sector with regards to science for stock assessment based on the expected demand on medium-term (5 years).

TESA to provide recommendations at the national level that will take into account regional particularities.

Gaps in scientific expertise, other than that related to stock assessment, are documented in each region.

Provide recommendations to address inter-
regional capacity and gaps not related to stock assessment and not adressed through existing Centers of Excellence.

Discussion with research management authorities (Federal/
Provincial/
Territorial, Academia, NGO,etc.) regarding our respective roles and responsibilities.  Investigate options for leveraging resources through collaborative agreements, sharing resources, exchange of services, etc.

Once DFO has finalized the Framework regarding its responsibility vis-à-vis science advice in support of fisheries (e.g. extent and frequency), DFO Science will initiate discussion with industry to determine needs for science information and advice that would need to be supported by non-government partners.

Develop regional and inter-regional strategic hiring plans and knowledge transfer plans to address remaining science skill gaps.

Continue participation in NSERC Capture Fisheries Research Network; currently funded for 5 years

March 2012

 

 

 

 

 

September 2012

 

 

March 2012

 

 

September 2012

 

 

 

 

March 2012

 

 

 

 

 

 

 

 

 

 

On-going from finalization of Framework

 

 

 

 

 

 

 

 

September 2012

 

 

 

On-going; for duration of Network

ANNEX A 
Evaluation Matrix

Evaluation Question

Key Indicators

Interviews

Docu-
ment/ litera-
ture review

Focus groups

Case
stu-
dies

Sur-
vey of DFO clients

 

 

DFO FRS

DFO
(other)

Sta-
ke
hol-
ders

 

 

 

 

ISSUE 1: RELEVANCE

 

 

 

 

 

 

 

 

1. To what extent are the FRS mandate and activities aligned with Government and Department priorities and objectives?

1.1 FRS mandate and activities relate directly to Departmental strategic outcomes for fisheries and science, to the Government of Canada in fulfilling its management responsibilities nationally and under the UN’s Law of the Sea.*

 

 

 

 

 

 

 

2. Does FRS continue to address the needs of Canadians?

2.1 Informed opinion by various stakeholders that there is an ongoing need for the FRS program and activities*

2.2 Evidence that there is a need for FRS to help Canada meet its international commitments for fisheries management

2.3 Evidence that there is a need for science-based evidence to inform DFO decisions

 

 

 

 

 

 

 

3. Is FRS consistent with federal and DFO roles and responsibilities?

3.1 Views from stakeholders that the federal government/DFO have legitimate roles in providing science-based advice and information for fisheries management

3.2 Evidence that there is consistency between FRS and federal/DFO roles and responsibilities

 

 

 

 

 

 

 

ISSUE 2: PERFORMANCE  
EFFECTIVENESS

 

 

 

 

 

 

 

 

4. To what extent have the intended outputs been produced?

4.1 Evidence that FRS reports are generated and published as planned and on time.* These include:

  1.  Science advisory reports (from 2005 onwards); prior to 2005, stock status, ecosystem status, and habitat status reports;
  2. Research documents;
  3. Proceedings;
  4. Science responses.

4.2 Evidence of other types of outputs besides CSAS publications

 

 

 

 

 

 

 

5. To what extent is FRS achieving its results?

5.1 DFO/Govt clients are aware of FRS and are satisfied with the information and advice it provides

5.2 DFO/Govt clients are using the information and advice from FRS (see question 8)

5.3 External clients are satisfied with the information and advice FRS provides

5.4 External clients are using the information and advice from FRS (see question 8)

 

 

 

 

 

 

 

PERFORMANCE -
EFFICIENCY AND ECONOMY

 

 

 

 

 

 

 

 

6. To what extent is the Fisheries Resources Science governance structure well designed and implemented?

6.1 FRS governance structure is well-articulated, with roles, responsibilities and accountabilities clearly described

6.2 Evidence of science advisory process in place, including prioritization of science requests and management of risks that may include issues related to species, personnel, intellectual property, etc (see question 14)

6.3 Evidence that FRS overall considers issues such as adequate and timely response to science requests, transparent and timely publication of research, and justification on what requests could and could not be met

6.4 Staff and clients are satisfied with the priority setting/ request selection process for FRS

 

 

 

 

 

 

 

7. Is there a performance measurement strategy and a reporting process/system in place to communicate the results of the program?

7.1 Evidence that results/outcomes are well-articulated, and a performance measurement strategy and reporting process/system have been developed for FRS*

 

 

 

 

 

 

 

8. Did the Fisheries Resources Science resource utilization and activities optimally produce expected levels of outputs? How could the efficiency of Fisheries Resources activities be improved?

8.1  Cost of FRS ratios by fishery are comparable or better than OECD country average:

- FRS costs/production of fishery (tons and value of benefits); and
- FRS costs/value of production of the fishery (dollars)

8.2  Extent to which the four functions of FRS (monitoring, data management, research, and advice) are well integrated

8.3  Views from stakeholders on how FRS can be more efficient in using its allocated resources*

8.4  FRS and DFO are satisfied with the amount of resources allocated to meet demands*

 

 

 

 

 

 

 

9. To what extent is FRS incorporating measures to minimize costs in achieving its objectives?

9.1  Evidence of types of cost-saving measures developed and implemented

 

 

 

 

 

 

 

10. To what extent is Fisheries Resources Science economical?

10.1 Trends in the average costs associated with FRS research (by fishery/species) over the last five to ten years

10.2 Views from stakeholders on whether FRS services were provided at the lowest possible cost

 

 

 

 

 

 

 


1 Science for Species at Risk (SSAR) will be evaluated and the findings will be presented in a SSAR Evaluation in 2011-12

2 DFO. Report on Plans and Priorities, 2010-2011.

3 DFO. Report on Plans and Priorities, 2010-2011

4 E.g. national landings of fish, crustaceans, molluscs and algae.

5 OECD. Review of Fisheries in OECD Countries 2009, Policies and Summary Statistics.

6 2008 Canada’s Fisheries Fast Facts, DFO.

7 Does not include aquaculture or seafood processing employees.

8 2009 Canada’s Fisheries Fast Facts, DFO.

9 The 1982 United Nations Convention on the Law of the Sea (UNCLOS) is the defining document of international oceans law. It has been in force since 1994. Its 157 signatories include Canada and all major developed countries. The Convention governs many aspects of oceans affairs, ranging from fisheries and navigation to marine pollution and scientific research.

10 Science Special Response Processes are used to respond to urgent and unforeseen issues or when a fully inclusive and thorough advisory meeting (e.g. standard peer review meeting or workshop) is not required because an advisory framework for the issue has already been developed by a fully SAGE (Scientific Advice for Government Effectiveness) compliant process and the issue is a straight-forward application of the framework.

11 The Wild Salmon Policy reflects a significant new approach to the conservation of the wild Pacific salmon. This new approach specifies clear objectives, establishes strategies to meet them, and presents a decision-making process to ensure that choices made about salmon conservation reflect societal values. The policy places conservation of salmon and their habitats as the first priority for resource management. It gives tangible effect to this principle by committing to safeguard the genetic diversity of wild salmon, and maintain habitat and ecosystem integrity. Its adoption represents Fisheries and Oceans Canada’s commitment to maintain healthy and diverse populations of salmon that will support sustainable fisheries now, and meet the needs of future generations.

12 Eco-certification is a market-based measure intended to improve the sustainability of fisheries.

13 The Ocean to Plate initiative was developed to improve collaboration and policy coherence among government agencies, both within the federal government and with other levels of government with the goal to improve resource sustainability and the competiveness and long-term economic viability of the seafood industry. http://www.dfo-mpo.gc.ca/media/back-fiche/2007/hq-ac17a-eng.htm

14 Note that is some regions, some peer review processes are done outside of CSAS (e.g. NAFO for straddling stocks).

15 CSAS mentioned that while it started formally recording science requests nationally in 2007-08, the requests were not compiled into a formal excel database until 2009. However, the science advisory process itself has been occurring in the regions for more than a decade.

16 This value does not include the “maybes”, which were requests that were awaiting decisions as to whether they were achievable (i.e. enough resources, time, etc). Some requests may have been postponed to the following year.

17 QC and Gulf are only compiling request that they have informally agreed with clients that they can deliver on in the next fiscal year – they control the demand and then they respond to 100% of the mutually agreed on demands.

18 Bibliometric data on DFO publications captured in the research abstract and citation database (Scopus) as well as internally in DFO, from Science-Metrix provided by DFO in January 2010. Science-Metrix also undertook a similar study in March 2009.

19 Scopus is an abstract and citation database of research literature.

20 A peer review science process only include participants, whether from DFO, academia, provincial agencies, industry, Aboriginal groups, ENGOs, who have specific knowledge and expertise to respond to the question defined in the meeting Terms of reference. Observers can also be allowed.  

21 A carapace is a dorsal (upper) section of the exoskeleton or shell of crustaceans including lobsters.

22 2007, DFO. A New Ecosystem Science Framework in Support of Integrated Management. Pg 14.

23 2009 Program Assessment of FRS, pg 3.

24 This action will also contribute to addressing part of the recommendation #3 below dealing with increase efficiency