THE AQUATIC ANIMAL HEALTH PROGRAM

EVALUATION REPORT

6B164

FINAL REPORT
OCTOBER 2014

EVALUATION DIRECTORATE


Table of Contents

ACKNOWLEDGEMENTS AND ACRONYMS


ACKNOWLEDGEMENTS

The Evaluation Directorate acknowledges and thanks all individuals who gave their time and input for this evaluation of the Aquatic Animal Health program. In particular, the Evaluation Directorate acknowledges and thanks all internal and external stakeholder groups and managers who took the time to share their thoughts through site visits, interviews and a questionnaire. The Evaluation Directorate also acknowledges the time and effort given by senior program management in assisting the evaluation team from the planning phase through reporting.

ACRONYMS


List of acronyms
AAH Aquatic Animal Health
CAAHRD Centre for Aquatic Animal Health Research and Diagnostics
CAAPBL Charlottetown Aquatic Animal Pathogen Biocontainment Laboratory
CFIA Canadian Food Inspection Agency
DFO Fisheries and Oceans Canada
DG Director General
DPR Departmental Performance Report
EBP Employee Benefits Plan
FHO Fish Health Official
FHPR Fish Health Protection Regulations
FTE Full time equivalent
ISO International Organization for Standardization
LIMS Laboratory Information Management System
MOU Memorandum of Understanding
NAAHLS National Aquatic Animal Health Laboratory System
NAAHP National Aquatic Animal Health Program
OIE World Organization for Animal Health
PAA Program Alignment Architecture
PM Performance Measurement
PMF Performance Measurement Framework
RMAF Results-based Management Accountability Framework
SPS Agreement on the Application of Sanitary and Phytosanitary Measures
WTO World Trade Organization

EXECUTIVE SUMMARY


Introduction

This report presents the results of an evaluation of the Aquatic Animal Health (AAH) program. The evaluation assessed the program’s relevance and performance, including effectiveness, efficiency and economy, in accordance with Treasury Board’s 2009 Policy on Evaluation. The Evaluation Directorate of Fisheries and Oceans Canada (DFO) conducted the evaluation between June 2013 and June 2014. The evaluation covers all activities undertaken by the AAH program during the period from 2008-09 to 2012-13.

Program Profile

The AAH program is aligned with the strategic outcome of “economically prosperous maritime sectors and fisheries” in the 2013-14 DFO Program Alignment Architecture (PAA). The objective of the program is to reduce the risk of the introduction or spread of serious infectious aquatic animal diseases in order to protect Canadian wild and cultured fish stocks.

The AAH program consists of three components:

  • Regulatory activities for the Fish Health Protection Regulations (FHPR) and the National Aquatic Animal Health Program (NAAHP);
  • Research and development for NAAHP; and
  • Scientific advice, support and dissemination for AAH.  

AAH regulatory responsibilities include compliance with the FHPR under the Fisheries Act. The FHPR was implemented in 1977 to control the import and domestic inter-provincial movement of live salmonid fish and their eggs to prevent the spread of infectious disease of concern. Currently, DFO is the sole responsible authority for FHPR. Compliance with FHPR requires that facilities obtain a Fish Health Certificate and a carrying permit for interprovincial movement of eggs and or fish. The responsibility for the domestic component of the NAAHP is being transferred from DFO to CFIA. As such, it is expected that DFO's current responsibilities regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs will be transferred to CFIA once the domestic component of the NAAHP is ready to be implemented. However, timelines surrounding this transfer of responsibilities have yet to be confirmed.

The objective of the NAAHP is to protect against the introduction or spread of serious infectious disease in wild and cultured aquatic animals that could impact Canada’s domestic aquatic resources. The Canadian Food Inspection Agency (CFIA) provides the overall lead for the NAAHP, while DFO delivers diagnostic testing, research and development, and provides scientific advice to support the program. DFO is meeting its responsibility for the delivery of NAAHP through its National Aquatic Animal Health Laboratory System (NAAHLS).

A key component of NAAHP is research and development. The Centre for Aquatic Animal Health Research and Diagnostics (CAAHRD) funding is provided to NAAHLS for research and development projects which directly benefit NAAHP. 

Finally, AAH also provides a broad array of professional scientific advice and support to various stakeholders. Scientific advisory products are based on expertise of the AAH program and could be on topics such as biology of a pathogen, risk of spreading a pathogen via a movement, on technical consideration around testing, or assessment of mitigation measures.

Evaluation Methodology

This evaluation adopted a theory-based approach whereby program performance was measured against intended results as articulated in the program logic model developed for the evaluation. The chosen design was able to demonstrate the extent to which the program is achieving issues of relevance, efficiency and economy. The analytical methods used for this evaluation were tailored to the nature and availability of the data which were in turn linked to the evaluation questions. Extensive use of triangulation was undertaken as an analytical method, where multiple lines of evidence helped corroborate findings.

The evaluation drew on the following lines of evidence to gather qualitative and quantitative data:

  • Document and administrative data review;
  • Comparative analysis of similar programs delivered in Australia and United States;
  • Key informant interviews with 30 participants (program personnel, stakeholders and CFIA); and
  • Questionnaire administered to a purposive sample of 7 FHPR facilities through telephone interviews.

Evaluation Findings

With regards to the relevance of the AAH program, there is clear evidence of a continued need for AAH regulatory activities such as FHPR and NAAHP to protect Canada’s multi-billion dollar fishery and aquaculture industry. Overall, the AAH program is aligned with the departmental strategic outcome of “economically prosperous maritime sectors and fisheries” and with federal government priorities of economic development and international trade. The AAH program is consistent with federal roles and responsibilities through the Fisheries Act and the FHPR.

With regards to effectiveness, the AAH program has been successful in achieving the majority of intended outcomes as outlined in the evaluation logic model. The extent to which facilities participating in the FHPR program are aware of regulatory obligations is not evident since the program has no mechanism to track awareness and compliance, and there is no consistent means of communicating regulatory obligations to participating facilities. Awareness of FHPR regulations is assumed by all facilities registered in the program. 

The evaluation determined that decision-makers, such as CFIA, have access to validated diagnostic tests. This allows decision-makers to confidently adhere to robust international standards. Data indicates that NAAHLS have completed 97% of tests requested by CFIA and all AAH program personnel and CFIA respondents reported having access to diagnostic tests results provided by NAAHLS. In addition, NAAHLS perform research and development projects to keep pace with the rapidly-evolving technology, to ensure the diagnostic testing is credible and compliant with international reference laboratory standards.

The evaluation also found that program stakeholders have access to scientific advisory products provided by the AAH program.  The full extent of products produced and disseminated by the program is not being captured for reporting purposes. Similarly, the extent to which AAH information is being used to inform decision-making is also not fully known as the program does not officially track the use of scientific products.

Overall, the majority of immediate and intermediate outcomes were achieved with the exception of those requiring tracking mechanisms. This suggests that the long term outcome can be realized from a program logic perspective. According to the available data from 2012 and 2013, no new foreign animal diseases were introduced or spread within Canada for those years. This is consistent with the objective and long term outcome of the program. 

In addition, both positive and negative unintended outcomes were reported.  For example, the implementation of NAAHP has raised the profile of the program with both internal and external stakeholders as well as provided improvements in infrastructure within NAAHLS. Another unintended outcome is the absence of a transition plan for the intended transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs. Although the pending transfer of responsibilities is well known by both departments, a plan for addressing workload and resourcing has not been developed nor have the timelines surrounding the transfer been clearly determined. This places DFO in a position of uncertainty as it impacts on future planning with respect to human and financial resources.

With regards to the efficiency of the program, the evaluation found that NAAHP roles and responsibilities were clearly defined at the outset of the program with governance structures initially in place. However, these structures have been minimally maintained since 2012 and roles and responsibilities for the overall AAH program within DFO are not clearly articulated within the department. 

There is also a need to leverage additional work from provinces and industry for the purposes of surveillance and testing. This was originally intended at the outset of NAAHP. According to the 2010 NAAHP evaluation, these external leveraging networks had not yet begun due to delays in NAAHP implementation. Since then, CFIA has developed a draft policy on third party laboratories and DFO is working with them as a network is established. CFIA respondents stated the absence of a third party network of laboratories is impacting NAAHLS ability to meet the needs of CFIA for the number of diagnostic tests required.

Finally, the evaluation found evidence that the AAH program has been reporting on NAAHP activities in the departmental Performance Measurement Framework (PMF) for the period covered by the evaluation. However, other AAH activities, such as FHPR and the provision of scientific advice are not reflected in departmental reporting documents. In June 2012, the program developed an AAH Performance Measurement (PM) Strategy. Prior to the implementation of the PM Strategy in 2013-14, data was collected and reported in the 2011-12 and 2012-13 DPR. However, the evaluation was unable to determine if the data collected was used to support decision-making. In addition, the indicators currently used in the PM Strategy do not measure all AAH activities and should be revised.

It was concluded that the NAAHP is undertaking activities and delivering products in an efficient manner.  The evaluation found that NAAHP was modelled after CFIA’s Terrestrial Animal Health Program which has demonstrated that it is more cost efficient to prevent the introduction of an exotic disease than to react to it once it has become well established. In addition, NAAHP is similar in design and delivery to programs in other countries. However, it was difficult to assess the efficiency of the overall AAH program, as there was limited evidence of other components of the program such as FHPR and the provision of scientific advice and dissemination.

Finally, with regard to the economy of the AAH program, the evaluation found some duplication within the AAH program and between AAH and other programs.  Elimination or a reduction in the overlap and duplication would provide a more economical use of the program resources.

Recommendations

Recommendation 1:
It is recommended that the AAH program clearly define all activities within the program and communicate them within the department and with stakeholders.

Recommendation 2:
It is recommended that the AAH program revise its Performance Measurement Strategy to include all AAH program activities and collect the necessary data.

Recommendation 3:
It is recommended that the AAH program develop a transition plan, in consultation with CFIA, for the planned transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs. The transition plan should include clearly articulated responsibilities, accountabilities and timelines.

Recommendation 4:
It is recommended that the AAH program consult CFIA to reinstate NAAHP governance structures, including clear roles and responsibilities for both departments.

Recommendation 5:
It is recommended that the AAH program work with CFIA to finalize and implement the draft third party lab policy. The policy should include clearly articulated responsibilities, accountabilities and timelines.

1. INTRODUCTION


1.1 PURPOSE OF THE EVALUATION

This report presents the results of the evaluation of the Aquatic Animal Health (AAH) program. The evaluation was conducted in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation, which requires all direct program spending to be evaluated every five years. The evaluation commenced in June 2013, in accordance with the Departmental Evaluation Plan, and was completed in June 2014. Recommendations stemming from the main findings were developed to support improvements to the program and to inform future decision-making.

1.2 SCOPE

The main objective of this evaluation was to determine the extent to which the program is managed effectively and efficiently and has achieved its stated outcomes. The evaluation covers the five-year period from 2008-09 to 2012-13 and is inclusive of all activities carried out by the AAH as per the program logic model developed for the evaluation. These activities include: 

  • Regulatory activities for the Fish Health Protection Regulations (FHPR) and the National Aquatic Animal Health Program (NAAHP);
  • Research and Development for NAAHP; and
  • Scientific Advice, Support and Dissemination for AAH.

The NAAHP is an interdepartmental program with the Canadian Food Inspection Agency (CFIA) providing lead regulatory and administrative authority for the program and Fisheries and Oceans Canada (DFO) providing diagnostic testing, research, and scientific advice to support the program. For the purpose of this evaluation, only DFO components of NAAHP (further discussed in 2.2 Program Activities) have been assessed. A previous evaluation of the NAAHP was jointly conducted by CFIA and DFO and was completed in 2010. The evaluation covered the period from 2005 to 2010.

1.3 REPORT STRUCTURE

The executive summary provides a brief summary of the evaluation findings, conclusions and recommendations. Section 2 presents an overview of the program. Section 3 describes the evaluation methodology, followed by a discussion of the main findings in section 4, and conclusions and recommendations in section 5.

2. PROGRAM PROFILE


2.1 PROGRAM MANDATE

An integrated approach to aquatic animal health protects the health of Canada's aquatic resources, wild and farmed, and provides greater economic stability and potential for growth for the industries and regions that depend on these resources.  Preventing the spread of aquatic animal diseases within Canada and guarding against the introduction of serious infectious aquatic animal diseases is critical to safeguarding the health of Canada’s aquatic resources and export markets for fish and seafood products. The presence of a serious infectious disease in Canadian waters can be economically devastating for aquaculture operations and the wild fisheries.

The AAH program is aligned with the strategic outcome of “economically prosperous maritime sectors and fisheries” in the 2013-14 DFO Program Alignment Architecture (PAA). The objective of the program is to reduce the risk of the introduction or spread of serious infectious aquatic animal diseases in order to protect Canadian wild and cultured fish stocks. The AAH program consists of three components; regulatory activities for FHPR and NAAHP, research and development for NAAHP, and scientific advice, support and dissemination. Federal responsibilities for the regulation and management of aquatic resources are shared between CFIA’s Health of Animals Act and DFO’s Fisheries Act. The Health of Animals Act provides CFIA with legislative authority to prevent the introduction into and spread within Canada of aquatic animal diseases through the NAAHP while DFO has the responsibility for the enforcement of the Fisheries Act, which includes the FHPR.

2.2 PROGRAM ACTIVITIES

For the purpose of the evaluation, the three main activities of AAH are:
  • Regulatory Activities for FHPR and NAAHP;
  • Research and Development for NAAHP; and
  • Scientific Advice, Support and Dissemination for AAH.

Regulatory Activities for FHPR and NAAHP

Fish Health Protection Regulations
AAH regulatory responsibilities include compliance with FHPR under the Fisheries Act. The FHPR was implemented in 1977 to control the import and domestic inter-provincial movement of live salmonid fish and their eggs to prevent the spread of infectious diseases of concern. Currently DFO is the sole responsible authority for FHPR for domestic movements of salmonids. The responsibility for the domestic component of the NAAHP is being transferred to CFIA. As such, it is expected that DFO's current responsibilities regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs will be transferred to CFIA once the domestic component of the NAAHP is ready to be implemented. However, timelines surrounding this transfer of responsibilities have yet to be confirmed.

Currently, the FHPR concerns itself with domestic interprovincial movement of salmonid species that are at risk for disseminating serious infectious diseases. It is preventative in nature and implements the necessary controls to prevent spread of fish disease agents and to alleviate this serious impediment to development of salmonid fisheries.  The regulations are designed to minimize the risk of the spread of infectious diseases through inspection of wild and cultured salmonids and to control the movement of infected fish between provinces/territories. The regulations apply to all fish species belonging to the salmonid family1 as well as to other fish species if any are reared within the same facility.

Compliance with the FHPR requires that facilities obtain a Fish Health Certificate.  To obtain this Certificate, a facility must have four satisfactory inspections with relevant testing over a period of not less than 18 months. Inspections must be conducted by a Fish Health Official (FHO) at intervals of not less than 90 days and not more than 270 days. On-going inspections and testing by the FHO are required.  In 2012-13, a total of 69 facilities were participating in the FHPR program.  Facilities include all sites used for the propagation or holding of eggs and/or fish.  Interprovincial movement of eggs and/or fish from one facility to another requires an interprovincial carrying permit which is issued by a Local Fish Health Officer. An interprovincial carrying permit may not be issued for the movement of eggs and/or fish originating from a facility that does not have a Fish Health Certificate. The Fish Health Certificate lists the presence or absence of pathogens of concern and informs the decisions of the Local Fish Health Officer as to whether or not to issue a permit.  A Manual of Compliance with guidelines for producers, descriptions of the roles of Fish Health Officials and Local Fish Health Officers, and diagnostic procedures exist to facilitate compliance with the FHPR.

National Aquatic Animal Health Program
The NAAHP was announced in 2005 to respond to international trade challenges based on the lack of a national regulatory control program for aquatic diseases, inadequate disease surveillance, and the inability to provide health certification of aquatic and harvest fisheries. The objective of the NAAHP is to protect against the introduction or spread of serious infectious disease in wild and cultured aquatic animals. Detection and reporting of aquatic animal diseases of international importance in wild and cultured aquatic animals is imperative to prevent and/or control serious disease outbreaks.

The World Organization for Animal Health (OIE) sets international standards to facilitate ongoing trade of aquatic animals. These standards minimize the risk of transfer of infectious diseases that pose a significant threat to vulnerable aquatic animal resources in importing countries. National authorities responsible for animal health are obliged to respect these OIE standards.

The CFIA provides the overall program lead for the NAAHP. Management and coordination of the program is governed through a Memorandum of Understanding (MOU) and a letter of agreement between DFO and CFIA. As overall lead for the NAAHP, the CFIA is responsible for establishment of import health standards of aquatic animals, providing aquatic animal health export certification, surveillance of aquatic diseases, and control measures to prevent spread of disease, while DFO delivers diagnostic testing, research and provides scientific advice to support the program. DFO meets its responsibilities for program delivery through its National Aquatic Animal Health Laboratory System (NAAHLS).

According to foundational documents, the NAAHP is organized into four main inter-related components which include program direction/regulation; field operations; diagnostics and research; and development.

CFIA Components (not evaluated)
Program Direction/Regulation: This includes the establishment of a federal AAH office, responsible for directing and assessing the NAAHP.  Activities include ministerial briefing and support, drafting of AAH regulations, negotiating international trade certification standards, developing a third party laboratory network, etc.
Field Operations: This is the delivery arm of the NAAHP.  Operations is comprised of all field program delivery activities such as, implementation of import inspection and control activities, export testing and certification, on farm/wild surveillance sampling, disease monitoring, etc.

DFO Components (evaluated)
Diagnostics (Laboratory): This involves the diagnostic testing required as a result of disease outbreaks and control activities.
Research and Development: Research under the NAAHP is focused on development/validation to keep pace with the rapidly evolving technology, to ensure that diagnostic testing is compliant and credible with international standards.

The division of roles and responsibilities was further clarified in a letter of agreement between CFIA and DFO dated April 26, 2006. DFO is responsible to develop and maintain the following eight expectations:

  • Establish a core group of laboratories that can serve as National Reference Laboratories for high priority infectious diseases established by the CFIA;
  • Ensure that these laboratories operate under a quality management system to ISO (International Organization for Standardization) 17025 standards;
  • Ensure that these laboratories are in compliance with the aquatic animal pathogen Biocontainment standards;
  • Establish a laboratory system for developing and conducting validated test methods that are fit for purposes and meet international standards for infectious diseases diagnosis required by CFIA;
  • Engage in research as necessary, and test method development to address deficiencies in the current test repertoire;
  • Establish, in conjunction with CFIA, a laboratory information and surveillance information system;
  • DFO will provide laboratory emergency operations / response preparedness as required; and
  • As appropriate, network provincial, academic and private laboratories under a national standard of methodology, quality and proficiency.

DFO NAAHP operational responsibilities are delivered through regional staff via the NAAHLS with direction, coordination and support from the DFO national office in Ottawa. The NAAHLS conduct diagnostic testing and research and development to support the delivery of NAAHP as well as the delivery of FHPR. Four laboratories were selected to create the NAAHLS.  Three of those were chosen based on their work as diagnostic testing laboratories for the department for over 30 years. They are the Gulf Fisheries Centre, Moncton, NB; the Freshwater Institute, Winnipeg, MB and the Pacific Biological Station, Nanaimo, BC. These laboratories already had well-established expertise in aquatic animal health and were responsible for delivering the FHPR and other AAH testing2. The Charlottetown Aquatic Animal Pathogen Biocontainment Laboratory (CAAPBL) in Charlottetown was added as a live-animal research laboratory to complete the NAAHLS. This laboratory was designed specifically to provide the NAAHP with the capacity to perform live-animal challenges with both foreign and domestic diseases3.

ISO accreditation demonstrates that labs provide consistent, credible results and are compliant with international standards. ISO accreditation outlines 230 criteria which must be met to obtain accreditation. Each lab is working towards meeting the required criteria.  Currently, NAAHLS are all within the 90th percentile of completion for ISO 17025 accreditation. As per foundational documents, it was expected that NAAHP would be fully implemented by 2008-09. Full implementation includes a quality assurance / quality control process and ISO 17025 accreditation (or a nationally acceptable and appropriate equivalent standard). According to the program, the timeline for ISO accreditation was not achievable due to several factors, such as the need to renovate existing laboratories to make them compliant to Biocontainment and ISO practices, the fact that other AAH activities had to continue during the same period, validation of test methods were not yet complete and test method needs, were not established until 2011. A new accreditation timeline reflective of program priorities and needs was created with the expectation that application for ISO accreditation would be done by the end of 2014-15.

Research and Development for NAAHP

NAAHP research and development activities are conducted within the NAAHLS. DFO is responsible for ensuring that NAAHPs research requirements are met.  To this end, the NAAHLS support targeted NAAHP research priorities, which are established in collaboration with CFIA through the Centre for Aquatic Animal Health Research and Diagnostics (CAAHRD).  This research can include, but is not limited to, the development and validation of new diagnostic test methods for the detection of infectious disease agents in aquatic animals, research on high priority pathogens, and the susceptibility of aquatic animals to specific pathogens.

Since the launch of the NAAHP, the NAAHLS laboratories have initiated test method validation. Validation is a process that determines the fitness of a test method, which has been properly developed, optimized and standardized, for its intended purpose.  It is an on-going process that assesses and characterizes a test method’s performance over time.  Moreover, a previously validated method may require further validation to account for technological advances, testing of other tissues or testing of a new species.  The validation processes are based on the OIE’s validation guidelines.

Scientific Advice, Support and Dissemination for AAH

As subject matter experts, AAH provides a broad array of professional scientific advice and support to various internal and external stakeholders.  Scientific advisory products are based on the expertise of the AAH program and could be on topics such as the biology of a pathogen, risk of spreading a pathogen via a movement, on technical consideration around testing, or assessment of mitigation measures. The advice is disseminated through various means such as peer reviewed journal articles, presentations at conferences, stakeholder meetings, technical assessments and/or other informal/formal products such as emails.


1 They apply to live and uneviscerated dead cultured fish, eggs (including fertilized eggs or gametes) of cultured and wild fish and products of dead, uneviscerated cultured fish destined to move across provincial boundaries within Canada.

2 The three diagnostic laboratories must have Biocontainment level 2 in vitro as well as meet international ISO 17025 General requirements for the competence of testing and calibration laboratories.

3 The CAAPBL is certified to level 3 Biocontainment standards and is equipped with a state of the art effluent treatment system.

2.3 EXPECTED RESULTS

The AAH program contributes to the department’s strategic outcome of “economically prosperous maritime sectors and fisheries”.  The expected result of the program is that “through its policy, programs and services, and while supporting the sustainable and effective use of Canada’s water resources, DFO contributes to the capacity of Canada’s Maritime Sectors and Fisheries to derive economic benefits and further enhance their competitiveness”. By protecting the competitiveness of Canadian seafood exports and encouraging investment in seafood production, the AAH program provides safeguards to the multi-billion dollar fishery and aquaculture industry.

The program contributes to the strategic outcome specifically as follows:

  • FHPR facilities are aware of regulatory obligations (immediate outcome)
  • Decision-makers have access to validated diagnostic test results (immediate outcome)
  • Stakeholders have access to scientific products (immediate outcome)
  • FHPR facilities comply with regulation (intermediate outcome)
  • Stakeholders make informed decisions based on scientific products (intermediate outcome)
  • Canada’s wild and cultured aquatic resources are protected from the introduction and spread of disease of concern (long-term outcome)

2.4 PROGRAM LOGIC MODEL AND PERFORMANCE MEASUREMENT

For the purpose of this evaluation, the logic model was developed in consultation with key AAH program personnel to streamline program activities and outcomes. The logic model covers key activities undertaken by the program during the five-year period from 2008-09 to 2012-13. It reflects the three main activities discussed above, namely regulatory activities, research and development and scientific advice, support and dissemination.

Aquatic Animal Health Program Logic Model

aquatic animal health program logic model

2.5 PROGRAM STAKEHOLDERS

Stakeholders of AAH include the CFIA, other DFO programs, other federal departments and agencies, provincial and territorial governments, Aboriginal communities, commercial fishery, recreational fishery, aquaculture industries, as well as scientific and academic communities.

Beneficiaries include Canadians at large, and more specifically First Nations, fishers, seafood producers, processors, distributors, retailers, exporters, and importers and the aquarium industry. Within DFO, there are several programs and processes that benefit from both the diagnostic testing and scientific research and development component,  such as Aquaculture Management, the Introduction and Transfers process, and the Salmon Enhancement Program.

2.6 GOVERNANCE

The AAH program is administered by the Strategic & Regulatory Science Directorate within the Ecosystems Science Sector in DFO.  The operational component of the program is delivered in the regions through the NAAHLS, which include the Gulf Fisheries Centre in Moncton, NB; the Charlottetown Aquatic Animal Pathogen Biocontainment Laboratory in Charlottetown, PE; the Freshwater Institute in Winnipeg, MB; and the Pacific Biological Station, Nanaimo in BC.  Regionally, the NAAHLS report to their respective Division Managers and Science Directors.  However, the NAAHLS have a functional reporting relationship to the AAH program in the National Capital Region.

Inter-regional DFO program coordination of the NAAHLS is supplied through a National Aquatic Animal Health Regulatory Science Committee which is chaired by the DFO National Manager of the Aquatic Animal Health Program with the National Lab Manager, the national coordinators (accreditation, diagnostic, research, Laboratory Information Management System (LIMS)) and the laboratory heads from each of the applicable regions participating.  The committee advises the Director, Aquaculture, Biotechnology and Aquatic Animal Health Science on operational matters, strategic directions, program design and implementation. 

The Deputy Minister of DFO is accountable to the Minister for the development and delivery of the DFO component of the NAAHP and other regulatory aquatic animal health activities.  Management and coordination of the NAAHP component of AAH is governed through a series of committees set by the MOU signed in 2006 between the CFIA and DFO. These committees include a Director General (DG) level Steering Committee and an Aquatic Animal Health Committee.  The DG level Steering Committee consists of DFO and CFIA officials that were responsible for various aspects of the NAAHP, including planning, advising, providing strategic direction, monitoring and analysis. This committee has met sporadically since inception of the program. The Aquatic Animal Health Committee is comprised of members from the veterinary community, the fishing and aquatic industries, the provincial/territorial governments and the Aboriginal community to advise the National Director of the Aquatic Animal Health Division (CFIA) and the manager of the Aquatic Animal Health program (DFO), on the development and operational matters pertaining to the NAAHP. The Aquatic Animal Health Committee has been active and met approximately twice a year.

2.7 PROGRAM RESOURCES

The following table presents the AAH expenditures and full time equivalents (FTEs) by fiscal year, for the period of 2008-09 to 2012-13. Table 1.0 depicts the total AAH authorities as outlined in the Departmental Performance Report (DPR) and expenditures, inclusive of NAAHP. Table 2.0 describes the initial commitment, budget and expenditures for NAAHP (DFO component only). Table 3.0 captures the FTEs for AAH.

Table 1.0: Financial Information - AAH (inclusive of NAAHP) in millions
  2008-09 2009-10* 2010-11 2011-12 2012-13
Authorities (DPR) 6,065.50 9,039.94 6,082.95 6,096.59 6,082.63
Expenditure 6,910.11 7,229.64 6,179.53 6,298.86 6,015.34

Source: DFO Finance Sector
*Please note the increase in 2009-10 authorities relates to the Aquatic Invasive Species coded to the AAH program activities; a correction has been made in 2010-11.

Table 2.0: Financial Information - NAAHP (DFO component only) in millions
  2008-09 2009-10 2010-11 2011-12 2012-13
Initial Commitments* 5,480.00 5,480.00 5,480.00 5,480.00 5,480.00
Budget** 5,167.85 4,316.52 4,554.80 4,684.75 4,113.21
Expenditure *** 5,242.31 4,265.86 4,054.00 4,190.79 3,978.43

Source: DFO Finance Sector
*Initial commitments as identified in foundational documents include: a) an amount from existing DFO reference levels of $1,500.00K and b) accommodation costs of $207.8K.  Expenditures are DFO only and do not include accommodation costs.
** Year-end budget includes EBP
*** Expenditure includes EBP

Table 3.0: AAH Full Time Equivalents
Year AAH NAAHP Non-NAAHP
2008-09 53.33 36.55 16.78
2009-10 56.00 36.11 19.89
2010-11 55.75 35.57 20.18
2011-12 53.09 33.37 19.72
2012-13 52.74 32.76 19.99

Source: DFO Finance Sector

2.8 RISK PROFILE

The AAH program identified4 three key risks that have potential of affecting the attainment of program objectives:

  • Partnerships: If an effective interdepartmental governance structure is not achieved then there is a risk that DFO and CFIA will not share a common understanding of the program priorities. This would lead to a divergence between program expectations and delivery capacity compromising the program’s ability to supply decision-makers access to sufficient diagnostic test results and to knowledge and advice to support the regulation of aquatic animal health.
  • Limited diagnostic testing capacity: If current levels of program resources are maintained then there is a risk that there will be insufficient laboratory capacity to respond to increasing industry demands and emerging diseases. This would lead to an insufficient number of diagnostic testing results available to decision-makers that would compromise their ability to effectively regulate aquatic animal health in Canada.
  • Limited scientific research capacity: If scientific research capacity remains unchanged then there is a risk that the program will not be able to address new or emerging aquatic animal diseases. This would lead to insufficient diagnostic testing methodology scope such that decision-makers would not have access to knowledge and advice necessary to support effective regulation of aquatic animal health.

4 Aquatic Animal Health Program Risk Profile, February 2014

3. METHODOLOGY


3.1 EVALUATION APPROACH & DESIGN

This evaluation adopted a theory-based approach whereby program performance was measured against intended results articulated in the program logic model.  The chosen design was able to demonstrate the extent to which the program is achieving issues of relevance, efficiency and economy.

The analytical methods used for this evaluation were tailored to the nature and availability of the data which were in turn linked to the evaluation questions. Extensive use of triangulation was undertaken as an analytical method, where multiple lines of evidence helped corroborate findings.

3.2 KEY ISSUES & EVALUATION QUESTIONS

The evaluation questions covered relevance and performance, including effectiveness, efficiency and economy. The evaluation questions were determined on the basis of the Policy on Evaluation (2009), a review of key program documents, and results from preliminary discussions with key program personnel. See Annex A for the detailed Evaluation Matrix.

3.3 DATA SOURCES

The evaluation drew on the following lines of evidence to gather data.

3.3.1 DOCUMENT REVIEW

The review of relevant program documentation provided a comprehensive perspective of the activities, outputs and outcomes completed by the program, and were used to inform the assessment of both relevance and performance.

The document review included an examination of various program documents and administrative data such as: planning documents, regulations and manuals, program performance measurement strategy, financial reports, previous NAAHP evaluation and various other documents related to the program.

3.3.2 COMPARATIVE ANALYSIS

A scan of public documents from Australia and United States was conducted to draw conclusions regarding similarities and differences in program design and delivery, as well as situate NAAHP in an international context.

3.3.3 KEY INFORMANT INTERVIEWS

Key informant interviews afforded insight into the perceptions, opinions and experiences of individuals who have a significant role or experience in the design and delivery of the program.

A total of 30 interviews were conducted, either by phone or in person, with key informants from the following groups:

  • National AAH program personnel (n=6)
  • Regional AAH program personnel (n=14)
  • DFO internal stakeholders (n=3)
  • CFIA (n=7)

3.3.4 QUESTIONNAIRE

A questionnaire was administered to a purposive sample of FHPR facilities through telephone interviews.  A sample of 7 FHPR facilities were selected from a total of 69 facilities based on regional location, size and availability. The questionnaire provided both qualitative and quantitative input.

3.4 METHODOLOGICAL LIMITATIONS & MITIGATION STRATEGIES

Limited Documentation. The majority of documents reviewed as part of this evaluation where NAAHP specific and did not address all activities carried out by the AAH program.  Rigour was improved by using additional lines of evidence such as interviews and questionnaires.

Sample size for Questionnaire of FHPR Facilities. FHPR facilities are key participants in the AAH program. The evaluation team was not able to conduct an online survey with the entire FHPR facilities as several of them are not accessible by email, and given time constraints, a survey by mail was not a practical alternative. Instead, the evaluation team decided to conduct a phone interview with a purposive sample of FHPR facilities in each region. Originally 13 facilities were selected but after several attempts, only 7 facilities could be reached. Given that this sub-group is not necessarily representative of all FHPR facilities, the questionnaire results cannot be generalized to the entire population of FHPR facilities, but they nevertheless provided some insight and informed the evaluation in combination with the other lines of evidence.

4. Major Findings


4.1 Relevance

Evaluation Question: Is there a continued need for the AAH program? To what extent is the AAH program aligned to departmental and federal government priorities? Is the AAH program consistent with federal roles and responsibilities?


There is clear evidence of a continued need for AAH regulatory activities such as FHPR and NAAHP to protect Canada’s multi-billion dollar fishery and aquaculture industry. Overall the AAH program is aligned with the departmental strategic outcome of “economically prosperous maritime sectors and fisheries” and with the federal government priorities of economic development and international trade.  The AAH program is consistent with federal roles and responsibilities through the Fisheries Act and the Fish Health Protection Regulations.


Continued Need

Disease is one of the most significant challenges facing sustainable productivity of aquaculture and harvest fisheries. In 2012, Canada exported fish and seafood products to 118 countries valued at $4.14 billion5. Protecting the health of aquatic resources in Canada is essential to ensure the long-term viability of this industry. Without appropriate and effective disease prevention measures, both farmed and wild fish populations are at risk from introduction and spread of disease. A coordinated federal aquatic animal health program is needed to meet new international trade requirements and to ensure that the Canadian seafood industry remains competitive. By protecting the competitiveness of Canadian seafood exports and encouraging investment in seafood production, safeguards will be in place for the multi-billion dollar fishery and aquaculture industry.

Studies of aquatic animal health programs in other countries suggest that public spending on animal health management provides a return on investment. Specifically in the case of the NAAHP, a detailed cost-benefit analysis of the program6 concluded that the costs are relatively small compared to the potential harm to the industry if it were to be seriously affected by disease introduction or spread. For example, an outbreak of Viral Hemorrhagic Septicemia in the late 1990s caused massive die-off among pilchards and herring in British Columbia and was estimated to cost the industry $20 million.

In 2004, the Report of the Commissioner of the Environment and Sustainable Development highlighted the need for a national aquatic animal health program in order to protect fish health in both aquaculture and wild stocks. The program is needed for trade purposes given that many countries expect Canadian exports of aquatic animal products to meet international standards established by the World Organization of Animal Health.  The Canadian Council of Fisheries and Aquaculture Minister also endorsed the creation of the program, along with all provinces, territories, aquatic veterinarians, and industry, including Canada’s aquaculture sector, who have strongly and repeatedly expressed the need for aquaculture to have access to a health management program equivalent to that available to terrestrial farm animals. All DFO and CFIA key informants interviewed as part of this evaluation agreed that there is a continued need for this program.

Departmental and Federal Government Priorities

The AAH program is directly linked to DFO’s strategic outcome of “economical prosperous maritime sectors and fisheries”. The objective of the program is to reduce the risk of the introduction or spread of serious infectious aquatic animal diseases in order to protect Canadian wild and cultured stocks. DFO contributes to this through its regulatory activities (NAAHP and FHPR) and through provision of scientific advice, support and dissemination.  AAH program personnel interviewed as part of the evaluation were of the opinion that the program aligns with the current federal and departmental priorities. 

Economic development has been a long-term priority of the federal government. By protecting the competitiveness of Canadian seafood exports and encouraging investment in seafood production, the NAAHP is supporting economic development of many small, coastal and rural communities that rely on the fishery and aquaculture industry. The federal government has also committed to protecting Canada’s traditional industries, such as fisheries, through measures aimed at marketing Canadian products abroad.

Federal Roles and Responsibilities

Canada is a signatory of international agreements, such as the Agreement on the Application of Sanitary and Phytosanitary Measures (SPS Agreement) of the World Trade Organization (WTO) and the scientific standards set for aquatic animal health management (the Aquatic Animal Health Code) by the World Organization for Animal Health (OIE). Under the SPS Agreement, the WTO sets constraints on member-state’s policies relating to food safety (bacterial contaminants, pesticides, inspection and labelling) as well as animal and plant health with respect to imported pests and diseases. The Aquatic Animal Health Code sets out standards for the improvement of aquatic animal health and welfare of farmed fish worldwide, including through standards for safe international trade in aquatic animals and their products.  As a member of the WTO and the OIE, and signatory of these international agreements, the federal government is responsible for aquatic animal health issues affecting international and inter-provincial trade matters and providing leadership in the development and implementation of the NAAHP program, which is national in scope.

Federal responsibilities for the regulation and management of aquatic resources are shared between CFIA’s Health of Animals Act and DFO’s Fisheries Act.  The Health of Animals Act provides CFIA with legislative authority for the NAAHP while DFO has the responsibility for the enforcement of the Fisheries Act, which includes the FHPR. More than half of DFO internal stakeholders interviewed were of the opinion that the AAH program is consistent with current federal roles and responsibilities.


5  http://www.dfo-mpo.gc.ca/media/npress-communique/2013/hq-ac03-eng.htm

6 2010 Evaluation of the National Aquatic Animal Health Program ( http://epe.lac-bac.gc.ca/100/206/301/cfia-acia/2011-09-21/www.inspection.gc.ca/english/agen/eval/aqua/evale.shtml)

4.2 EFFECTIVENESS

FISH HEALTH PROTECTION REGULATIONS

Evaluation Question: To what extent are facilities aware of regulatory obligations? To what extent do facilities comply with regulations?


The extent to which facilities are aware of FHPR regulatory obligations is not evident as there is no official tracking of awareness and no consistent means of communicating regulatory obligations to participating facilities. Awareness is assumed by facilities’ participation in the program. The evaluation was also unable to assess the extent to which facilities comply with regulatory obligations as there was no evidence of a mechanism to track compliance and it is therefore difficult to conclude whether there are compliance issues.


Awareness of Regulatory Obligations

In 1977, the AAH program developed a Fish Health Regulations Manual of Compliance which was since updated in 2011. The purpose of the manual is to outline the administrative and inspection procedures to be followed as well as provide a step-by-step procedure for handling fish samples for testing. The manual is available online. Other than the Manual of Compliance, the evaluation found no evidence of communications sent to facilities participating in the FHPR program regarding their regulatory obligations.

The AAH program does not track facilities’ awareness of FHPR regulatory obligations. Facilities’ awareness is assumed through their participation in the FHPR program. Participation is defined as facilities that hold a current fish health certificate for the purpose of moving salmonids into other provinces.

Table 4.0: Number of Facilities Participating in FHPR
Year 2008-09 2009-10 2010-11 2011-12 2012-13
Number of Facilities participating in FHPR 83 80 91 79 69

Almost all AAH program personnel interviewed stated that FHPR facilities are aware of their regulatory obligations but were not able to communicate how they became aware of the obligations.  According to FHPR facility staff interviewed, there was no consistent means as to how FHPR obligations were communicated. For example, each interviewee became aware of the program and its regulatory obligations in a different manner, such as a previous owner, a DFO representative or the province.

Compliance with FHPR

The evaluation found that the AAH program does not have a mechanism to track facilities’ compliance with FHPR and as such there is no formal record of any compliance issues between 2008-09 and 2012-13.  More than half of the AAH program personnel interviewed stated that they were not aware of any FHPR compliance issues.  However, just over a quarter of AAH program personnel located in the regions did identify some instances where facilities were not in compliance with the FHPR.  This implies there may be compliance issues. However, with no mechanism to track compliance, it is difficult to conclude there are in fact issues.

NATIONAL AQUATIC ANIMAL HEALTH PROGRAM

Evaluation Question: To what extent do decision-makers have access to validated diagnostic test results?


CFIA has access to validated diagnostic test results.


Validated Diagnostic Tests and Results

As lead of the overall NAAHP design and direction, CFIA is responsible for determining the number and kind of diagnostic tests needed for program delivery each year. Having access to validated diagnostic test results allows decision makers, such as CFIA, to confidently adhere to robust international standards.  As of 2012-13, NAAHLS has successfully validated 20 test methods, four of which have completed protocols (infectious hematopoietic necrosis virus (IHNV), infectious pancreatic necrosis virus (IPNV), infectious salmon anemia virus (ISAV) and viral hemorrhagic septicemia virus (VHSV)) and dossiers. These are the most commonly requested by CFIA.

Due to the delays in implementation of NAAHP, diagnostic testing only officially began in 2012-13 and as a result, there is no official data available between 2008-09 and 2011-12 regarding the number of tests requested by CFIA and subsequently conducted by NAAHLS. According to the most recent administrative data, CFIA forecasted approximately three times more tests than they actually requested be conducted by NAAHLS. More specifically, in 2012-13, CFIA forecasted between 23,986 and 56,910 tests would be required but only ended up submitting official requests for 16,277 tests to be conducted by NAAHLS. Data indicates that NAAHLS completed 97% of those tests. The significant gap between forecasted and actual tests requested by CFIA poses a challenge for NAAHLS to efficiently plan operational and human resource needs.

Access to Diagnostic Test Results

All AAH program personnel and CFIA respondents interviewed reported having access to diagnostic test results provided by NAAHLS. According to CFIA respondents, access was viewed as limited as a result of LIMS not being directly accessible as originally intended. In addition, the timeliness of test results did not always meet CFIA decision making needs. LIMS was originally designed to be a shared information management system between NAAHLS and CFIA, however, due to technological incompatibilities, it was determined that LIMS would only be used within NAAHLS. As a result, LIMS evolved into a web-based laboratory information system that was implemented nationwide within NAAHLS to allow total traceability and tracking of NAAHP samples from point of collection through to reporting of results. As such, LIMS is not accessible to CFIA, such that they cannot enter submission data or retrieve reports of analysis directly from the system.  NAAHLS staff must therefore submit reports to CFIA outside of the LIMS system, via a separate transaction. For example, NAAHLS is to input data into LIMS for their own operational requirements as well as prepare a separate report to send to CFIA.

According to CFIA respondents, this method of information sharing results in CFIA receiving information in a format that is difficult to read and interpret and often not the type of information they desire. In addition, nearly three quarters of CFIA respondents interviewed mentioned that access to diagnostic test results is not timely and that turnaround time is an issue impacting their work.

Research and Development in Support of NAAHP

A key component of NAAHP is research and development. Research under NAAHP maintains and builds the science-based underpinnings to risk analysis, export certification and various elements of the regulatory program. This research includes test development and validation to keep pace with the rapidly-evolving technology, to ensure the diagnostic testing is credible and compliant with the international reference laboratory standards.

The Centre for Aquatic Animal Health Research and Diagnostics (CAAHRD) funding is provided to NAAHLS for research and development projects which directly benefits NAAHP.  CAARHD funding has been used by NAAHLS to perform research for the development and validation of diagnostic tests and methodologies, to substantiate and improve quality management practices in the laboratories, to increase knowledge of pathogen host interactions, to test susceptibility, and to assess in-vivo challenges with foreign animal diseases. Over the last five years, the number of projects funded in support of NAAHP objectives has remained fairly constant further supporting the development and validation of diagnostic tests thus improving access to validated diagnostic test results.

Table 6.0: Number of CAAHRD projects that support NAAHP objectives
number of CAAHRD projects funded by year
Year 2008-09 2009-10 2010-11 2011-12 2012-13
Number of CAAHRD projects funded 9 7 12 10 8

SCIENTIFIC ADVICE, SUPPORT AND DISSEMINATION ACTIVITY

Evaluation Question: To what extent do stakeholders have access to scientific products? To what extent do stakeholders make informed decisions based on scientific products?


Internal and external stakeholders have access to scientific advisory products provided by the AAH program. The full extent of products produced and disseminated is not being captured by the program. While the AAH program makes scientific advisory products available to stakeholders, the extent of use to inform decision making is not fully known. The program does not officially track the use of scientific advisory products.


Access to Scientific Advisory Products

For the purpose of this evaluation, scientific advisory products have been defined as any AAH advice and/or products that are the objective synthesis of scientific and technical information for use by internal and external stakeholders.  Scientific advisory products are based on the expertise of the AAH program and could be on topics such as the biology of a pathogen, risk of spreading a pathogen via a movement, technical considerations around testing, or assessment of mitigation measures.

It was reported in the 2012-13 Departmental Performance Report (DPR) that 100% of requested scientific advice was provided to stakeholders. More specifically, the program provided scientific advice and technical assessments for 721 applications for the Introductions and Transfers committee and 7 methods assessments which were requested by the CFIA.   

The National Code on Introductions and Transfers of Aquatic Organisms is an objective decision-making framework and consistent national process for assessing and managing the potential ecological, disease and genetic risks associated with intentionally moving live aquatic organisms into, between or within Canadian watersheds and fish-rearing facilities. The introductions and transfers process includes a committee of experts who assess the potential habitat, genetic or health impact of a proposed release or transfer of fish (which require an authorization pursuant to the Fishery (General) Regulations enacted under the Fisheries Act) into receiving waters. NAAHLS is responsible for providing scientific advice on aquatic animal health elements of proposed movements, specifically i) the health data provided (e.g. interpretation of information, appropriateness of methodology, etc.) and gaps, if applicable, ii) if there are specific health risks associated to the proposed movement, and iii) the effectiveness of mitigation measures, if applicable.  The scientific advice is provided to the introductions and transfers committee and is used as part of the assessment to determine movement.   

All DFO internal stakeholders interviewed stated having access to scientific advisory products produced by AAH. According to interviewees, the following internal and external stakeholders received scientific advisory products: Introductions and Transfers committee, Aquaculture Management, DFO legal services, as well as CFIA, industry, provinces and academia. Interviewees referenced far more scientific advisory products than were identified and tracked by the PM Strategy for reporting purposes. For example, interviewees referenced scientific advisory products were made available through presentations, meetings, research projects, published literature, peer reviewed journals, conference and online. This indicates that the program is not capturing the full scope of scientific advisory products produced and disseminated to internal and external stakeholders.

Informed Decision Making

Three quarters of AAH program personnel interviewed believe that stakeholders are using scientific advisory products to inform decisions. AAH makes their scientific advisory products available to decision makers however the extent of use to inform decision making is unknown. It is assumed that scientific advisory products are used by stakeholders for decision making purposes but the program has no way of knowing because there is no formal tracking mechanism.

The scientific advisory products provided by the AAH program are not necessarily the sole product considered when making informed decisions. Stakeholders who seek specific scientific advice from AAH also consider information and advice from other sources when making informed decision. For example, with respect to issuing an introductions and transfer licence, AAH scientific advisory products are considered in addition to other data and information. Therefore the decision to issue an introduction and transfer licence takes into account several sources of data and not solely the AAH scientific advice. As a result, a final decision does not necessarily solely reflect the AAH scientific advisory products.

LONG-TERM OUTCOME

Evaluation Question: To what extent has the AAH program achieved its long-term outcomes?


The majority of the immediate and intermediate outcomes were achieved with the exception of those requiring tracking mechanisms. This suggests that the AAH long term outcome can be realized from a program logic perspective. Evidence also suggests that no new foreign animal diseases were introduced and/or spread within Canada for the years 2012 and 2013. This is consistent with the objective of the long term outcome.


According to program logic, the overall achievement of the long term outcome is expected to stem from the achievement of the immediate and intermediate outcomes. The evaluation found that the majority of the immediate and intermediate outcomes were achieved with the exception of those requiring tracking mechanisms. For example, the extent of awareness and compliance of FHPR regulatory obligations is not evident in addition to the extent of use of AAH scientific advisory products to inform decision making is not fully known. Nevertheless, there is no evidence to suggest that the long-term outcome will not be fully achieved once appropriate tracking mechanisms are implemented.

To assess progress towards achieving the AAH program long term outcome, the evaluation examined the incidence of introduction and/or spread of foreign animal disease into Canada. According to available data for years 2012 and 2013, there were no incidences of new foreign animal diseases reported by CFIA7, thus indicating no new foreign animal diseases were introduced into the country during this period. This is consistent with the objective of the AAH long term outcome of protecting Canada’s wild and cultured aquatic resources from the introduction or spread of disease.

UNINTENDED OUTCOMES

Evaluation Question: Have there been any unintended (positive or negative) outcomes?


The evaluation found both positive and negative unintended outcomes which include the raised profile of the program as a result of NAAHP in addition to the absence of a transition plan for the intended transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs.


Several unintended program outcomes were identified during the interview process with AAH program personnel and CFIA respondents. The implementation of NAAHP has raised the profile of the program with both internal and external stakeholders as well as provided improvements in infrastructure within NAAHLS. For example, the influx of money NAAHLS received because of NAAHP has enabled the labs to improve infrastructure which would have otherwise not been possible. Improved infrastructure has contributed to the ISO accreditation process which ultimately raised the profile of NAAHP both domestically and internationally.

The absence of a transition plan for the intended transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs was also cited as a common unintended outcome. Although the pending transfer of responsibilities is well known by both departments, a plan for addressing workload and resourcing has not been developed nor have the timelines surrounding the transfer been clearly determined. This places DFO in a position of uncertainty as it impacts on future planning with respect to human and financial resources.


7 http://inspection.gc.ca/animals/aquatic-animals/diseases/reportable/2014/eng/1339174937153/1339175227861

4.3 EFFICIENCY

DESIGN AND DELIVERY

Evaluation Question: Is the AAH program design and delivery appropriate for achieving program results?


Overall AAH program roles and responsibilities and activities are not well understood within the department. With regard to NAAHP roles and responsibilities, the evaluation found that they were defined at the outset of the program but are not well understood between departments. NAAHP governance structures were initially established but have been minimally maintained since 2012.

The evaluation found that program resources / capacity for FHPR and NAAHP are not fully adequate for the achievement of expected results. The continued pressure to deliver regulatory activities for interprovincial carrying of live salmonid fish and their eggs that will be transferred from DFO to CFIA and the establishment of a third party network are contributing factors.


Governance Structure

According to foundational documents, the development and implementation of NAAHP is to be co-delivered by DFO and CFIA. CFIA is responsible for leading all federal aquatic animal health management activities, which includes the overall program design and direction of NAAHP. DFO is responsible for providing all aquatic animal health laboratory services as well as supporting the implementation of NAAHP. When NAAHP was originally designed (2005-06), it was organized into four main inter-related components which include program direction/regulation; field operations; diagnostics and research and development. The following year (April 2006), the division of roles and responsibilities in the delivery of NAAHP were further clarified in a letter of agreement signed between CFIA and DFO. As per the letter, eight expectations were provided by CFIA to guide DFO’s support in the development and implementation of NAAHP.

The evaluation found that five of the eight DFO expectations have been achieved while the other three (ensuring that laboratories operate under a quality management system to ISO 17025 standards; establishing, in conjunction with CFIA a laboratory information and surveillance information system; and, as appropriate, network provincial, academic and private laboratories under a national standard of methodology, quality and proficiency) are still in various stages of completion. 

Interview results suggest NAAHP roles and responsibilities are not well understood between CFIA and DFO. According to more than half of AAH program personnel interviewed, roles and responsibilities with regard to NAAHP are clearly defined but not well understood. Although the roles and responsibilities are outlined in foundational documents, the most common reason cited for roles and responsibilities’ not being well understood is the lack of communication between departments. Most of the CFIA respondents interviewed stated the roles and responsibilities between themselves and DFO were not clearly defined or understood. The lack of clarity and understanding were viewed as a result of two different departments jointly managing a program under two different departmental mandates and priorities.

While the document review found the NAAHP roles and responsibilities to be clearly defined at the outset of the program, there was no evidence to suggest these roles and responsibilities have been revised to reflect the current operating environment. This is supported by interview results which suggest that the roles and responsibilities are not well understood between departments. This is further impacted by the limited activity of NAAHP governance structures.

According to the document review, an Aquatic Animal Health Committee was in place with meetings held approximately twice per year. A joint DFO-CFIA national management team was also established although regular meetings were sporadic after the 2012 re-organizational structure occurred within CFIA. The evaluation found marginal evidence that the DG-level Steering Committee occurred. Due to minimal NAAHP governance structures, DFO and CFIA are operating within their own internal governance structures. This disjointed delivery approach is impacting communications between departments and ultimately impacting the delivery of NAAHP.

With respect to the overall AAH program, more than half of AAH program personnel stated that AAH roles and responsibilities internal to DFO were defined but not well understood within the department. Several interviewees were of the opinion that management within DFO does not fully understand what activities are being conducted within in the AAH program. However it was felt that this may be a result of the program not effectively communicating what it is they do.

ISO 17025 Accreditation

The OIE sets health standards to facilitate ongoing international trade in aquatic animals.  These standards minimize the risk of transfer of named infectious diseases that pose a significant threat to vulnerable aquatic animal resources in importing countries. National authorities responsible for animal health are expected to base their aquatic animal health control measures on the OIE standards, including the requirement for diagnostic facilities to be accredited against a quality management standard such as ISO 17025 or the OIE Standard. The process includes the use of validated diagnostic test methods. ISO accreditation demonstrates that laboratories provide decision makers with consistent credible results and are compliant with international standards.

NAAHLS have implemented a quality management system with the aim of obtaining accreditation under ISO 17025. In order for NAAHLS to become ISO accredited, each lab must meet 230 required criteria as identified in ISO 17025 General requirements for the competence of testing and calibration laboratories. Since the inception of the NAAHP, NAAHLS has steadily increased the number of criteria it meets as required for accreditation. The administrative data demonstrates that NAAHLS are within the 90 percent completion for accreditation.  To date, the Fresh Water Institute in Manitoba is 100% complete; the Pacific Biological Station in BC is 99% complete and the Gulf Fisheries Centre in Moncton is 91% complete. The Pacific Biological Station in BC has been 99% complete since December 2010. The Charlottetown Aquatic Animal Pathogen and Biocontainment Facility has a quality management system in place, but accreditation is not required as the facility’s focus is solely on research.

Table 5.0: Criteria completed in support of ISO 17025 Accreditation criteria completed in support of ISO 17025 accreditation

As per foundational documents, it was expected that NAAHP would be fully implemented by 2008-09. Full implementation includes a quality assurance / quality control process and ISO 17025 accreditation (or a nationally acceptable and appropriate equivalent standard). According to the program, the timeline for ISO accreditation was not achievable due to several factors, such as the need to renovate existing laboratories to make them compliant to Biocontainment and ISO practices, the fact that other AAH activities had to continue during the same period, validation of test methods were not yet complete and test method needs, were not established until 2011. In 2012, a letter was exchanged between CFIA and DFO stating the intention to achieve ISO accreditation by the end of fiscal 2014-15. A broad multi-year plan was developed by DFO outlining steps and requirements for each lab from 2013 to 2015-16 in order to achieve accreditation. Until NAAHLS has reached full ISO accreditation, Canada cannot request that testing in other countries be done in an ISO accredited laboratory because they themselves are not yet ISO accredited.

Factors Affecting the Achievement of Expected Results

Results from the 2010 NAAHP evaluation found that CFIA is responsible for mandatory reporting of disease, import control, emergency response and domestic disease control under the NAAHP. However, until CFIA implements its domestic disease control measures, DFO will remain responsible for delivering these activities. This responsibility has put added pressure on NAAHLS to deliver both the FHPR and the NAAHP diagnostic work simultaneously. Once responsibilities regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs are transferred from DFO to CFIA, resources at NAAHLS could be redirected to contribute to furthering the goals of the NAAHP. According to foundational documents, any significant change in workload as a result of changes in responsibilities will be discussed in advance and agreed upon by both parties. As such, a transition plan for addressing changes in workload with respect to the intended transfer of responsibilities from DFO to CFIA regarding regulatory activities should be discussed by CFIA and DFO; however the evaluation was unable to find evidence of such a plan.

When program resources were allocated at the outset of NAAHP, they took into account that the program would be leveraging work from provinces and industry for the purposes of surveillance and testing. According to the 2010 NAAHP evaluation, these relationships had yet to begin due to delays in implementation. As a result, the following recommendation was developed; the program should make use of CFIA’s assessment of the surveillance and monitoring capacity of the provinces, territories and industry and DFO should evaluate the diagnostic laboratory capacity of provinces, territories and other third party laboratories. Options for the future role of these groups as possible partners in the NAAHP should be jointly investigated by CFIA and DFO. In the process of addressing this recommendation, it was determined that CFIA has the regulatory lead for this activity and therefore is responsible for the establishment of a third party network. According to the document review, CFIA has developed a policy document for a third party laboratory network in which DFO will continue to work with the agency to achieve this initiative. According to interview results, CFIA respondents stated the absence of a third party network of labs is impacting NAAHLS ability to meet the needs of CFIA on the number of diagnostic tests required. For example, one of the biggest factors impacting NAAHP performance is the limited capacity for diagnostic testing.

A lack of resources, both human and financial was the most common cited factor impacting the achievement of NAAHP expected results. AAH program personnel interviewed raised the issue of having to use casual and term employees within NAAHLS as both time consuming and contributing to the lack of expertise within AAH. The time spent hiring and training new staff takes permanent staff away from their actual work for long periods of time. In addition, it is very difficult to recruit and retain experienced personnel when only term or casual positions are available. Interviewees also cited poor communication between CFIA and DFO, a need for stronger vision and leadership as well as limited understanding of roles and responsibilities as significant factors impacting the performance of NAAHP.

Interview results also suggest that lack of understanding of AAH within DFO has a significant impact on the program’s performance and ultimately its ability to achieve its expected results. The lack of understanding is likely a result of the AAH program not having a clearly articulated role within the department. DFO internal stakeholders interviewed identified the lack of understanding of the AAH program as a result of AAH not having a strong voice or communication strategy to educate the department on the work being done and its impacts. For example, NAAHP is a cross-cutting activity in DFO but their role is considered minor and not well understood. Improved messaging could result in better collaboration with other programs on research projects that may overlap or the ability to leverage from each other.

AVAILABILITY OF PERFORMANCE DATA

Evaluation Question: Are appropriate performance data being collected and stored? If so, is this information being used to inform senior management/decision-makers?


The AAH PM Strategy is not measuring the full extent of AAH work being done thus not able to tell an accurate or complete performance story. Some data has been available to senior management through the Performance Measurement Framework. However, the evaluation found no evidence to demonstrate how the data was used to inform decisions.


When NAAHP was created in 2005-06, there were requirements to develop a Results Based Management Framework (RMAF), annual progress reports and an information database as a means of collecting and tracking performance information.

A joint RMAF was developed between DFO and CFIA in 2006.  In 2010, an evaluation of NAAHP found that CFIA and DFO were not collecting data against the RMAF performance indicators and a recommendation outlined in the management action plan was developed to address this finding. The current evaluation was also unable to find any performance data collected against indicators identified in the RMAF. However, the evaluation did find evidence that the AAH program has been reporting on NAAHP activities in the departmental Performance Measurement Framework (PMF) for the period covered by the evaluation. The PMF indicators have slightly changed over the period of the evaluation, examples include the percentage of tests analysed for disease, the percentage of DFO diagnostic labs that are granted ISO accreditation, the number of publically available products on aquatic animal health completed by DFO Science, and the percentage of completed research studies as required by CFIA and DFO. However, other AAH activities, such as FHPR and the provision of scientific advice are not reflected in the PMF.

According to foundational documents, DFO and CFIA agreed to exchange information needed to meet NAAHP obligations through a shared information management system. LIMS was originally designed to be a shared information management system between NAAHLS and CFIA, however, due to technological incompatibilities, it was determined that LIMS would only be used in NAAHLS.  The revised purpose of LIMS is to allow total traceability and tracking of NAAHP samples from point of collection through to reporting of results within NAAHLS. The system is not currently used for the provision of performance measurement data. According to interview results, a quarter of AAH program personnel noted that LIMS is not fully complete or operational and is not consistently used in the same manner by all labs within NAAHLS which ultimately impacts the quality of data.

In June 2012, the program developed an AAH PM Strategy. Prior to the implementation of the PM Strategy in 2013-14, data was collected and reported in the 2011-12 and 2012-13 DPR.  The evaluation was unable to determine if the data collected was used to support decision-making. Interview results suggest that it was too early to determine if information collected from the PM Strategy was being used for decision making.

Most AAH program personnel interviewed stated that the indicators currently used in the PM Strategy do not measure all AAH activities and therefore improvements are needed.   For example, the program currently measures the number of samples received by CFIA for testing, the number of tests actually performed, along with test turnaround time.  However, the program does not account for the number of samples that were rejected and could not be tested. It was also noted that the current indicators focus mainly on NAAHP activities and do not accurately reflect other AAH program activities, such as research activities and scientific advisory products. As a result, the program is not able to tell a complete and accurate performance story.

EFFICIENT USE OF RESOURCES

Evaluation Question: Is the AAH program undertaking activities and delivering products in the most efficient manner?


It is difficult to assess the overall efficiency of the AAH program as there was limited evidence of AAH activities other than NAAHP. NAAHP is undertaking activities and delivering products in an efficient manner. NAAHP was modelled after a similar federal government program demonstrated to be cost efficient, and it is similar in design and delivery to programs in other countries.


AAH Resources

Between 2008-09 and 2012-13, financial data and FTE information for the overall AAH program was available inclusive of NAAHP. No detailed financial data was available to reflect the other AAH program activities, such as FHPR and scientific advice, support and dissemination activities.

Table 7.0: NAAHP and Non-NAAHP Expenditures and FTEs
Year NAAHP Non-NAAHP AAH
Expenditures FTE Expenditures FTE Total Expenditures Total FTE
2008-09 5,242,310.28 36.55 1,667,797.30 16.78 6,910,107.58 53.33
2009-10 4,265,858.47 36.11 2,963,778.29 19.89 7,229,636.76 56.00
2010-11 4,054,001.39 35.57 2,125,526.26 20.18 6,179,527.66 55.75
2011-12 4,190,791.74 33.37 2,108,066.85 19.72 6,298,858.59 53.09
2012-13 3,978,430.68 32.76 2,036,910.18 19.99 6,015,340.86 52.74

Source: DFO Finance Sector

On average, between 2008-09 and 2012-13, 66% of AAH program funding was directed to NAAHP and 64% of AAH FTEs were allocated to NAAHP activities. The remaining 34% of AAH funding is assumed to have been directed to the other AAH program activities however the evaluation was unable to link the funding to specific activities such as FHPR and scientific advice, support and dissemination.

Table 8.0: NAAHP and Non-NAAHP Expenditures from 2008-09 to 2012-13 criteria completed in support of ISO 17025 accreditation

NAAHP is designed after the CFIA’s Terrestrial Animal Health Program and the OIE’s Aquatic Animal Health Code activities. CFIA has a long history of managing health of terrestrial animals in Canada. The Terrestrial Animal Health Program has demonstrated that it is more cost efficient to prevent the introduction of an exotic disease than to react to it once it has become well established. In addition, studies of aquatic animal health programs in other countries suggest that public spending on animal health provides a return on investment. These findings suggest that NAAHP is modeled after programs that have proven to be cost efficient.

Comparative Analysis

Public documents from Australia and the United States were analysed for comparison with NAAHP to draw conclusions regarding the similarities and differences in their respective national aquatic animal health program design and delivery approaches. These countries were chosen for analysis as they were previously compared in the 2010 NAAHP evaluation and are also members of the OIE.

The United States developed the National Aquatic Animal Health Plan in 2009 to form a coordinated government effort to ensure aquatic animal health. The evaluation was unable to find evidence that the plan has been officially implemented. Officials from federal, state, local, and tribal governments, in partnership with aquaculture and other interested stakeholders collaborated on the plan. It is the summation of ideas on how the federal government, in collaboration with stakeholders, should develop policies, programs and potential regulations to address aquatic animal diseases in order to benefit aquaculture and aquatic animal resources in the United States. Responsibilities regarding aquatic animal health are shared across three federal departments and involve partnership with other levels of government.

The Australian Aquatic Animal Health Program is a federal program that was implemented in 1998 which leads and coordinates the national management of aquatic animal health in Australia. The program supports national committees responsible for aquatic animal health and aquatic animal disease emergency preparedness, manages the high-level national strategic plans for aquatic animal health, manages disease surveillance and reporting, and participates in international fora and initiatives surrounding aquatic animal health.

Table 9.0: National Aquatic Animal Health Programs in United States, Australia and Canada
Table 9.0
  United States* Australia Canada
Guiding legislation X(planned) X X
Federal responsibility X(planned) X X
Under one department   X  
Use of third party labs X X  
ISO accredited X X (NVSL)  
Responsible for export X   X
Responsible for import X   X
Responsible for domestic movement X X X (FHPR)
Members of the OIE X X X
Program implementation 2009 (Plan Released) 1998 2005
Fees charged/cost recovery X    

*The program elements reflected here for the US are not yet implemented nor is the legislation implemented; they are still in the planning stages. Program elements for Canada and Australia reflect those elements which have been implemented.

The Australian program and the US plan are similar to NAAHP with regard to design and delivery, with the exception of the use of a network of third party labs and ISO accreditation. NAAHLS plan to have all labs ISO accredited by 2014-15 and have also begun discussing the possibility of implementing a network of third party labs. Ultimately Canada’s NAAHP design and delivery largely reflects that of other countries and the transformations underway will further align Canada with other countries.

Program Improvements

The following program improvements were most commonly suggested by interviewees to increase the likelihood of success and improve the efficiency of the AAH program:

  • With regard to AAH within DFO, improve messaging within the department to bring greater awareness to the program.
  • With regard to NAAHP, develop and clearly communicate a joint plan to address issues like governance structure, roles and responsibilities, workload and communication; implement a third party network of labs to assist with capacity issues; and have NAAHP reside all within one department rather than co-delivered by two separate departments.

When asked if resources could be reduced or reallocated to make the AAH program more effective or efficient, the majority of interviewees were of the opinion that reductions were not possible as there is nothing left to reduce. Several suggestions were made regarding redirecting resources. The most common suggestions were to reassign resources to NAAHLS / NAAHP activities once regulatory activities regulatory for interprovincial carrying of live salmonid fish and their eggs will be transferred from DFO to CFIA and to redirect funding to NAAHLS research to better align with NAAHP priorities and ensure it meets the needs of both CFIA and DFO. Almost all facilities interviewed were of the opinion that the FHPR program was working well.

4.4 ECONOMY

Evaluation Question: Is the AAH program achieving its intended outcomes in the most economical manner?


The evaluation found some duplication and overlap within the AAH program and between AAH and other programs. Elimination or a reduction in the overlap and duplication could provide a more economical use of the program resources.


To determine if the AAH program is achieving its intended outcomes in the most economical manner, the evaluation examined whether there is overlap or duplication between AAH and other programs, as well as whether AAH resources are adequate.

The document review found that there is duplication and overlap between NAAHLS and laboratories external to NAAHLS. The laboratories outside of NAAHLS are not ISO accredited and follow different testing procedures, yet they can provide test results of fish diseases to stakeholders. Although NAAHLS is not fully ISO accredited, they have a solid quality management program in place and follow the same strict testing policies and procedures as if they were ISO accredited. Consequently, it is possible that inconsistent results may materialize between the NAAHLS and outside laboratories due to the differences in methodologies, procedures and processes. The results from laboratories outside of NAAHLS may not hold the same credibility as the NAAHLS laboratories, and could contribute to confusion with industry stakeholders as to the accuracy of the results. 

According to over one-third of AAH program personnel interviewed, there is potential overlap between AAH and other programs within DFO. The most cited overlap stated by interviewees was between AAH and the Genome BC project. For example, an interviewee stated, “if the results of the BC Genome project are different from ours then there is an issue. Their methods and validation processes are different than ours. The issue of overlap with AAH could be a problem when the results are presented or published to the public.” CFIA respondents also mentioned the Genome BC project stating that conflicting results from different methodologies could be problematic. Also, some projects outside of AAH which test for listed pathogens (Program for Aquaculture Regulatory Research, Aquaculture Collaborative Research and Development Program, Genome BC) are approved for funding without prior discussion with AAH. As a result, some monitoring activities funded by these initiatives may include testing which is already covered by NAAHP surveillance and testing.

The evaluation examined if components of AAH including FHPR and NAAHP could or should be conducted by other organizations. An audit of the NAAHP undertaken by the European Commission Health and Consumers Directorate General in 2012 found that the NAAHLS under DFO “have implemented an effective quality management system with comprehensive standard operating procedure for a variety of diagnostic tests that are carried out in accordance with OIE standards.” Although quality management may not be an issue for NAAHLS, according to more than half of the AAH program personnel interviewed, capacity and resource issues are of concern. Interviewees stated that establishing a network of third party labs for diagnostic work might help with DFO capacity issues and resources in the long term but would be more costly and resource intensive in the short term. However, a cost benefit analysis would need to be conducted to determine if this would be the case. More than half of CFIA respondents interviewed supported the notion that export testing could be conducted by third party labs, allowing for more testing.

5. CONCLUSIONS AND RECOMMENDATIONS


Ensuring that Canada’s wild and cultured aquatic resources are protected from the introduction or spread of disease is the mandate of the AAH program. As discussed in the relevance section of this report, there is a need for the program, specifically the regulatory components, NAAHP and FHPR, to ensure the long-term viability of the international aquatic trade industry and to ensure that the Canadian seafood industry remains competitive. The AAH program is also well aligned with the federal priorities of economic development and international trade and contributes to the departmental outcome of “economically prosperous maritime sectors and fisheries”.

The extent to which facilities are aware of FHPR regulatory obligations is not evident as there is no official tracking of awareness. Awareness is assumed by facilities’ participation in the program. The evaluation was also unable to assess the extent to which facilities comply with regulatory obligations as there was no evidence of a mechanism to track compliance and it is therefore difficult to conclude whether there are compliance issues.

The evaluation found that CFIA has access to validated diagnostic test results for NAAHP.  However, access was viewed as limited by CFIA as a result of LIMS not being directly accessible as originally intended. In addition, the timeliness of test results did not always meet CFIA decision making needs.  As well, CFIA has forecasted more tests than they officially requested to be completed by NAAHLS, impacting the ability of DFO to forecast resource requirements for NAAHLS.

The evaluation also concluded that internal and external stakeholders have access to scientific advisory products. However, the full extent of the products that are produced and disseminated are not captured by the existing PM Strategy and the degree to which the advisory products are used to make an informed decision is not tracked and therefore unknown. The AAH PM Strategy is not measuring the full extent of AAH work being done thus not able to tell an accurate or complete performance story. Some data has been available to senior management through the PMF. However, the evaluation found no evidence to demonstrate how the data was used to inform decisions.

It was difficult to assess the overall efficiency and economy of the AAH program as there was limited evidence of AAH activities other than NAAHP. NAAHP is undertaking activities and delivering products in an efficient manner. NAAHP was modelled after a similar federal government program demonstrated to be cost efficient, and it is similar in design and delivery to programs and plans in other countries. According to the evaluation findings, it is expected that AAH will realize greater efficiencies through further clarity on AAH program roles and responsibilities within the department. In addition, renewal of NAAHP governance structures, with clearly defined roles and responsibilities will bring a greater understanding between DFO and CFIA. With regards to the intended transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs, the evaluation did not find any evidence of a formal transition plan to address workload and resourcing issues. Finally, greater efficiencies and economy could be further achieved by the development of third party labs, reducing the capacity and work requirements of DFO's NAAHLS. Interviewees from both CFIA and DFO are aware of the need for third party labs but have identified that realization of the network in the short term may be intensive in terms of resources and capacity and that a cost benefit analysis is a necessity.

Recommendations

Recommendation 1:
It is recommended that the AAH program clearly define all activities within the program and communicate them within the department and with stakeholders.

Recommendation 2:
It is recommended that the AAH program revise its Performance Measurement Strategy to include all AAH program activities and collect the necessary data.

Recommendation 3:
It is recommended that the AAH program develop a transition plan, in consultation with CFIA, for the planned transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs. The transition plan should include clearly articulated responsibilities, accountabilities and timelines.

Recommendation 4:
It is recommended that the AAH program consult CFIA to reinstate NAAHP governance structures, including clear roles and responsibilities for both departments.

Recommendation 5:
It is recommended that the AAH program work with CFIA to finalize and implement the draft third party lab policy. The policy should include clearly articulated responsibilities, accountabilities and timelines.

Annex A: Evaluation Matrix


ANNEX A: AQUATIC ANIMAL HEALTH PROGRAM EVALUATION MATRIX

annex a
Issue/Question Indicators Line of Evidence
Interviews Survey Document Review
DFO CFIA
1.0 RELEVANCE  
1.1 Is there a continued need for the AAH program? Views on the importance/need of the AAH program x x x  
Evidence to support continued need       x
Presence / absence of other programs that compliment or duplicate the objectives of the AAH program       x
1.2 To what extent is the AAH program aligned to departmental and federal government priorities? Degree of alignment of AAH program with current federal government objectives and priorities       x
Degree of alignment of AAH program with current departmental strategic outcomes       x
1.3 Is the AAH program consistent with federal roles and responsibilities? AAH program mandate aligned with federal government jurisdiction       x
Views on the appropriateness of federal roles and responsibilities x      
2.0 EFFECTIVENESS  
2.1 To what extent has the AAH program achieved its immediate outcomes?
2.1.1 To what extent are facilities aware of regulatory obligations? Number and  percentage of facilities aware of regulatory obligations       x
Views and/or documents on facilities aware of regulatory obligations x   x x
2.1.2 To what extent do decision-makers have access to validated diagnostic test results? Number and percentage of planned tests that were performed        x
Number and percentage of research and development projects that support NAAHP objectives       x
Views/Evidence on having access to validated diagnostic test results x x   X
2.1.3 To what extent do stakeholders have access to scientific products? Number and percentage of requested scientific advisory products completed       x
Views on access to scientific advisory products x      
2.2 To what extent has the AAH program achieved its intermediate outcomes?
2.2.1 To what extent do facilities comply with regulations? Number and percentage of facilities in compliance with regulations       x
Views on extent to which facilities comply with regulations x   x  
2.2.2 To what extent do stakeholders make informed decisions based on scientific products? Number and percentage of scientific advisory products completed that inform decision-making       x
Views on the extent to which scientific advisory products are used in decision-making x      
2.3 To what extent has the AAH program achieved its long-term outcomes?
2.3.1 To what extent are Canada’s wild and cultured aquatic resources protected from the introduction or spread of diseases of concern? Number of CFIA reports to the OIE (on changes to Canada’s aquatic animal health disease status) that used DFO diagnostic test results and scientific knowledge       x
2.4 Have there been any unintended (positive or negative) outcomes? Presence/absence of unintended outcomes       x
Views on whether unintended outcomes occurred x x   x
3.0 EFFICIENCY  
3.1 Is the AAH program design and delivery appropriate for achieving program results? Clearly defined and understood governance structure, including program processes, roles and responsibilities x x   x
Number and percentage of criteria completed in support of Standards Council of Canada ISO 17025 accreditation for each of the three DFO NAAHLS laboratories       x
Program resources/capacity are adequate for achievement of expected results       x
Views on the appropriateness of program activities, processes and governance structures x x    
3.2 Are appropriate performance data being collected and stored? If so, is this information being used to inform senior management / decision makers? Existence of effective performance measurement plan; evidence of data collection and a performance data system       X
Extent to which performance data informs AAH and departmental wide decision making processes       X
Views on the extent to which the performance measurement strategy can be improved x      
3.3 Is the AAH program undertaking activities and delivering products in the most efficient manner?
  • How could the efficiency of the AAH program’s activities be improved?
  • Are there alternative, more efficient, ways of delivering the program?
Comparison of program activities and products delivered by other similar programs       x
Administrative costs of AAH program       x
Evidence/views on how the efficiency of program activities could be improved x x x x
4.0 ECONOMY  
4.1 Is the AAH program achieving its intended outcomes in the most economical manner? Evidence /views on whether there is overlap / duplication with other programs x x x x
Views/evidence on the adequacy of program resources x   x x

Annex B: Management Action Plan


ANNEX B: MANAGEMENT ACTION PLAN

Annex B. Management Action Plan
RECOMMENDATION

Rationale 1: The evaluation concluded that the overall AAH program roles and responsibilities and activities are not well understood within the department and with stakeholders. AAH could realize greater efficiencies through further clarity on AAH program roles and responsibilities within the department and with stakeholders.

Recommendation 1: It is recommended that the AAH program clearly define activities within the program and communicate them within the department and with stakeholders.
STRATEGY
Under the direction of the Director, Aquaculture, Biotechnology and Aquatic Animal Health the activities associated with the AAH will be reviewed and clearly defined within the department and with all stakeholders including the CFIA.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Review the current AAH activities as defined in 1.5 of the 2014-15 PAA November 2014    
Develop options for the AAH program description and activities December 2014    
Discuss with internal and external stakeholders as appropriate January 2014    
Revise AAH program description and activities for reporting purposes as appropriate March 2015    
Communicate and implement changes internally and externally May 2015    
RECOMMENDATION

Rationale 2: The evaluation found that the AAH Performance Measurement (PM) Strategy is not measuring the full extent of AAH work being done thus not able to tell an accurate or complete performance story.

Recommendation 2: It is recommended that the AAH program revise its Performance Measurement Strategy to include all AAH program activities and collect necessary data.
STRATEGY
The AAH program will revise its Performance Measurement Strategy to reflect the decisions made on the relevant program activities as stated in recommendation 1.  The revised PM Strategy will include appropriate indicators and data collection strategies for all related activities.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Review and revise AAH PM Strategy including logic model and matrix. October 2014    
Develop data collection tools November 2014    
Obtain Senior Management - Science approval of relevant changes December 2014    
Communicate changes to AAH program staff January 2014    
Implement revised data collection strategies as appropriate April 2015    
RECOMMENDATION

Rationale 3: The evaluation was unable to find evidence of a formal transition plan or timeline to address workload and resourcing issues related to the intended transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs. Until the transfer of responsibilities is completed, DFO will remain responsible for delivering these activities. This places DFO in a position of uncertainty as it impacts on future planning.

Recommendation 3: It is recommended that the AAH program develop a transition plan, in consultation with CFIA, for the planned transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs. The transition plan should include clearly articulated responsibilities, accountabilities and timelines.

STRATEGY
The AAH program is consultation with CFIA, will develop a transition plan for the intended transfer of responsibilities from DFO to CFIA regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs. The transition plan will minimize the impact on regulated parties.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Confirm with CFIA the date for the implementation of related NAAHP components (i.e. the Domestic Movement Control Program) June 2014    
Develop transition plan in consultation with CFIA and Strategic Policy and Ecosystems and Fisheries management to address the transfer of responsibilities from DFO to CFIA regarding regulatory activities for the interprovincial carrying of live salmonids and their eggs. August 2014    
Inform regulated parties and other stakeholders of program changes September 2014    
Initiate transfer of responsibilities from DFO to CFIA regarding regulatory activities for the interprovincial carrying of live salmonids and their eggs. January 2015    
RECOMMENDATION

Rationale 4: The evaluation found that NAAHP roles and responsibilities were defined at the outset of the program with governance structures initially in place. Evidence suggests that the roles and responsibilities are not well understood and have not been revised to reflect the current working environment. In addition, NAAHP governance structures initially established have not been maintained since 2012. Renewal of NAAHP governance structures, with clearly defined roles and responsibilities will bring a greater understanding between DFO and CFIA.

Recommendation 4: It is recommended that the AAH program consult CFIA to reinstate NAAHP governance structures, including clear roles and responsibilities for both departments.

STRATEGY
Under the direction of the Director General, Strategic and Regulatory Science and in consultation with the CFIA the current governance structures will be reviewed to clarify the roles and responsibilities of both departments for the NAAHP.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
DG SRS to discuss with CFIA counterparts interdepartmental governance structure October 2014    
Governance structure proposed February 2015    
RECOMMENDATION

Rationale 5: A need to leverage additional work from provinces and industry for the purpose of NAAHP surveillance and testing was identified at the outset of the program. The evaluation was unable to find evidence that these leveraging networks had begun. The evaluation concluded that greater efficiencies and economy could be further achieved by the development of third party labs. As the regulatory lead, CFIA has developed a draft policy document for the development of a third party network of labs.

Recommendation 5: It is recommended that the AAH program work with CFIA to finalize and implement the draft third party lab policy.  The policy should include clearly articulated responsibilities, accountabilities and timelines.
STRATEGY
The AAH program will work with the CFIA to finalize and implement the draft third party lab policy.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Based on revised policy, work with CFIA to identify DFO program needs April 2014