Skip to content | Skip to institutional links

Common menu bar links

Institutional links

Reports
Resources
Proactive Disclosure

Proactive Disclosure

EVALUATION OF THE AQUACULTURE COLLABORATIVE RESEARCH AND DEVELOPMENT PROGRAM AND THE AQUACULTURE ENVIRONMENTAL AND BIOLOGICAL SCIENCE PROGRAM

FINAL REPORT
JUNE 2012

EVALUATION DIRECTORATE
DIRECTION GÉNÉRALE DE L’ÉVALUATION

ACRONYMS

ACRDP

Aquaculture Collaborative Research and Development Program

AEBSP

Aquaculture Environmental and Biological Science Program

APF

Aquaculture Policy Framework

DFO

Department of Fisheries and Oceans

FADS

Federal Aquaculture Development Strategy

FTE

Full Time Equivalent

NHQ

National Headquarters

NSC

National Steering Committee

NSERC

Natural Sciences and Engineering Research Council of Canada

PAA

Program Activity Architecture

PARR

Program for Aquaculture Regulatory Research

PSA

Program for Sustainable Aquaculture

R&D

Research and Development

RMC

Regional Management Committee

SAP

Sustainable Aquaculture Program

TRC

Technical Review Committee

This evaluation report presents the results of the evaluation of the Aquaculture Collaborative Research and Development Program (Collaborative Program) and the Aquaculture Environmental and Biological Science Program (Science Program). In accordance with the Treasury Board Policy on Evaluation (2009), the evaluation assessed the core issues of relevance and performance, including effectiveness, efficiency and economy.

This evaluation covered the period from 2006/07 to 2010/11 and was undertaken between November 2011 and April 2012 by Fisheries and Oceans Canada’s (DFO) Evaluation Directorate.

Program Profile

The Collaborative Program was launched in 2001 to undertake research and development projects that are proposed and jointly funded with private sector partners. The Collaborative Program had an ongoing budget of $4.275 million annually until 2011-12 when it was reduced to $3.675 million. In 2012-13, the Collaborative Program will be funded at $2 million per year. Three broad areas were identified for research collaboration:

Best Performance in Fish Production;
Environmental Performance, and
Optimal Fish Health.

Also created in 2001 under the Program for Sustainable Aquaculture, the Aquaculture Environmental and Biological Science Program is a partner program to the Collaborative Program. It provides the administrative capacity and scientists who produce the scientific research reports as described in the collaborative agreements. The Science Program had an initial budget of $2.5 million annually including 16 Full Time Equivalent positions. The funding level for the Science Program has declined over the years and in 2011-12 was approximately $1.9 million.

The Collaborative Program teams industry with DFO researchers to undertake research activities that lie within the mandate of DFO but are based on the priorities and needs of the aquaculture industry.

In 2011, the Collaborative Program was modified and will no longer fund projects that fall under the areas of Best Performance in Fish Production or freshwater aquaculture. Funding for the program was also reduced by $2 million. The program delivery model has also changed from a predominantly regional oriented model to one that is led by National Headquarters with involvement by the regions. The Science Program was also impacted by budget reductions in 2011 by a reduction of $1 million in salary dollars resulting in the loss of 11 Full Time Equivalent positions. This reduction will be implemented during 2012-13.

Evaluation Methodology

The evaluation used a non-experimental design incorporating a logic model, multiple lines of evidence, and qualitative and quantitative data. The methodologies used for this evaluation included: document review; literature review; interviews, on-line survey, case studies and regional site visits. Within the methodologies employed, there were a few limitations and challenges. None of them was prejudicial to the validity of the evaluation results as strategies were put in place to mitigate these limitations.

Evaluation Findings And Recommendations

Relevance

The Collaborative Program is aligned with DFO and federal government responsibilities.
The federal government, motivated by aquaculture’s promise of economic benefits, has supported aquaculture activities with DFO as the lead department for over 25 years. To assist industry with its research and development needs, the Collaborative Program was established in 2001 as a collaborative industry led program to carry out joint research and development projects to help improve the competitiveness of the aquaculture industry. However, the economic developmental role of DFO and its focus on aquaculture is not clearly provided for in the Fisheries Act which assigns a regulatory role to the Department for the management of fisheries and makes no specific mention of aquaculture.¹ The Department is aware of this anomaly and is working to resolve it. This issue will be examined further in a subsequent evaluation of the Sustainable Aquaculture Program.

The program is aligned with Federal and DFO priorities. The Minister has however recently emphasized DFO’s mandate with respect to aquaculture. Accordingly, the Collaborative Program no longer supports research on fish production, as it is not aligned with the Department’s core mandate.

There is a continuing need for the Collaborative Program. Research continues to be needed due to the evolving nature of the aquaculture industry which is dominated by smaller players with little capacity to conduct research on their own. Collaborative Program projects contribute to DFO’s role as they have provided research results that support monitoring of environmental impacts. These benefits, however, are not always documented. Continuation of the Collaborative Program to meet requirements for optimal fish health and environmental performance would result in benefits to industry, as well as DFO and its regulatory partners.

¹ On February 9th, 2009, the British Columbia Supreme Court held that the activity of finfish aquaculture is a fishery and falls under the exclusive jurisdiction of the federal government under the Constitution Act, 1867.

Effectiveness

The evaluation found that the program has made some progress in achieving its expected immediate and intermediate outcomes.

The Collaborative Program is achieving two of its immediate outcomes:

Projects are aligned with industry needs and priorities; and
Aquaculture stakeholders are engaged.

For the third immediate outcome “industry and management are aware of scientific evidence”, the results from research projects are not adequately being shared with all industry stakeholders and partners. While the project collaborators benefit from direct contact with the DFO researcher, the inconsistent production of final reports and updating the DFO website with complete project information for all completed projects, is hampering the transfer and sharing of scientific information emanating from the program.

Recommendation # 1:

It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science, ensure that project results are effectively communicated to collaborators in particular and to the industry in general. Specific improvements should include:

Final reports, that are easily understood and in plain language, are produced for all projects on a timely basis; and
The DFO web site as it pertains to the Aquaculture Collaborative Research and Development Program is maintained on a regular basis.

Regarding the intermediate outcome of utilization of the results of the Collaborative Program research, the evaluation concludes that results from research projects are being utilized to some extent, however, there is no systematic follow-up of completed projects to determine the degree to which they are being used.

The evaluators were able to identify evidence that some of the research results were used or have the potential to contribute to the aquaculture industry. For example, document reviews and the case studies carried out for the evaluation provided examples of successful projects.

Recommendation # 2:

It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science, ensure that the Aquaculture Collaborative Research and Development Program systematically monitor the utilization of research projects findings and their impacts on aquaculture operations and regulatory agencies.

The Science Program’s role within this initiative was to conduct research projects and communicate research results. While it was the intention that the Science Program funding provide the research capacity to carry out the Collaborative Program research, no assessment could be made with respect to attributing the achievement of this outcome to the Science Program. Evaluators were unable to conclude whether the DFO researcher for a Collaborative Program research project was indeed funded by the Science Program or through some other science funding source.

Efficiency & Economy

In October 2011 the Collaborative Program’s decentralized program delivery model changed over to an NHQ led model with a national focus. Because of the limited time that the new program model has been in place, the evaluation focused on areas that could be improved regardless of the delivery model in place.

The three Collaborative Program research and development objectives were very broad. As a result, all proposals submitted by industry were considered to be addressing a national priority. To confirm the on-going relevance of industry and DFO priorities in the context of the evolving nature of the aquaculture industry, a systematic process for the setting and review of priorities is needed.

Recommendation # 3:

It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science review industry and DFO priorities periodically so as to reflect changing needs of the aquaculture industry and the Department.

The evaluation was unable to determine the overall cost of the Collaborative Program as the funding provided to the program through the Science Program are not identified. In short, the amount of funds contributed by the Science Program to deliver the Collaborative Program could not be determined.

The Science Program was designed to provide research and scientific advice, support strategies to the aquaculture industry and administer the Collaborative Program. However, given the absence of an adequate process for tracking AEBSP funding, the evaluators were unable to identify the extent to which it was contributing to the achievement of outcomes for the ACRDP or other programs which it enables.

Recommendation # 4

It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science establish a process for tracking the funding of the Aquaculture Environmental and Biological Science Program so that its contribution to the achievement of the expected outcomes of the programs which it enables can be identified.

The Science Program provides the researchers and salaries that are used to carry out the Collaborative Program activities and outputs to yield the desired outcomes. While the Science Program scientists support other programs to some extent, their key activities relate to delivering the Collaborative Program. In essence the Science Program is the salary component of the Collaborative Program. Consolidating the two programs would provide a more complete picture of the delivery of the Collaborative Program.

Recommendation # 5

It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science determine whether the Aquaculture Environmental and Biological Science Program and the Aquaculture Collaborative Research and Development Program should be consolidated in order to more fully include the basic components required for a program as defined by Treasury Board.

1. Introduction

1.1 Context of the Evaluation

This report presents the results of the evaluation of the Aquaculture Collaborative Research and Development Program (ACRDP) and the Aquaculture Environmental and Biological Science Program (AEBSP). The Programs are identified as sub-activities under the Sustainable Aquaculture Program (SAP) in DFO’s 2012-13 Program Activity Architecture (PAA). The evaluation is in accordance with the Treasury Board of Canada’s Policy on Evaluation, which requires all direct program spending be evaluated every five years. The evaluation had been included as a project in the 2012-13 Departmental Evaluation Plan; however, at the request of the ACRDP National Steering Committee (NSC) and the Aquaculture Science Branch, the evaluation commenced in 2011-12 and will be completed in 2012-13. The evaluation will also coincide with and inform the renewal of the Sustainable Aquaculture Program. Recommendations stemming from the main findings are formulated to allow for improvements to the program where necessary and to inform future decision-making.

1.2 Structure of the Report

Of Special Interest to the Reader

Throughout the planning and conduct phases of this evaluation, it became increasingly apparent to evaluators that there were several issues with respect to the design and delivery of the AEBSP. The reader will observe that the AEBSP is a key associated program of the ACRDP. It was created to administer the ACRDP and strengthen regional science capacity for aquaculture. The AEBSP also provided some of the scientists needed to produce the scientific research as described in the ACRDP collaborative agreements. However, it became evident that the AEBSP was not evaluable due to its lack of conformity to the Treasury Board definition of a program. Due to its intertwined nature with the ACRDP, it was scoped into the evaluation and to the extent possible, assessed for its relevance and performance.

The reader may be interested in the following sections in this report as they relate specifically to discussions surrounding the AEBSP:

Program Profile	pages 2, 3, 9
Effectiveness	pages 21
Efficiency	pages 25, 26
Conclusions	page 28

2. Program Profile

2.1 Mandate/Background/Objectives

The ACRDP is a DFO initiative depicted under the SAP in DFO’s Program Activity Architecture. The ACRDP was initially launched in 2001 to undertake research and development projects that are proposed and jointly funded with private sector partners. The ACRDP had an ongoing annual budget of $4.275 million until 2011-12 when it was reduced to $3.675 million. Three broad areas were identified for ACRDP research collaboration: Best Performance in Fish Production; Environmental Performance and Optimal Fish Health.

In 2011, the ACRDP was modified and will no longer fund projects that fall under the areas of Best Performance in Fish Production or freshwater aquaculture. Starting in 2012-13, ongoing funding for the program will be $2 million, in operations and maintenance funding. The program delivery model has also changed from a predominantly regional oriented model to one that is led by NHQ.

Also created in 2001 under the Program for Sustainable Aquaculture, the Aquaculture Environmental and Biological Science Program (AEBSP) was funded to support environmental and biological science towards the promotion of the environmental sustainability of the aquaculture industry. The resources initially approved for the AEBSP was in the amount of $2.5 million annually including 16 Full-time Equivalent positions. The funding level for the AEBSP has declined over the years and in 2011-12 was approximately $1.9 million. This Program was designed to enhance capacity to:

establish an integrated research and scientific advice on the ecosystem effects of aquaculture;
support the development of effective area wide and integrated management strategies that promote sustainability of the aquaculture industry;
strengthen regional biological science capacity; and
administer the ACRDP.

Since its creation in 2001, the AEBSP has evolved into the mechanism used to provide research support not only to ACRDP but to other science related programs for aquaculture. The funding includes the initial amount provided through the Program for Sustainable Aquaculture, as well ongoing funding (A-Base) that had been allocated to the Ecosystem and Oceans Science Sector. The funding currently allocated to this program is approximately $8 million annually.

The AEBSP provides scientists to the ACRDP who produce the scientific research reports as described in ACRDP collaborative agreements. They also collaborate with industry applicants in the preparation of proposals that are submitted for consideration to the ACRDP. The AEBSP was also affected by budget reductions in 2011 and was reduced by $1 million in salaries resulting in the loss of 11 Full-time Equivalent positions. This reduction will be implemented during the 2012-13 fiscal year.

2.2 Program Activities

Collaboration and Leadership

The ACRDP teams industry with DFO researchers to undertake research activities that are intended to lie within the mandate of DFO but are based on the priorities and needs of the aquaculture industry.

Prior to the 2012-13 fiscal year, project proposals were accepted twice annually, on March 1st and December 1st. There may have also been additional calls for proposal submissions throughout the year based on funding availability.

DFO officials review proposals to ensure completeness, accuracy and eligibility under ACRDP criteria. Prior to October 2011, all eligible projects underwent a two part peer-review: first, a technical/scientific review by internal DFO science staff and/or external scientists, followed by a review by the Regional Management Committees (RMC). The Committee made recommendations to the Regional Directors of Science, who had the authority to approve projects.

Subsequent to the October 2011 program design changes, project proposals will be received, from all regions, at NHQ and reviewed for funding eligibility by DFO officials. The proposals will still undergo a two part peer review process as described above except that the RMC review will be replaced by review through the ACRDP Technical Review Committee (TRC). The TRC will recommend for projects for approval to the Director General of the Strategic and Regulatory Science Directorate in the Ecosystems and Oceans Science Sector.

The primary outputs produced by these activities are collaborative agreements which identify scope and methodologies for the projects to be carried out.

Knowledge Generation

Lead researchers undertake research activities as prescribed within the collaborative agreements. Research activities may include, but are not limited to, laboratory work at DFO facilities and fieldwork at industry sites. In most cases, the lead researcher is a DFO employee whose services are provided as a DFO in-kind contribution; or, where a particular expertise is not available within DFO, an individual on a service contract with DFO; or, under special circumstances, an individual recommended by the industry applicant whose salary is covered by the collaborative agreement funding. The lead researcher may be assisted with on-site fieldwork and laboratory related duties by graduate students and/or DFO technicians/biologists or term or casual employees. Industry applicants participate in research activities by providing access to their facilities, construction and equipment installation, monitoring and in some cases participate in field data collection. Lead researchers and to some extent, university graduate students produce and publish peer reviewed journal articles. University graduate students also produce theses on aquaculture for their area of study, which in some cases, may be published in peer-reviewed journals.

The main outputs produced by the research activities include research reports, project progress reports, final project reports, intellectual property, research papers; innovative concepts, technology, project fact sheets, technical reports, peer-reviewed publications and trained personnel.

Knowledge Transfer

The ACRDP disseminates knowledge generated by research to the aquaculture industry through several communication channels. Research findings are presented at aquaculture industry meetings and/or events or at relevant conferences attended by aquaculture industry representatives. Presentations are made by DFO staff either via targeted presentations/workshops; expert panel; keynote speakers and/or; kiosks. Various publications are produced, such as the Canadian Aquaculture Research and Development Review, published bi-annually, which includes summaries of aquaculture research activities plus special full length features on completed projects across the country; and fact sheets which highlight promising and best industry practices. Researchers and graduate students generate academic papers and publish these in peer reviewed journals; finally information about the ACRDP, fact sheets, publications etc., is posted on the DFO website.

2.3 Program Expected Results/Performance Measurement

The key objective of the ACRDP is “to improve the competitiveness of the Canadian aquaculture industry through collaborative research and development activities between the Canadian aquaculture industry, DFO and other key partners.” The objective of the AEBSP is to support environmental and biological science towards the promotion of the environmental sustainability of the aquaculture industry.

ACRDP is an applied research program that teams industry with DFO researchers to focus on issues identified by industry as constraints to development.

There are three immediate expected outcomes:

R&D projects are aligned with industry needs and priorities
Aquaculture stakeholders are engaged
Industry and management awareness of information.

There is one intermediate expected outcome:

Fish production, fish health and environmental performance technologies are utilized by industry and management.

The program contributes to one long-term outcome:

A Canadian aquaculture sector that is environmentally and socially sustainable, demonstrating improved industry competitiveness, increased production capacity and having an increased market share.

This evaluation assessed the extent to which the program was achieving its immediate and intermediate outcomes.

2.4 Governance

The ACRDP is administered by the Aquaculture Science Branch of the Strategic and Regulatory Science Directorate in the Ecosystems and Oceans Science Sector of DFO. Program oversight is provided through the NSC (chaired by the Director General of the Strategic and Regulatory Science Directorate). It includes representatives from DFO’s Ecosystems and Oceans Science, Ecosystems and Fisheries Management, and Program Policy Sectors, as well as industry, and provinces.

The explicit mandate of the ACRDP National Steering Committee (NSC) is to “evaluate the program, report on its performance and make recommendations on its management for continuous improvement.”

In each region, a RMC had been established to advise on regional research priorities and to evaluate and recommend proposals for funding. The RMC membership included the Regional ACRDP Chair (or Co-Chairs), the Regional Coordinator and representatives from industry, provinces, academia, DFO regional aquaculture management representatives and other government departments (e.g., National Research Council-Industrial Research Assistance Program). Proposals recommended for funding by the RMC were presented to the Regional Directors of Science for final approval.

With the program changes announced on October 2011, the RMCs will no longer exist. The NSC will continue in its advisory role and a new Technical Review Committee has been established to review and recommend projects for approval. The projects will be reviewed and recommended for approval to the Director General, Strategic and Regulatory Directorate in the Ecosystems and Oceans Science Sector.

2.5 Program Logic Model

The logic model for the ACRDP is presented on the following page.

2.6 Program Partners/Clients/Stakeholders

As the ACRDP is a collaborative research program, all research projects must have an industry partner. Eligible industry applicants are marine aquaculture producers operating within Canada who are directly involved in producing marine aquatic species for pre-commercial or commercial purposes². Producers undertaking commercial or developmental production activities on existing or new aquaculture species or aquaculture companies or associations involved with sea ranching mariculture operations are eligible to apply. Industry producer associations or consortia of producers are also eligible to apply. Other aquaculture sector stakeholders are eligible to participate as a partner with an industry producer.

Clients (program beneficiaries):

Aquaculture producers operating within Canada who are directly involved in producing an aquatic species for pre-commercial or commercial purposes and/or undertaking commercial or developmental production activities on new or existing aquaculture species.
Aquaculture companies including support companies for aquaculture operations, e.g., feed companies, net building or cleaning.
DFO internal clients in the Habitat Management and Aquaculture Management Directorates are also beneficiaries of ACRDP projects

Stakeholders (interested parties):

Other aquaculture sector stakeholders participating as a partner with an industry producer (e.g., universities, research councils, conservation groups, and other federal departments and agencies).

Collaborators (co-deliverers):

Any aquaculture producer, company, industry association or sector stakeholder identified in the collaborative agreement is considered a co-deliverer with respect to the research activities agreed to in the collaborative agreement. A schedule to the collaborative agreement contains a detailed description of the Project activities, deliverables, timeframes to be carried out by DFO and the industry partner under the agreement or by a third party under contract agreement with DFO.

² Aquaculture production is defined as growing an aquatic species and further, that the aquaculture producer has ownership of the product or has an aquaculture license or lease to culture the product.

Some partners also participate in the distribution of knowledge such as the industry associations (e.g., Canadian Aquaculture Industry Alliance and provincial and regional associations).
First Nation communities.
Provincial Governments.

2.7 Program Resources/Budget

Funding allocations for ACRDP and AEBSP (including A-Base funding) are presented in the following two tables.

**Table 1: ACRDP funding allocations 2005-2012:**
Funding ($000)	2006-07	2007-08	2008-09	2009-10	2010-11	2011-12
Funding allocations	$ 4.500	$ 4.500	$ 4.500	$ 4.275	$ 4.275	$ 3.675

**Table 2: AEBSP A-Base funding allocations 2006-2012:**
Funding ($000)	2006-07	2007-08	2008-09	2009-10	2010-11	2011-12
AEBSP	$ 2.189	$ 2.169	$ 2.104	$ 1.953	$ 1.934	$ 1.885
A-Base	$ 6.349	$ 5.914	$ 5.997	$ 5.950	$ 6.241	$ 6.214
Total	$ 8.538	$ 8.083	$ 8.101	$ 7.903	$ 8.175	$ 8.099

3. Methodology

This section outlines the approach, the evaluation design, evaluation questions, the methodological approach, analytical methods as well as the limitations of the evaluation.

3.1 Project Management

The evaluation was conducted by an evaluation team led by a senior evaluation manager from the Evaluation Directorate at DFO and assisted by two external consultants. A working group was established consisting of representatives from the program and the evaluation team. The working group reviewed and provided feedback on this report.

3.2 Evaluation Approach and Design

The evaluation used a logic model approach combined with multiple lines of evidence drawing on both qualitative (e.g., interviews, program documents) and quantitative data (e.g., administrative data, survey). This approach is a non-experimental design, in which measurements are taken after the program has been implemented with no control group. This model was chosen because the program is a full coverage program, delivered across Canada. This model is appropriate to demonstrate the extent to which a program achieves issues of relevance and performance (effectiveness, efficiency and economy). The evaluation employed a variety of methods (e.g., interviews, document review) where the evidence drawn from these methods were triangulated to establish the key findings and recommendations.

This evaluation covered the period from 2006/07 to 2010/11 and was undertaken between November 2011 and April 2012 by Fisheries and Oceans Canada’s Evaluation Directorate. The evaluation is inclusive of the National Headquarters (NHQ) as well as the Newfoundland and Labrador, Maritimes, Gulf, Quebec, Central and Arctic, and Pacific regions.

3.3 Evaluation Questions

The evaluation questions were determined on the basis of the Treasury Board Policy on Evaluation (2009), by reviewing documents, and the results of the planning phase interviews with key program contacts. An evaluation matrix, presented inAnnex A, is organized by evaluation issue (relevance and performance including effectiveness, efficiency and economy) and outlines the evaluation questions.

3.4 Data Sources

3.4.1 Document and File Review

A review of documentation was conducted to identify evidence available to assess most evaluation issues. The following types of documents were reviewed:

government-wide documents (federal legislations, regulations and policies pertaining to aquaculture, Auditor General’s reports, federal Speeches from the Throne); and
department-level documents (Departmental Performance Reports, Reports on Plans and Priorities, DFO policies, etc.).

An Internet search was also conducted to identify information pertinent to the evaluation.

3.4.2 Interviews

Interviews were conducted with DFO management and staff, as well as representatives of aquaculture associations and provincial governments. In total, 41 individuals were interviewed: 26 were from DFO regions; 5 were from NHQ; 7 were representatives of aquaculture associations; and 3 were provincial officials. An additional 15 interviews, including industry representatives, were also conducted for the 5 case studies.

3.4.3 Survey

As part of the evaluation, DFO’s Evaluation Directorate conducted an online survey of ACRDP collaborators to address a number of evaluation questions related to the relevance and performance of the program.

The survey was administered electronically and was managed by the Evaluation Directorate in collaboration with the program. The survey focussed on ACRDP performance and identified opportunities for improvement.

The survey sought the views of ACRDP collaborators or representatives from organizations that have collaborated on or benefited from the ACRDP research projects within the last 5 years. Invitations to complete the survey were forwarded by email to 102 collaborators; 26 of the e-mail addresses were no longer valid leaving 76 potential collaborators to respond. In total, 32 ACRDP collaborators completed the survey (a response rate of 42%) which is generally considered an adequate response rate for online surveys. However, the total number of responses limited the analyses of data.

3.4.4 Case Studies

Five case studies were conducted covering the Department’s six regions: Newfoundland and Labrador; Maritimes and Gulf (two regions); Quebec; Central and Arctic; and Pacific. The case studies provided an in-depth view of the operation of the program, the nature of the research projects conducted, project success, utilization of results and project management. The 15 interviews mentioned, earlier, included DFO scientists (5), industry partners (4), other DFO staff (5) and a provincial government representative.

3.4.5 Site Visit

A site visit was made to locations in the Gulf (Moncton) and Maritimes regions (St. Andrews Biological Station) which permitted the evaluators to review specific project files, conduct interviews with the regional ACRDP coordinator and Co-Chairs, hold focus groups with researchers and obtain the views of senior staff concerning the operations of the program. The visit also gave the evaluators an opportunity to meet with representatives of an industry association at their offices.

3.5 Analytical Methods

The analysis methods used for this evaluation were tailored to the nature and availability of the data gathered. Extensive use of triangulation of evidence was used at arriving at the evaluation findings.

As part of the analysis of the interview data, the scales in the table below were used to report on the proportion and frequency of responses from respondents offering a particular perspective.

**Proportion of response terms**
Proportion Terms	Frequency Terms	Percentage Range
All	Always	100%
Almost all	Almost always	80-99%
Many	Often, usually	50-79%
Some	Sometimes	20-49%
Few	Seldom	10-19%
Almost None	Almost never	1-9%
None	Never	0%

3.6 Methodological Limitations

Within the methodologies employed, there were a few limitations and challenges. In order to minimize impacts on evaluation results, the different methods described above were combined to reach the same conclusions and reinforce their validity. That is, the results were compared from the different lines of evidence to confirm the validity of the findings. A combination of qualitative and quantitative data contributed further to the evaluation’s rigour. This ensured that none of the limitations associated with a given line of evidence was prejudicial to the validity of the evaluation results.

One limitation was that the interviewees may overly emphasize the positive aspects of the program, thus limiting the credibility of information provided. In order to mitigate these possible impacts, comments and findings were balanced with more “objective” data, namely the document review and the survey, which in itself did pose some limitations.

The limitations in interpreting the data from the online survey stemmed from two challenges: modest sample size and sampling error.

Modest sample size. Although the population was not large (with an upper limit of 102), the sample of 76 respondents presented some limitations as to the validity of data interpretation. For instance, a sample of this size does not lend itself to more complex analyses. Thus, the data that inform this report were approached in a relatively conservative manner; and data were not subjected to overly complex analyses.
Sampling error. The ability to reach potential respondents was limited by the fact that a portion of the invitations (25%) were not delivered due to inactive email addresses.

In order to improve the survey response rate, three attempts were made to contact each valid email address with a reminder or encouragement to complete the survey.

4. Major Findings

4.1 Relevance

The evaluation looked at whether there is a continuing need for the ACRDP and the extent to which the program is consistent with the federal government’s role and aligned with DFO’s roles, responsibilities and priorities. The evidence confirms there is a continuing need for collaborative science projects that can be of benefit to DFO and the aquaculture industry.

Key Findings: DFO is the lead federal department for aquaculture. The Department has historically been given lead responsibility for aquaculture, although this is not clearly stated in the Fisheries Act. Modifications to the ACRDP in 2011 ensure the program aligns solely with the Department’s core mandate to address regulatory matters. The collaborative nature of the program is essential to realizing benefits for both DFO and industry. Project benefits are not always documented.

Federal Responsibility and DFO’s Role

The economic developmental role of DFO and its focus on aquaculture is not clearly stated in the Fisheries Act which assigns a role to the Minister for the management of fisheries and makes no specific mention of aquaculture. This issue will be examined further in a subsequent evaluation of the Sustainable Aquaculture Program.

The federal government through, Fisheries and Oceans Canada, has been active in aquaculture over an extended period of time. The Department has been the lead federal agency responsible for aquaculture for close to three decades, with this responsibility first assigned by the Prime Minister in 1984. Over the ensuing years, the federal government has continued to affirm DFO’s role in aquaculture development and science through its approval of the Department’s aquaculture budgets and program plans. The decreasing availability of wild fish stocks, a need to encourage other industry employment opportunities and the movement towards the aquaculture industry worldwide have all given impetus to the federal government’s involvement.

In the early years, the Department held an extensive series of consultations which led to the Federal Aquaculture Development Strategy in 1995, a tool for fostering partnerships and cooperation between industry and all levels of government. These collaborative efforts were expected to help maximize the sustainable use of aquatic resources and increase industry's productivity, with a view to generating wealth and employment opportunities for Canadians. The strategy, [Cabinet Confidence], reaffirmed DFO’s role as the lead federal agency. DFO was tasked with creating policy conditions for the sustainable development of the aquaculture industry and its competitiveness in global markets.

In 2001, the federal government approved the Program for Sustainable Aquaculture (PSA), which included the ACRDP, as well as the Aquaculture and Environmental and Biological Science Program. Following the approval of the PSA, the Department released an Aquaculture Policy Framework (APF) in 2002 to guide its actions with respect to aquaculture, one of the fastest growing food production industries in the world. Through the ACRDP, the Department partners with industry by jointly undertaking research and development projects to help improve the competitiveness of the aquaculture industry. The ACRDP was referenced in the 2008 program approval document for the Department’sSustainable Aquaculture Program (SAP) as a sub-activity supporting applied research which the SAP could build upon.

On February 9th, 2009, the British Columbia Supreme Court held that the activity of finfish aquaculture is a fishery and falls under the exclusive jurisdiction of the federal government under the Constitution Act, 1867 and struck down most of the provisions of the British Columbia regulatory scheme related to finfish aquaculture. Subsequent to the Court’s decision, the federal government enacted the Pacific Aquaculture Regulations which empowers the Minister to issue licences for aquaculture activities in British Columbia and in particular waters off its coasts. This decision is applicable only to the Province of British Columbia. In the other provinces or territories, the federal government has not enacted regulations applicable specifically to aquaculture activities.

Alignment with Federal and DFO Priorities

The ACRDP is a sub-activity of the Sustainable Aquaculture Program and is part of DFO’s Program Activity Architecture (PAA) aligned with the Strategic Outcome of an Economically Prosperous Maritime Sectors and Fisheries. This in turn is consistent with two of the federal strategic outcomes identified in the 2010/11 Performance Report of the Government of Canada, namely strong economic growth, and an innovative and knowledge-based economy.

The ACRDP’s link to the federal objective of strong economic growth is evidenced by information contained in the Department’s Report to Launch the Aquaculture Sustainability Reporting Initiative issued in 2012. This report highlights the economic impact of aquaculture in Canada, noting that the industry generates about $2 billion in total economic activity, with over $1 billion in GDP and about $0.5 billion in labour income. Aquaculture production occurs across Canada, with activity concentrated in British Columbia and the Atlantic provinces.

The program’s link to the federal objective of an innovative and knowledge-based economy is evidenced by the research and development work undertaken through the ACRDP. This work has supported three broad areas, as previously noted: Best Performance in Fish Production, Environmental Performance and Optimal Fish Health. Going forward, the Department no longer supports research on fish production, as it was deemed not aligned with the Department’s core mandate. This change was introduced in October 2011, along with a budget reduction of $2 million. The other two areas continue to be aligned with the Department’s responsibilities related to environmental interactions and fish health, as confirmed by the Minister of Fisheries and Oceans in the House of Commons January 30, 2012 (Question # 292 on the Order Paper).

Continuing Need for the Program

There is a continuing need for the ACRDP, operating in a manner that meets DFO’s responsibilities and, at the same time, assisting industry with issues affecting fish health and environmental performance.

Research continues to be needed to provide a stronger base of scientific knowledge about the dynamic and changing conditions of the oceans and the impact on aquaculture operations. The evolving nature of the aquaculture industry and its changing needs has demonstrated a continuing need for joint research. While the salmon farming industry has matured considerably since the ACRDP inception, the shellfish industry is still dominated by a large number of smaller players with little capacity (financial or expertise) to conduct research on their own. In addition, in the evidence gathered for the evaluation, the existence of other sources of scientific knowledge capable of addressing industry and regulatory issues were not identified. The collaborative nature of the program has provided benefits to industry, through joint research, and to DFO and other regulators in support of their regulatory roles.

The case studies conducted for this evaluation demonstrate that ACRDP projects contribute to DFO’s regulatory role and also provide benefits to industry. For example, one project investigated a means of monitoring the impact of aquaculture cages on the underlying ocean floor. DFO Habitat Management staff used the knowledge gained about the changes occurring to inform their on-going surveillance of ecosystem indicators and monitoring of habitat conditions for any harmful, disruptive or destructive occurrences. The project benefited the two industry partners, as it validated an innovative technology that would reduce their monitoring costs.

Although such benefits are an integral and essential reason for undertaking an ACRDP project, they have not always been explicitly stated in project documents, either at the outset or upon completion of a project. Furthermore, the program as a whole has given rise to other benefits. First, the collaborative nature of the program has been instrumental in building a strong partnership between industry and DFO science. Second, DFO scientists benefit from industry providing assistance and access to large-scale industrial sites for research activities, and industry benefits from its active involvement in conducting mutually beneficial research. Third, DFO scientists benefit from a more direct understanding of industrial operations, an important element in understanding how regulations can impact on business in the development of a sustainable aquaculture industry.

Industry has expressed a strong need for the program to continue as originally planned. Almost all survey respondents indicated a strong need for all three of the priority areas. Many respondents also indicated that the research and development work could not be done without ACRDP assistance. Industry associations noted that DFO research published in peer reviewed journals is highly regarded and considered to be unbiased and can assist in responding to concerns brought forward by environmental groups.

Industry has valued DFO’s use of applied research to gain insight into the impact of aquaculture operations on the environment and to help guide operations in a manner that will minimize these impacts. There were, however, indications made during the interviews for the evaluation that some industry participants may not be interested in continuing collaboration with DFO to assist in regulation and policy making.

4.2 Effectiveness

Effectiveness is defined by the Treasury Board Policy on Evaluation (2009) as the extent to which a program is achieving its expected outcomes. For the ACRDP, the evaluation examined the extent to which it is achieving its immediate and intermediate outcomes.

Immediate outcomes as stated in the program logic model are:

Research and development projects are aligned with industry needs and priorities
Aquaculture stakeholders are engaged
Industry and management awareness of the scientific evidence

Intermediate outcome as depicted on the program logic model is

Fish production, fish health and environmental performance science are utilized by the aquaculture industry and management

The evaluation found that the program has made progress in achieving its immediate outcomes. Based on qualitative evidence, the evaluation concluded that the program was making some progress in achieving its intermediate outcome.

Achievement of Immediate Outcomes

Key Findings:ACRDP projects are aligned with industry needs and its priorities, and aquaculture stakeholders are actively involved in the process. However, the program is not making optimal use of mechanisms available for transferring ACRDP research projects results to stakeholders and partners.

Industry Needs and Priorities

The collaborative nature of the program ensures that the ACRDP is aligned with industry needs and priorities. ACRDP projects are initiated by industry proponents who are involved in the development of the proposals with DFO staff. A proposal is formally submitted to the Department and currently assessed by the TRC of which aquaculture stakeholders are members. Both departmental staff and industry participants considered the project selection process appropriate.

When a project proposal is approved by the Director General of the Strategic and Regulatory Science Directorate at NHQ, a collaborative agreement is signed which includes a commitment for the project proponent to contribute a pre-determined amount to the total cost of the project. The project proponent is required to contribute a minimum of 30% of the cost of the project either through in kind or cash contributions. Data provided by program management shows that, overall, industry contributed 11.7% cash and 37.7% total (cash and in-kind) of the ACRDP project costs from 2006 to 2011. The commitment to make this contribution was considered by many of those interviewed to be an indication that a project is addressing an industry priority.

The survey of industry participants revealed that 83% think that the ACRDP projects met industry priorities to varying degrees, ranging from “a great extent” to “a limited extent”.

Stakeholder Engagement

Aquaculture stakeholders are engaged in many aspects of the delivery of the ACRDP. Industry stakeholders are members of the RMCs and the NSC and are involved in the project selection process. Industry representatives expressed satisfaction with their involvement in the RMCs and NSC. Project applicants participate in the development of the research projects, and many were engaged in the conduct of the projects on an on-going basis and were satisfied with their level of engagement in the ACRDP. This included working closely with DFO scientists to provide access to fish farms, helping them understand industry operations, facilitating gathering of samples for analysis and receiving results. Many of those interviewed and survey respondents were satisfied with the planning and implementation aspects of the project management process for the ACRDP.

Transfer of Research Results

The transfer of ACRDP research results to aquaculture stakeholders and partners is key to the success of the program. If project participants or other industry stakeholders are not informed of research results, the program’s value as a whole is diminished and the usefulness to the aquaculture industry and DFO is reduced.

Through interviews and the online survey, it was found that aquaculture stakeholders and partners were better informed of the results of projects that they had been directly involved with then those that they had not participated in.

Communication Mechanisms

A number of communication mechanisms are used to disseminate research results to project applicants, in particular, and to the aquaculture industry and other stakeholders and partners, in general. These include project fact sheets, peer reviewed publications, conferences, technical reports and workshops. A good example is the Canadian Aquaculture R&D Review which provides a compendium of aquaculture research and development projects. These projects have been carried out in various organizations and laboratories across the country and are supported by various funding mechanisms. The Review is currently produced with support from the ACRDP and the Aquaculture Association of Canada.

However, two key means of communicating are not being used to their fullest potential:

a.) Final Project Reports

These reports have not always been produced on a timely basis. Of the five case studies conducted for the evaluation, only one had a final report. Industry members have indicated they need these reports to assist in improving their operations. The lack of final reports has been an impediment to the technology transfer aspect of the ACRDP.

b.) DFO Website

DFO’s website could be an effective means of sharing results from ACRDP research projects with others. However, the site has an extremely limited amount of such information for its targeted audiences. The site provides administrative information such as the program’s objectives, eligibility criteria, application process and so forth. Abstracts for most of the projects (318) that have been funded (357) are available and include references to published papers relating to the project. However, with the exception of nine fact sheets on completed projects, the site does not directly provide any information on project results.

There is considerable room to improve ACRDP content on DFO’s website. Posting of such documents as final reports, fact sheets, conference papers and so forth, along with project contact details, would enable industry members and other interested parties to become aware of ACRDP research activities and results. We understand that there are some limitations, such as, the length and cost of translating some documents, on what can be efficiently posted on the website, nevertheless, it is an area that could be enhanced.

If knowledge is generated, but not effectively communicated, fish farmers will not reap the rewards of lessons learned and best practices. Final project reports have not been consistently produced on a timely basis. Industry indicated their preference of communicating research results in this format.

Connectivity has greatly improved over the last decade and websites are cost effective means to reach target audiences over other media, such as print. In spite of many individuals indicating their satisfaction with communication of results from other projects, there is still a need for improvement on the mechanisms used for communicating ACRDP research results.

Recommendation # 1:

It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science ensure that project results are effectively communicated to collaborators in particular and to the industry in general. Specific improvements should include:

Final reports, that are easily understood and in plain language, are produced for all projects on a timely basis; and
The DFO website as it pertains to the ACRDP is maintained on a regular basis.

Aquaculture Environmental and Biological Science Program

Key Finding:The extent of AEBSP’s contribution to the achievement of expected results for the ACRDP or any other science program could not be determined.

AEBSP’s role is to provide funding for the researchers who conduct research projects and communicate their results to stakeholders and partners of the ACRDP and other aquaculture science activities. Given that there are no measurable results attributable specifically to the AEBSP, the program appears to be a funding mechanism providing salaries for researchers in support of DFO’s aquaculture science programs. While AEBSP funding provided the research capacity to carry out the ACRDP projects, the evaluators were unable to assess the extent to which the achievement of ACRDP outcomes could be attributed to the AEBSP.

Achievement of Intermediate Outcome

Utilization of Results

Key Finding: The evaluation concludes that results from ACRDP research projects are being used to some extent. However, there is no systematic follow-up of completed ACRDP projects to determine the degree to which they are being used.

The expected intermediate outcome that the scientific evidence produced from the research projects would be used to help improve the aquaculture industry could not be readily assessed. Most ACRDP researchers indicated that there is no mechanism to measure the extent to which research results are used. In addition, there is no formal follow-up on the actual use of research results, although the researchers believe they would know if research results were being applied through informal discussions with project proponents.

Still, the evaluators found evidence that some research results were used to improve the performance of the aquaculture industry. For example, almost all survey respondents indicated that they could make some use of the new knowledge arising from ACRDP projects and that they had in fact adopted new practices or knowledge to some degree. Many indicated that their aquaculture operations had improved as a result of ACRDP projects. Much of the information obtained, however, provides only qualitative indications of the impact that the ACRDP has had on the industry.

:The evaluation involved examining the use of results for the three broad objectives of ACRDP: Best Performance in Fish Production, Optimal Fish Health, and Environmental Performance.

a) Best Performance in Fish Production

Many survey respondents noted that the projects have led to new technologies and practices which, overall, have improved aquaculture operations in the area of fish production. DFO researchers indicated there are varying degrees of success for a project and that while there may be some immediate indications of success, it may take years to know if it has made a difference in aquaculture operations or the industry overall.

One of the case studies for the evaluation found that project results impacted positively on production costs for aquaculture operations. For example, a project dealing with finfish diets to reduce phosphorous discharge into lakes resulted in a more effective feed composition, leading to lower feed costs.

b) Optimal Fish Health

Both survey respondents and researchers reported that projects had contributed to improving fish health. Many survey respondents indicated that new technologies and practices arising from the ACRDP projects overall have improved aquaculture fish health. DFO researchers described how ACRDP’s research work in bay area management techniques has been accepted by industry and regulators as a means for restricting fish disease and improving fish health.

The case studies also provided some indication of the use of results to improve fish health. For example, work carried out on a sea lice treatment product in New Brunswick resulted in cost savings to the industry by preventing what would otherwise have been the necessary destruction of fish.

c) Environmental Performance

Many survey respondents and the case studies indicated that ACRDP research projects have benefited environmental performance. For example, one project resulted in a new and less expensive approach to monitoring the environmental impact of aquaculture sites on the ocean floor. The results also provide a more accurate assessment of bottom conditions, which is of benefit to Habitat Management staff in managing fish habitat.

As mentioned earlier, the transfer of research results is a key component of success. Being aware of lessons learned and best practices are a precondition to utilizing and incorporating that knowledge into everyday practices. In spite of there being evidence that research results were used to improve the performance of the aquaculture industry, there is a lack of follow up on the actual use of research results. For example, effectively monitoring impacts of research projects could enable the ACRDP to target its resources on research areas that yield the highest return on investment. It could also assist in priority setting. This could lead to more focused communication efforts which could reduce the amount time it would take for aquaculture operations to adopt new practices.

Recommendation # 2:

It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science ensure that the ACRDP systematically monitor the utilization of research projects findings and their impacts on aquaculture operations and regulatory agencies.

4.3 Efficiencies and Economy

The modifications made to the ACRDP in October 2011 have resulted in changes to the program delivery model from a regional focus to a national one. Because of the limited time that the new program model has been in place, it is too soon to provide an assessment of the structure and processes that have been established for the new delivery model. Instead, the evaluation team focused attention on areas that could be improved regardless of the delivery model in place. The evaluation noted areas such as priority setting, performance measurement and tracking of program costs, that could be improved regardless of the delivery model in place.

Priority Setting

Key Finding: There is a need for a systematic process for setting ACRDP priorities and reviewing these periodically to confirm on-going relevance with evolving industry needs and DFO priorities

When the ACRDP was created in 2000, the Department established three broad research and development objectives:

Best Performance in Fish Production
Optimal Fish Health
Environmental Performance

These objectives remained until best performance in fish production was eliminated from the program in October 2011. While these three broad categories were identified as objectives for the program, they came to be considered by many as the ACRDP national priorities. With the categories being so broad, all proposals submitted by industry were considered to be addressing a national priority.

There is a need for a more systematic way of establishing and re-assessing priorities for ACRDP projects on an on-going basis. Since the program’s inception, little consideration has been given to the evolving nature of the aquaculture industry and its changing needs. For example, the salmon farming industry has matured considerably, while the shellfish industry is still dominated by a large number of smaller players. Much feedback was received during the evaluation on the lack of strategic direction and priorities against which to assess the project proposals. ACRDP priorities should reflect those set out in the DFO Five-Year Research Agenda and each proposal needs to demonstrate its alignment to one of these.

Given the limited resources that are available to the ACRDP for research, it is important that priority setting be carried out so that funds are allocated to those projects that will provide the most benefits to DFO and industry.

Recommendation # 3:

Performance Measurement

Key Finding: The ACRDP does not have its own performance measurement strategy but will be included in the overall performance measurement strategy for the Sustainable Aquaculture Program

A performance measurement strategy involves the selection, development and ongoing use of performance measures for program management and decision making. The key performance indicators and targets for the ACRDP have not yet been identified. As part of the Department’s initiative to have performance measurement strategies in place for all of its programs, the ACRDP is included as a component in the overall strategy for the Sustainable Aquaculture Program.

Program Resources

Key Finding: The total program costs for the ACRDP are not known.

The ACRDP provided operations and maintenance funds in the amount of approximately $4 million annually to support research projects. In addition, some of the salaries in respect of work carried out for the ACRDP, are paid by funding from the AEBSP. The AEBSP was initially established in 2001, with funding of $2.5 million annually including 16 FTEs and was part of the Program for Sustainable Aquaculture. The Biological Science component of the AEBSP, as mentioned in the program approval document for the PSA, was established to support biological science research activity and to provide the capacity to administer the ACRDP. Initially, this portion of the program was funded with 10 FTEs and $1.5 million per year in O&M distributed across the regions and at NHQ to strengthen research and development capacity in DFO research centres and management of the ACRDP at the national level.

The amount of funds contributed by the AEBSP to assist in delivering the ACRDP could not be determined. Consequently, the total program costs for delivering the ACRDP are not known.

The AEBSP was designed to provide research and scientific advice, and support strategies that promote the sustainability of the aquaculture industry and administer the ACRDP. Over time, the AEBSP has evolved and its funding now includes the initial amount provided through the Program for Sustainable Aquaculture in 2001, as well as ongoing funding (A-Base) that had been allocated to the Ecosystem and Oceans Science Sector for aquaculture related work. The funding allocated to this program is approximately $8 million annually.

The AEBSP has become the mechanism used to provide research capacity not only to ACRDP but to other science related programs for aquaculture. However, given the absence of an adequate process for tracking the AEBSP funding, the evaluators were unable to identify the amounts attributable to the ACRDP or to any other science related program for aquaculture.

Because of the inability of the Ecosystems and Oceans Science Sector to identify funds expended on specific programs through the AEBSP, the evaluators were unable to assess the extent to which the program was contributing to the achievement of outcomes for any program to which it enables.

Recommendation # 4

Treasury Board’s Policy on Management, Resources and Results Structure defines a program as a group of related resource inputs and activities that are managed to meet specific needs and to achieve intended results, and that are treated as a budgetary unit. The ACRDP as such accepts and reviews research project proposals from the aquaculture industry and jointly undertakes and funds projects with successful applicants through collaborative agreements. However, the ACRDP consists solely of A-base Operations & Maintenance funding and as a result, all of its funding provided through these collaborative agreements is to reimburse research related expenses and does not include salaries for DFO researchers and the administration of the program. The AEBSP provides the researchers and salaries that are used to carry out the ACRDP activities and outputs to yield the desired outcomes. While the AEBSP scientists support other programs to some extent, their key activities relate to delivering the ACRDP. In essence the AEBSP is the salary component of the ACRDP. Consolidating the two programs would provide a more complete picture of the delivery of the ACRDP.

Recommendation # 5

Leveraging Collaborators’ Contributions

Key Finding: Industry is exceeding the current required minimum contribution of 30% cash and in-kind contributions to ACRDP projects, achieving an average of 36.7% for the period 2001 to 2011. However, the original target for industry to contribute 50% of the costs of ACRDP projects has not been achieved.

The original mandate of the program was to phase in industry contributions as industry capacity developed, with a 50% contribution target to be reached within five years of the program start. This target was never achieved as the industry, with perhaps the exception of some of the larger companies, generally did not have the financial resources to support R&D initiatives at a 50% contribution level. This issue was discussed at the NSC in 2004, and based on the recommendation of the Committee, the Department decided that the status quo be maintained in regards to industry contributions to ACRDP projects.

While the ACRDP is exceeding the current minimum contributions, it is well short of the target of 50% industry contribution to ACRDP project costs. Since the decision of 2004, industry is required to contribute a minimum of 30% (7.5% cash, 22.5% in-kind) of the ACRDP funding dollars to each project. Contributions can also be included from other public or private sector partners. The funding of each approved project is established through a collaborative agreement between the industry proponent, DFO and other partners. Data provided by program management shows that, overall, industry contributed 11.7% cash and 37.7% total (cash and in-kind) of the ACRDP project costs from 2006 to 2011.

5. Conclusions

There is a continuing need for the ACRDP to assist in achieving optimal fish health and environmental performance as it would benefit industry, DFO and other aquaculture regulatory agencies.

The federal government through Fisheries and Oceans Canada has been active in aquaculture over an extended period of time. The British Columbia Supreme Court recently ruled that the Province did not have jurisdiction to enact laws for the purpose of regulating finfish aquaculture in British Columbia. As a result of that decision, the federal government enacted the Pacific Aquaculture Regulations which empower the Minister of Fisheries and Oceans Canada to issue aquaculture licences for aquaculture activities in British Columbia and in particular waters off its coasts.

The federal government has been committed to the development of the aquaculture industry for more than 25 years. During this time, several initiatives to assist the aquaculture industry have been supported and DFO has been named as the lead federal department for aquaculture. The role of ACRDP was to help improve the competitiveness of the aquaculture industry.

There is evidence to conclude that the ACRDP is aligned with federal government priorities. Even if there is no specific mention of aquaculture in the Fisheries Act over the years the federal government has continued to affirm DFO’s role in aquaculture development and science through its approval of the Department’s aquaculture budgets and program plans.

The evaluation concludes that the program has made some progress in achieving its expected immediate and intermediate outcomes. The ACRDP is achieving two of its immediate outcomes:

ACRDP projects being aligned with industry needs and priorities; and
Aquaculture stakeholders are engaged.

For the third immediate outcome: industry and management are aware of scientific evidence, it is concluded that ACRDP efforts at technology transfer are weak and do not reach out to all the stakeholders and partners. The lack of final project reports, fact sheets and an up-to-date DFO website is hampering technology transfer of the program.

The evaluation concludes that results from ACRDP research projects are being utilized to some extent, however, there is no systematic follow-up of completed ACRDP projects to determine the degree to which they are being used.

The modifications made to the ACRDP in October 2011 have resulted in changes to the program delivery model, particularly going from a regionally delivery model to one with a national focus. Because of the limited time that the new program model has been in place, it is too soon to provide an assessment of the structure and processes that have been established for the new delivery model. Instead, the evaluation team focused its attention on areas that could be improved regardless of the delivery model in place. The evaluation noted areas such as priority setting, performance measurement and tracking of program costs, that could be improved regardless of the delivery model in place.

The evaluation was unable to determine the overall costs of the ACRDP as the funding provided to the program through the AEBSP are not tracked. While the AEBSP did provide funding for the research capacity to carry out ACRDP, no assessment can be made of the effectiveness of the AEBSP as a separate program.

Given that the ACRDP consists solely of A-base O&M funding and does not include salaries for researchers and the administration of the program, there is no evidence of the ACRDP having its own management framework. Consolidating the ACRDP funding under the AEBSP program would better align the two activities with Treasury Board’s definition of a program.

Annex A: Evaluation Matrix

This evaluation matrix summarizes the indicators and data collection methodologies required to address each of the evaluation questions developed for this evaluation.
Evaluation Issue/ Question	Indicator	Methodology
Relevance
1.1 To what extent is there a continued need for ACRDP?	Evidence/demonstration of the on-going demand for the program from the aquaculture industry State of science capability of industry, provinces Evidence of evolving aquaculture issues requiring DFO science input. Evidence the program responds to changes in government and departmental policies and priorities Growth and future demand for farmed seafood and its acceptability	Document review; Case studies Web Survey Interviews
1.2 To what extent is the program consistent with Federal government roles and DFO priorities?	Consistency with Departmental strategic objectives and priorities Consistency with federal government objectives and priorities.	Document review; Case studies Interviews
1.3 Are program activities aligned with the roles and responsibilities of DFO and appropriate to the federal government?	Constitutional need or link with federal legislation;	Document review; Interviews
1.4 To what extent do the program activities not duplicate the activities of other federal/provincial programs or funded organizations?	No duplicative programs funding similar R&D services, e.g. DFO programs such as Aquaculture Innovation and Market Access Program (AIMAP), Program for Aquaculture Regulatory Research (PARR); other federal agencies such as National Science and Engineering Research Council (NSERC) or the Industrial Research Assistance Program; and Provincial programs; Genome Canada Client’s/beneficiaries views on whether the program or component of the programs should be transferred to another level of government, non-government organization or to the private sector. Aquacultures research and development work done by industry without ACRDP involvement.	Document review; Case studies Web Survey Interviews
Performance Effectiveness
2.1 To what extent are research and development projects aligned with industry needs and priorities?	Project selection process and views on industry needs of DFO managers, scientists and industry. Industry views on the project selection process. Industry views on the alignment of project results with their needs Industry contribution ($ and in-kind) to completed projects. Industry view on whether the project could have been undertaken on its own or with some other organization	Interviews; Web survey Document Reviews
2.2 To what extent is industry and management engaged?	Industry views on the project management process Industry views on whether the project process allows them to be engaged in an ongoing manner. Industry views on whether the projects produced results that were in line with their expectations Input and involvement of industry representatives that sit on the ACRDP NMC and RMCs and type/level of engagement; Level of satisfaction expressed by industry with the program.	Document reviews Interviews; Web survey
2.3 To what extent are industry and management aware of the scientific evidence?	Project participant are kept informed of project progress and the final project results Other relevant aquaculture firms are made aware of the project results Types of media or vehicles used to disseminate findings e.g. publications; fact sheets, etc. Perceived effectiveness of media and distribution methods;	Interviews ACRDP Case studies; Web survey
2.4 To what extent are fish production, fish health and environmental performance science results used by the aquaculture industry and management?	Project results are made available to industry participants in a manner and format such that they can use and implement the results; Evidence that direct and indirect clients are understanding and adopting research results. Evidence that research results have led to new aquaculture technologies and practices Actual or potential applications of science results to fish health, production and or environmental performance, by industry , ACDRP management, DFO scientists	Interviews ACRDP Case studies; Web survey
2.5 What external/internal factors influenced and have impacted on the achievement of outcomes?	Internal factors that influence outcomes and their importance in project selection, project execution and knowledge transfer to the private sector Financial issues Annual program resources (Full Time Equivalents (FTE), Operation and Maintenance) for both AEBSP and ACRDP). External factors that influence outcomes and their importance in project selection, project execution and knowledge transfer to the private sector Barriers and challenges to accessing science knowledge produced	Interviews ACRDP Case studies; Web survey
2.6 What are some of the lessons learned from the ACRDP which would help improve the program outcomes?	Unintended outcomes, both positive and negative and their importance	Interviews; ACRDP Case Studies;
3.1 Is the design and delivery of the program appropriate to produce the desired outputs?	Appropriateness of program management and delivery processes (program awareness, project review, project implementation, delivery of final report, assistance to industry in understanding of scientific findings) in delivering on the program outputs: collaboration agreements; research reports and publications. Program delivery processes that do not add significant value Availability of performance measurement system	Interviews; Web survey Case Studies; Document review
3.2 How could the efficiency of the programs be improved?	Changes to delivery processes at HQ Changes to delivery processes at Regions Changes to project management practices	Interviews; Web survey Case Studies; Document review
4.1 Is the program operating in a way that minimizes the use of resources to achieve its intended outcomes?	Leverage achieved with additional resources from other departments and industry or other sources – ratio of funds/in-kind leverage – total funds/in-kind leverage against ACRDP investments; Improvements that could be made to the collaborative model for conducting applied aquaculture Opportunities to improve leverage	Interviews Document review

Annex B: Management Action Plan

Management Action Plan in Response to Evaluation of the Aquaculture Collaborative Research and Development Program and the Aquaculture Environmental and Biological Science Program
Recommendations
Context: While the project collaborators benefit from direct contact with the DFO researcher, the inconsistent production of final reports and updating the DFO website with complete project information for all completed projects, is hampering the transfer and sharing of scientific information emanating from the program. Recommendation 1: It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science, ensure that project results are effectively communicated to collaborators in particular and to the industry in general. Specific improvements should include: Final reports that are easily understood and in plain language, are produced for all projects on a timely basis; and The DFO website as it pertains to the ACRDP is maintained on a regular basis.
Strategy
ACRDP Collaborative Agreements stipulate that final reports are due within three months of the project completion date. To ensure that these reports are completed and received on a timely basis, Aquaculture Science Branch (ASB) will develop an approach to enforcing this reporting deadline. This approach will be communicated to all project proponents and could include such measures as providing a reminder to project managers about their reporting requirements once projects are completed and /or withholding funding for future projects until such a time as overdue project reports are received. Given the changing management model from a Regionally to Nationally managed Program, there is an opportunity for a more consistent approach to ensuring industry and DFO sign-off and delivery of ACRDP final reports To ensure that final reports are easily understood and in plain language, ASB will review the current ACRDP final report template and ensure that the information requested is most appropriate and in line with in communicating the project outcome(s). To ensure that the ACRDP web site is maintained on a regular basis, ASB will develop an Action Plan for addressing the regular updating of the ACRDP website. The plan will include a summary of information intended to be shared on the website (e.g., project results), a schedule for information gathering, translation of material, approval of updated material and work planning with web-developers to upload new content. ASB has dedicated resources in helping with this objective and other communications around ACRDP.

Management Actions	Actions Completed	Actions Outstanding	Target Date	Supporting Evidence
Enforcement of Final Report submission deadline.		The Director General of Strategic and Regulatory Science Directorate will send a reminder letter to Regional Directors of Science to communicate the enforcement of final report deadlines (as stipulated in Collaborative Agreements) for ACRDP projects and to note that there will be repercussions if these timelines are not adhered to (e.g., for future funding).	September 30, 2012	Letter to RDSs outlining requirements.
Final Report Adjustments – ASB will conduct a review / analysis of the ACRDP final report template to assess for areas of improvement.		ASB will analyse current final reports and the reporting template to develop recommendations for improvement such that the reports can clearly communicate project outcomes. A revised template / guidance document will be developed for use by program proponents.	March 31, 2013	Revised final report template developed and shared with proponents.
Website maintained on a regular basis - ASB will develop an Action Plan for updating the ACRDP website to current and subsequently keeping it up to date.	ASB has already ensured that summaries on funded projects to date have been uploaded to the website. This will be incorporated in the Action Plan.	ASB will develop an Action Plan for ensuring that the ACRDP website is brought and kept up to date, identifying an annual calendar of activities and time lines of completion.	March 31, 2013	Action Plan developed and utilized in maintaining the ACRDP website.
Recommendations
Context: Results from research projects are being utilized to some extent, however, there is no systematic follow-up of completed projects to determine the degree to which they are being used. Recommendation 2: It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science ensure that the ACRDP systematically monitor the utilization of research projects findings and their impacts on the aquaculture operations and regulatory agencies.
Strategy
ASB will develop an engagement strategy to better gather and summarize information concerning the output and utilization of research projects. The engagement strategy will include a regular follow-up process to facilitate communication from program users regarding the utilization of research project findings and impacts on the aquaculture operations and regulatory agencies, and the subsequent development of a report on project impacts.
Management Actions	Actions Completed	Actions Outstanding	Target Date	Supporting Evidence
Development of an engagement strategy to better gather and summarize information concerning the output and utilization of ACRDP research projects.		Development of an engagement strategy which will incorporate a follow-up process with project stakeholders to gather information on impact and utilization of ACRDP project outcomes and the development of a regular report based on the findings.	March 31, 2013	Engagement Strategy developed and implemented.
Recommendations
Context: To confirm the on-going relevance of industry and DFO priorities in the context of the evolving nature of the aquaculture industry, a systematic process for the setting and review of priorities is needed. Recommendation 3: It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science, review industry and DFO priorities periodically so as to reflect changing needs of the aquaculture industry and the Department.
Strategy
ASB will initiate an annual priority setting exercise for the ACRDP, consulting with clients, to determine priority areas in which research should be focussed for the year. This information will guide project applications and funding recommendations by the Technical Review Committee (TRC) for each year.
Management Actions	Actions Completed	Actions Outstanding	Target Date	Supporting Evidence
Establishment of a priority setting exercise – to collect information on research priorities from stakeholders and integrate these with DFO research priorities.	ASB has initiated a priority setting practice as an initial means to generate discussion on priorities for the program. The initial priority setting discussion was held at the program’s general meeting, March 27 – 28th, in Ottawa, where program stakeholders and DFO staff were present.	Finalize developed priorities and share with stakeholders.	December 31, 2012	ACRDP research priorities are developed, shared with stakeholders.
Recommendations
Context: Given the absence of an adequate process for tracking AEBSP funding, the evaluators were unable to identify the extent to which it was contributing to the achievement of outcomes for the ACRDP or other programs which it enables. Recommendation 4: It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science establish a process for tracking the funding of the Aquaculture Environmental and Biological Science Program so that its contribution to the achievement of the expected outcomes of the programs which it enables can be identified.
Strategy
The Aquaculture Environmental and Biological Science Program principally supplies salary dollars for researchers contributing to the main outcomes of the sub-activities under the Sustainable Aquaculture Program. To track salary funding, the Aquaculture Science Branch will put in place an annual survey to measure the portion of time each AEBSP researcher spends on these outcomes – including ACRDP, Canadian Science Advisory Secretariat (CSAS) processes, the Program for Regulatory Research (PARR) and Other Aquaculture Science Activities.
Management Actions	Actions Completed	Actions Outstanding	Target Date	Supporting Evidence
ASB will develop an annual process for follow-up with AEBSP scientists to determine the proportion of their time spent on ACRDP, CSAS, PARR, and other activities, over the previous year.		ASB will develop a follow-up process to engage with AEBSP scientists annually to determine the proportion of their time spent on ACRDP, CSAS, PARR, and other activities, over that previous year.	March 31, 2013	Developed follow-up process and communication to scientists.
Recommendations
Context: AEBSP provides the researchers and salaries that are used to carry out the ACRDP activities and outputs to yield the desired outcomes. While AEBSP scientists support other programs to some extent, their key activities relate to delivering the ACRDP. In essence, the AEBSP is the salary component of the ACRDP. Consolidating the two programs would provide a more complete picture of the delivery of the ACRDP. Recommendation 5: It is recommended that the Assistant Deputy Minister, Ecosystems and Oceans Science determine whether Aquaculture Biological and Environmental Science Program and the Aquaculture Collaborative Research and Development Program should be consolidated in order to more fully include the basic components required for a program as defined by Treasury Board.
Strategy
ASB will meet with Corporate Planning, Performance and Risk Management (CPP&RM) to discuss how ACRDP and AEBSP meet the criteria of the Treasury Board Secretariat’s (TBS) Policy on Management, Resources and Results Structure definition of a program and determine if there is merit in consolidating the sub-activities under the PAA.
Management Actions	Actions Completed	Actions Outstanding	Target Date	Supporting Evidence
Aquaculture Science Branch will meet with Corporate Planning, Performance and Risk Management to assess if ACRDP and AEBSP meet the Treasury Board’s definition of a program and will determine if there is merit in consolidating the sub-activities under the PAA.	Initial meeting between ASB and CPP&RM was held on May 31st, 2012. It was determined that since there is discrete and dedicated annual departmental O&M funding for ACRDP ($2.2 million) for carrying out aquaculture science projects with industry and since all other criteria are met, it is compliant with TBS`s MRRS policy.	Aquaculture Science Branch will conduct an analysis determine if there is there is merit in consolidating the AEBSP and ACRDP.	March 31, 2013	Assessment on how ACRDP and AEBSP meet the definition of a program and analysis of the merits of consolidating the sub-activities.

Common menu bar links

Institutional links

Reports

Resources

Proactive Disclosure

EVALUATION OF THE AQUACULTURE COLLABORATIVE RESEARCH AND DEVELOPMENT PROGRAM AND THE AQUACULTURE ENVIRONMENTAL AND BIOLOGICAL SCIENCE PROGRAM

FINAL REPORT JUNE 2012

EVALUATION DIRECTORATE DIRECTION GÉNÉRALE DE L’ÉVALUATION

TABLE OF CONTENTS

ACRONYMS

EXECUTIVE SUMMARY

Introduction

Program Profile

Evaluation Methodology

Evaluation Findings And Recommendations

Relevance

Effectiveness

Efficiency & Economy

1. Introduction

1.1 Context of the Evaluation

1.2 Structure of the Report

Of Special Interest to the Reader

2. Program Profile

2.1 Mandate/Background/Objectives

2.2 Program Activities

Collaboration and Leadership

Knowledge Generation

Knowledge Transfer

2.3 Program Expected Results/Performance Measurement

2.4 Governance

2.5 Program Logic Model

2.6 Program Partners/Clients/Stakeholders

2.7 Program Resources/Budget

3. Methodology

3.1 Project Management

3.2 Evaluation Approach and Design

3.3 Evaluation Questions

3.4 Data Sources

3.4.1 Document and File Review

3.4.2 Interviews

3.4.3 Survey

3.4.4 Case Studies

3.4.5 Site Visit

3.5 Analytical Methods

3.6 Methodological Limitations

4. Major Findings

4.1 Relevance

Federal Responsibility and DFO’s Role

Alignment with Federal and DFO Priorities

Continuing Need for the Program

4.2 Effectiveness

4.3 Efficiencies and Economy

5. Conclusions

Annex A: Evaluation Matrix

Annex B: Management Action Plan

FINAL REPORT
JUNE 2012

EVALUATION DIRECTORATE
DIRECTION GÉNÉRALE DE L’ÉVALUATION