EVALUATION OF THE BRITISH COLUMBIA
AQUACULTURE REGULATORY PROGRAM

6B165
FINAL REPORT
AUGUST 13, 2015

EVALUATION DIRECTORATE


Table of Contents

ACKNOWLEDGEMENTS AND ACRONYMS


ACKNOWLEDGEMENTS

The Evaluation Directorate acknowledges and thanks all individuals who gave of their time and input for this evaluation of the British Columbia Aquaculture Regulatory Program (BCARP), in particular, all stakeholders and staff members who shared their thoughts through site visits, in-person or phone interviews and an online survey. The Evaluation Directorate also acknowledges the time and effort on the part of BCARP staff given to the evaluation team.

ACRONYMS


List of acronyms
AEO Aquaculture Environmental Operations
AMAC Aquaculture Management Advisory Committee
AQUIIS Aquaculture Integrated Information System
ARM Aquaculture Resource Management
BC British Columbia
BCARP British Columbia Aquaculture Regulatory Program
C&P Conservation and Protection
CFIA Canadian Food Inspection Agency
DFO Fisheries and Oceans Canada
ENGO Environmental Non-Governmental Organization
FN First Nations
FTE Full-time equivalent
GDP Gross domestic product
IMAP Integrated Management of Aquaculture Plans
IM/IT Information Management/Information Technology
NHQ National Headquarters
O&M Operations and Maintenance
PAR Pacific Aquaculture Regulations
SEP Salmonid Enhancement Program

EXECUTIVE SUMMARY


INTRODUCTION

This report presents the results of the evaluation of the British Columbia Aquaculture Regulatory Program (BCARP). The evaluation was conducted by DFO’s Evaluation Directorate in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation.  The main objective of this evaluation is to determine if the program is still relevant and the extent to which BCARP is managed efficiently and has achieved its stated outcomes. The program operates almost entirely in the Pacific region but has a few staff members at National Headquarters (NHQ) in Ottawa. The evaluation covers a four-year period from December 2010 to March 2015. Recommendations stemming from the main findings were developed to support improvements to the program and to inform future decision-making.

PROGRAM PROFILE

BCARP was implemented by Fisheries and Oceans Canada (DFO) in December 2010 after a British Columbia Supreme Court Decision ruled that finfish aquaculture is a fishery and therefore falls under the exclusive jurisdiction of the federal government. The federal government included shellfish and freshwater aquaculture within its new responsibilities but not aquatic plant cultivation. The BC Supreme Court decision gave rise to the Pacific Aquaculture Regulations as an annex of the Fisheries Act.

In 2014-15, BCARP comprises 50 employees and an annual budget of 5.6 million dollars. Aquaculture Programs, Aquaculture Resource Management and Aquaculture Environmental Operations are three key groups of BCARP. Together these groups ensure program governance, the development and implementation of conditions of licences as well as fish health and environmental monitoring. BCARP also has a dedicated Conservation and Protection unit to oversee compliance of aquaculture operators with the Fisheries Act and the Pacific Aquaculture Regulations.  Finally, BCARP has two staff in the Pacific Region’s Communications Directorate and four staff at National Headquarters.

EVALUATION METHODOLOGY

This evaluation adopted non-experimental approach whereby program performance was measured with a logic model combined with multiple lines of evidence to demonstrate the extent to which the program is achieving issues of relevance, effectiveness, efficiency and economy.  The evaluation questions were determined on the basis of the Treasury Board’s Policy on Evaluation, a review of key program documents, and results from preliminary discussions with key program personnel.

The following lines of evidence were used for the evaluation: document review; key informant interviews (n=56) with senior management and program personnel as well as external stakeholders; site visits throughout Vancouver Island; and two online surveys which were done jointly with the evaluation of the Salmonid Enhancement Program.

A number of methodological limitations were encountered such as: availability of performance data and program documentation, survey responses, timing of surveys and site visits, all of which were mitigated.

EVALUATION FINDINGS AND RECOMMENDATIONS

RELEVANCE

The British Columbia Aquaculture Regulatory Program was implemented by DFO in December 2010 following a BC Supreme Court decision which ruled that finfish aquaculture is a fishery and therefore falls under federal jurisdiction. BCARP is needed as it regulates and ensures that BC aquaculture industry is compliant with regulations and legislation which contribute to the viability and sustainability of the BC aquaculture industry. BCARP aligns with two DFO strategic outcomes: “Economically Prosperous Maritime Sectors and Fisheries” and “Sustainable Aquatic Ecosystems”. BCARP also aligns to the whole of government framework priority of “Strong Economic Growth”. BCARP derives its mandated activities from four pieces of legislation: Fisheries Act, Oceans Act, Pacific Aquaculture Regulations and Fishery (General) Regulations.

EFFECTIVENESS

There are three sectors of aquaculture in BC – marine finfish (mostly salmon), shellfish (mostly oysters and clams) and freshwater/land-based. In 2013, 89% of BC aquaculture production was salmon. The shellfish and freshwater sectors represented around 11% of BC aquaculture production and do not present the same level of risks as the marine finfish sector. Due to higher risks associated with the marine finfish sector and public concern, BCARP’s efforts have been concentrated on this sector.

The evaluation assessed the extent to which the program has achieved its intended outcomes. Overall, BCARP has made good progress on most immediate and intermediate outcomes. Long term outcomes were not assessed since the program has been in existence for only four years.

The evaluation found that tools and advice provided by Aquaculture Programs are used to support decision-making. Public reporting has improved since BCARP undertook the management and regulation of the aquaculture industry in BC. Information on BC aquaculture sites is available and accessible to the public. This publicly accessible information contributes to transparency and accountability in DFO’s management of the aquaculture sector in British Columbia.

BCARP took over the responsibilities of regulating and managing the aquaculture in British Columbia from the province in December 2010. BCARP was successful in replacing aquaculture licences that were previously issued by the Province of British Columbia. Licenced operators have, for the most part, the necessary information, oversight and guidance required for their operations.

BCARP’s contribution to Aquaculture industry growth and economic opportunities was limited due, in part, to external factors, such as a moratorium on issuing new marine finfish licences and the appreciation of the Canadian dollar.

Conservation and Protection efforts have been mostly allocated to the marine finfish sector. Non-compliance issues are addressed and the marine finfish sector is compliant with regulations and legislation for the most part. As more Conservation and Protection resources shift to the shellfish and freshwater sectors, a more comprehensive understanding of compliance and/or gaps in these sectors will be available.

Finally, BCARP is informed about the health of BC aquaculture stock and the environmental performance of sites. Fish health inspections have covered every active marine finfish facility (100%) during a three-year period. Environmental monitoring also occurs for shellfish and freshwater sites – approximately 10% of the active sites in each of these sectors are inspected annually. BCARP’s Aquaculture Environmental Operations do not play a significant role in notifying the Canadian Food Inspection Agency of reportable diseases.

EFFICIENCY AND ECONOMY

BCARP demonstrates efficiencies but certain improvements could be made in terms of streamlining the decision making process, human resources allocation, improving information management/information technology, clarification of roles and responsibilities as well as gathering performance data.  

BCARP will introduce multi-year licences for federally regulated aquaculture operations in British Columbia that should save time, money and promote industry growth. Changes to conditions of licence have become less frequent as the conditions have stabilized over the last few years. Further streamlining the decision making process would help minimize costs for the review of licence applications.

There are human resources allocation challenges that were observed by the evaluation team. For example, staff tend to work in silos (within their activity areas and by aquaculture sector) and could be more versatile. This combined with an uneven work balance in some activity resulted in inefficiencies. As the program evolved, the alignment of staff with respect to current program priorities and activities and upcoming changes should be examined to ensure optimal allocation of resources. The importance of having versatile employees in a program as diverse as BCARP would be a clear advantage.

There is duplication of staff time and effort spent with respect to data management. The Aquaculture Integrated Information System (AQUIIS) that was developed for BCARP is continuing to evolve to better meet program needs.

Strong coordination between National Headquarters and BCARP would: foster engagement of all stakeholders; streamline decisions; and secure buy-in across DFO sectors. The Service Level Agreement between NHQ and BCARP, which expired in 2014, could be re-examined to address these issues.

Finally, BCARP should ensure that it has the appropriate performance measures in place to assist in program decision-making and accountability.

RECOMMENDATIONS

Recommendation 1: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, reassess BCARP’s human resources allocation to ensure alignment of staff with respect to current program priorities and activities and upcoming changes.

Recommendation 2: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, develop BCARP’s Performance information tools (refined logic model and develop performance indicators). Once the tools are developed, ensure that data is collected and reported consistently.

Recommendation 3: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, develop an agreement for BCARP between Ecosystems and Fisheries Management and the Pacific Region.  The agreement should clarify roles and responsibilities between National Headquarters and the Pacific Region related to improving the AQUIIS database as well as increasing engagement of internal and external committees, streamlining decision-making and strengthening coordination and buy-in across sectors.

1. INTRODUCTION


1.1 PURPOSE OF THE EVALUATION

This report presents the results of the evaluation of the British Columbia Aquaculture Regulatory Program. The evaluation was conducted by Fisheries and Oceans Canada’s Evaluation Directorate in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation, which requires all direct program spending be evaluated every five years. The evaluation commenced in 2013, in accordance with the multi-year Departmental Evaluation Plan, and was completed in June 2015. Recommendations stemming from the main findings were developed to support improvements to the program and to inform future decision-making.

1.2 SCOPE

The main objective of this evaluation is to determine if the program is relevant, the extent to which the program is managed efficiently and has achieved its intended outcomes. The evaluation covers a four-year period from program’s inception in December 2010 through to March 2015. The evaluation is inclusive of both the Pacific Region and the National Capital Region. As the program is less than five years of age, the evaluation examined the design and delivery of the program and focused on assessing immediate and intermediate outcomes as well as efficiency and economy.

1.3 REPORT STRUCTURE

The executive summary provides a brief outline of the evaluation findings, conclusions and recommendations. Section 2 presents an overview of the British Columbia Aquaculture Regulatory Program. Section 3 describes the evaluation methodology, followed by a discussion of the main findings in section 4 and conclusions and recommendations in section 5. Annex A presents the Management Action Plan. Finally, the evaluation matrix, in Annex B, presents the key issues investigated (relevance, effectiveness, efficiency and economy), the specific evaluation questions addressed and the corresponding lines of evidence used for each question.

2. PROGRAM PROFILE


2.1 PROGRAM CONTEXT

The British Columbia Aquaculture Regulatory Program (BCARP) was implemented in December 2010. BCARP is a program within Fisheries and Oceans Canada (DFO) that manages, administers and regulates aquaculture in British Columbia and governs the activities of the aquaculture industry in that province, including finfish, shellfish and freshwater/land-based operations. Prior to BCARP, it was the Province of British Columbia that regulated aquaculture activities.

BCARP is a Sub-Activity in DFO’s Program Alignment Architecture located under the Strategic Outcome of Economically Prosperous Maritime Sectors and Fisheries, and under the Program Activity of Sustainable Aquaculture Program. BCARP budget allocation in 2014-15 was 5.6 million with 50 Full-time equivalents (FTEs).

BCARP was developed by DFO in response to a February 2009 British Columbia Supreme Court decision which held that the activity of finfish aquaculture is a fishery and falls under the exclusive jurisdiction of the federal government. While the Court case focused on finfish, the federal government’s position is that the federal jurisdiction applies equally to shellfish and freshwater aquaculture operations with the exclusion of aquatic plant cultivation1. Specifically, the program is designed to implement federal regulations under the Fisheries Act (Pacific Aquaculture Regulations) and to ensure that aquaculture in British Columbia (BC)  is sustainable and is conducted in a manner that minimizes the risks to wild fish stocks. Aquaculture is a fast growing segment of the fishing industry and one of the best potential sources for expanding economic growth and prosperity in coastal and rural Canada.


1  http://www.parl.gc.ca/content/sen/committee/412/POFO/15EV-51771-E.HTM

2.2 PROGRAM ACTIVITIES

BCARP encompasses a Regional Aquaculture Coordination Office (formed by three groups: Aquaculture Programs, Aquaculture Resource Management and Aquaculture Environmental Operations); a dedicated Conservation and Protection Unit; staff within the Pacific region Communications Directorate; and staff working within the National Capital Region. BCARP does not have dedicated staff for licencing. A licensing agent, who reports to the Regional Director of Fisheries Management, is responsible for issuing aquaculture licences.

Activities, which are described below, are organised under five main categories: 1) Aquaculture Programs; 2) Aquaculture Resource Management; 3) Aquaculture Environmental Operations; 4) Aquaculture Conservation and Protection; and 5) Communications.

Aquaculture Programs

The Aquaculture Programs Group, located in Vancouver, conducts work related to program governance, coordination of operational policies, Aboriginal engagement, ecosystem approach, and other region-wide issues. It is also responsible for coordinating advice to external partners regarding aquaculture in British Columbia.

Aquaculture Programs also plays a lead role in Information Management Systems as BCARP provides stakeholders and the general public with information and data on the environmental and operational performance of the aquaculture industry in BC. To ensure the availability of data, information reported by the industry on a quarterly or annual basis is generally released three months after it is collected. When it is possible and appropriate to inform the public on a more immediate basis (i.e., event-initiated reports such as fish escapes), data is released as soon as possible.

The Aquaculture Programs ensures liaison with scientific research on aquaculture undertaken by the DFO Sustainable Aquaculture Science Program with, in some cases, collaboration with industry. Finally, Aquaculture Programs is responsible for two Aquaculture Management Advisory Committees (AMAC), one for marine finfish and one for shellfish.

Aquaculture Resource Management

Aquaculture Resource Management (ARM) staff are located in Nanaimo and Port Hardy. ARM is responsible for developing Integrated Management of Aquaculture Plans (IMAPs). IMAPs ensure that aquaculture operations have an operational framework which complies with the Pacific Aquaculture Regulations and other federal government legislations and regulations. In other words, Aquaculture Resource Management is responsible for providing the framework for licensing aquaculture operations.

In accordance with the Pacific Aquaculture Regulations, Conditions of Licence were developed for each aquaculture sector (i.e., marine finfish, shellfish and freshwater). These conditions clearly state the obligations of each licence holder and prescribe standards and protocols for the operation of aquaculture facilities. Licence holders must record and report data and information to BCARP in several categories such as fish health, facility operations, marine mammal interactions and fish escapes.

Aquaculture Resource Management also conducts work related to operational procedures and undertakes consultations and engagement with First Nations and other stakeholders when licensing aquaculture operations.

Aquaculture Environmental Operations

Aquaculture Environmental Operations (AEO) staff are located in Courtenay and Campbell River and include: biologists, technicians and veterinarians who assess aquaculture projects and monitor fish health and environmental issues. AEO has its own laboratory in Courtenay for sample testing purposes. AEO verifies that Pacific Aquaculture Regulations are being upheld, and that aquaculture facilities are abiding by certain aspects of their conditions of licence2. AEO staff conduct a number of audit, surveillance and monitoring activities to evaluate whether the aquaculture industry is operating in a sustainable manner. The data gathered also provides valuable insights into the environmental and operational performance of the aquaculture industry in British Columbia. Akin to industry, AEO notifies the Canadian Food Inspection Agency (CFIA) of potential incidences of reportable diseases.

Audit, surveillance and monitoring activities include the following:

  • Fish health sampling;
  • Sea lice audits;
  • Atlantic Salmon Watch stream surveys;
  • Ocean bottom benthic habitat sampling; and
  • Marine Finfish, Shellfish and Freshwater site inspections.

AEO inputs and analyzes the data collected from its audit, surveillance and monitoring activities, as well as data submitted by industry.  Quarterly and annual reports are prepared by AEO and submitted to Communications for formatting, translation and posting on BCARP internet site.  Examples of the data that is reported by AEO include: sea lice audits by site; benthic performance; incidental catch; escapes and fish health management plan (FHMP) compliance farm level.

AEO also ensures constant communications and feedback to industry as well as providing advice to ARM regarding licence applications and amendments. Finally, AEO is sometimes called to collaborate with other DFO scientific directorates in order to undertake its work.

Conservation and Protection

DFO’s Conservation and Protection Directorate (C&P) received funding dedicated to aquaculture in British Columbia. A total of 12 FTEs, including eight fishery officers stationed in Campbell River and Nanaimo, are responsible for aquaculture enforcement. The area chief for aquaculture leads the program and reports to the regional director of C&P. The primary role of this C&P unit is enforcing the compliance of aquaculture operators with the Fisheries Act and the Pacific Aquaculture Regulations. DFO fishery officers’ conduct investigations and take enforcement actions based on site visits, information from the public and inspections undertaken by other DFO staff.

Communications

Transparency and public reporting are important priorities for the program. The Pacific Region’s Communications Directorate receives funding dedicated to two FTEs which are responsible for managing BCARP’s public and media relations as well as posting information related to aquaculture facilities on BCARP internet site.

Staff in the National Capital Region

BCARP has four positions in the National Capital Region dedicated to: operational implementation; oversight; performance management; and to support the Aquaculture Information Management System.

Licencing

BCARP does not have any dedicated FTEs for licencing. One licensing agent in the DFO’s South Coast area office (Nanaimo), which reports to the Regional Director of Fisheries Management, is responsible for issuing aquaculture licences. In 2014-15, the number of licences issued was 115 marine finfish, 459 shellfish and 109 freshwater.


2 The remaining aspects of the conditions of licence are verified by Compliance and Protection.

2.3 PROGRAM LOGIC MODEL

A performance measurement strategy was not developed for the British Columbia Aquaculture Regulatory Program. Instead, an overall Performance Measurement Strategy was developed for the Sustainable Aquaculture Program in 2012. For the purpose of this evaluation, a logic model (figure 1) was developed in consultation with key program personnel to identify program activities and outcomes. The logic model covers key activities undertaken by the program. It reflects the five main categories of activities discussed above: 1) Aquaculture Programs; 2) Aquaculture Resource Management; 3) Aquaculture Environmental Operations; 4) Aquaculture Conservation and Protection; and 5) Communications.

Figure 1. British Columbia Aquaculture Regulatory Program Logic Model

British Columbia Aquaculture Regulatory Program Logic Model

2.4 PROGRAM INTERESTS AND STAKEHOLDERS

Different interests and stakeholders are involved in the aquaculture in British Columbia. External partners are as follows:

  • Other Federal Departments including Environment Canada, Western Economic Diversification Canada, Transport Canada which issues approvals to aquaculture affecting navigation and the Canadian Food Inspection Agency which addresses aspects of international and trade-related fish health and food safety;
  • The Province of BC is responsible for issuing land tenures to occupy the seabed (through leasing agreement) where operations take place in either the marine or freshwater environment, licence marine plant cultivation, and manage business aspects of aquaculture such as work place health and safety within the province;
  • First Nations;
  • Industry;
  • Environmental Non-Government Organizations; and,
  • Conservation Non-Governmental Organizations

2.5 GOVERNANCE

The implementation of BCARP is under the responsibility of the Regional Director General (RDG), Pacific Region, functionally reporting to the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management. The BCARP budget comes from the Director General of the Aquaculture Management Directorate in Ottawa.

At the operational level BCARP reports to the BC Aquaculture Implementation Governance Committee which serves as the Pacific Region’s primary management forum for strategic oversight and decision-making on the implementation of DFO’s roles, responsibilities and operations with respect to BC aquaculture management. The Directors of Aquaculture Committee is an interagency forum to ensure coordination among involved agencies in aquaculture policy development, planning initiatives, program delivery, communications, consultations, and management of emerging issues.

At the regional level, the Aquaculture Management Committee serves as the Pacific Region’s overall management forum for Federal/Provincial strategic oversight, decision-making, and implementation with respect to the Canada-BC Agreement on Aquaculture Management.  At the regional level, there is also two Aquaculture Management Advisory Committees (AMAC) – one for marine finfish and one for shellfish. These advisory committees bring together industry licence-holders, industry associations, First Nations, environmental organizations, local government, BC government and DFO to discuss high level issues with regards to a specific sector of aquaculture in BC.

Finally, at the national level there is the National Industry Liaison Committee. DFO is also a member of the Strategic Management Committee on Aquaculture which is part of the intergovernmental Canadian Council of Fisheries and Aquaculture Ministers Forum.

2.6 PROGRAM RESOURCES

Table 1. British Columbia Aquaculture Regulatory Program Funding (Budget and Expenditures) and FTEs

Budget and Expenditures
BCARP Funding and FTEs          
  2010-11 2011-12 2012-13 2013-14 2014-15
Budget (millions of dollars) $10.2M $6.9M $6.0M $6.3M $5.6M
Total Expenditures Salaries and Wages $1.6M $4.2M $4.0M $4.3M $4.3M
Total Expenditures O&M $8.0M $3.1M $1.8M $1.1M $1.1M
Total Expenditures (millions of dollars) $9.6M $7.3M $5.8M $5.4M $5.4M
Total FTEs 18.52 52.97 51.38 47.49 50.11

Source: BCARP administrative data

2.7 RISK PROFILE

BCARP developed a program risk profile in March 2013. This exercise identified the six mission critical risks which are listed in table 2.

Table 2. BCARP Mission Critical Risks

Budget and Expenditures
Key Risk Risk Statement
1. Information Management Systems Information management systems and processes will not be in place or will not be designed to meet the needs of the program.
2. Information for Decision-Making and Reporting Decision-makers will not have timely access to the information necessary to inform decisions and reporting requirements.
3. Government�s Response to Cohen The federal response to the Cohen report may impact the ability of BCARP to deliver on the program objectives and outcomes.
4. Human Resources BCARP may not have sufficient human resource (HR) capacity to deliver on program objectives and outcomes.
5. Science Capacity DFO will not have the science capacity to support timely, science-based decision-making under BCARP.
6. Litigation Litigation pressures will impact the ability of BCARP to deliver on program objectives and outcomes.

3. METHODOLOGY


3.1 SCOPE

The evaluation of BCARP covered a four-year period from the program’s inception in December 2010 through to March 2015. The work was undertaken by DFO’s Evaluation Directorate between December 2013 and May 2015. The evaluation covered both the operations and management in the Pacific region as well as key staff members related to BCARP in the National Capital Region. Directors and managers of the program assisted throughout the evaluation. In addition to participating in interviews, they provided documents and statistical data, identified other interviewees and stakeholders to contact, and provided feedback to the evaluation team on the logic model and the evaluation report.

3.2 EVALUATION APPROACH AND DESIGN

A non-experimental design was chosen. The evaluation used a logic model combined with multiple lines of evidence drawing on both qualitative (interviews, program documents) and quantitative data (administrative data, survey). The various sources of input provided the basis for corroborating the evidence (triangulation) to arrive at valid findings and conclusions. The logic model used for this evaluation was developed by the Evaluation Directorate in concert with the program.

3.3 KEY ISSUES AND EVALUATION QUESTIONS

The evaluation questions covered relevance and performance; the latter encompass effectiveness, efficiency and economy. The evaluation questions were determined on the basis of the Treasury Board’s Policy on Evaluation (2009), a review of key program documents, and results from preliminary discussions with key program personnel. Annex A provides the detailed Evaluation Matrix.

3.4 DATA SOURCES

The evaluation drew on the following lines of evidence to gather data:

DOCUMENT REVIEW

A review of documents was conducted to assess most evaluation issues and to develop a thorough understanding of BCARP. The document review included government-wide documents such as federal legislation, regulations and policies. Other information reviewed included program administrative data, program documents, and previous evaluations related to aquaculture.

KEY INFORMANT INTERVIEWS

Key informant interviews helped in gaining a better understanding of the perceptions and opinions of individuals (senior management, program personnel and external stakeholders) involved with BCARP. The interviews contributed to many of the evaluation questions, such as relevance, the extent to which BCARP has achieved its results, and suggestions were provided for improving the efficiency and economy as well as the design and delivery of the program. Stakeholder interviewees were chosen through snowball sampling. In sociology and statistics research, snowball sampling is a non-probability sampling technique where existing study subjects recruit future subjects from among their acquaintances.

A total of 56 individuals were interviewed individually or in small group interviews (2-3 persons).

  • DFO senior management (NHQ and Pacific Region (n=12))
  • BCARP Staff (n=32)
  • External stakeholders (n=12)

Separate interview guides were developed and tailored to each group. Interviews were conducted in-person or by telephone and took approximately 60 minutes. Interviews consisted primarily of open-ended questions with a few ranking questions. Interviews were conducted in the official language chosen by the participants.

SITE VISITS

The evaluation team conducted extensive site visits throughout Vancouver Island (Nanaimo, Campbell River, Courtenay, Comox and Port McNeill) and at the Vancouver Head office at the end of April and early May 2014. Program representatives accompanied the evaluation team during these visits. These visits contributed to understanding stakeholder and employee perceptions, opinions and experiences with the program.

SURVEY

The BCARP evaluation capitalized on two surveys conducted for the evaluation of the Salmonid Enhancement Program (SEP) which included certain questions related to BCARP. These questions addressed efficiency issues notably the governance and management aspect of BCARP. A disclaimer was included informing the respondents that the information could be used for the BCARP evaluation.

Two online surveys were conducted, one with SEP employees and one with SEP external stakeholders. Surveys were available in English, as the program is delivered exclusively in the Pacific Region. The surveys were launched in July 2014 and closed in September 2014. The survey tools were pre-tested by the evaluation team.  A notification e-mail was sent to all participants prior to the launch of the surveys and two reminders were sent out closer to the closing date.

Overall, the SEP employee survey had a response rate of 42.9%. A total of 90 surveys were completed (of which 64 employees responded to BCARP questions) out of the possible 210 staff members invited to participate in the survey.    The stakeholder survey had a response rate of 26.5%. A total of 95 surveys were completed (of which 51 stakeholders responded to BCARP questions) out of the possible 359 stakeholders invited to participate in the survey.

The SEP employee survey had a response rate of 42.9% with a confidence level of 95% and a margin of error of +/- 7.9% (19 times out of 20). The stakeholder survey had a response rate of 26.5% with a confidence level of 95% and a margin of error of +/- 8.6% (19 times out of 20). 

3.5 METHODOLOGICAL LIMITATIONS AND MITIGATION STRATEGIES

Although the evaluation encountered some challenges and limitations that are outlined below, these limitations were mitigated, as much as possible, through the use of multiple lines of evidence and triangulation of data. This approach was taken in order to demonstrate reliability and validity of the findings and to ensure that conclusions and recommendations are based on objective and documented evidence.

Survey Responses

Two surveys were conducted for the evaluation of the Salmonid Enhancement Program and included some questions related to BCARP. Survey results provided insight only from SEP employees and SEP stakeholders.  Therefore, aquaculture facilities operators which are not affiliated to SEP, such as commercial aquaculture operations producing marine finfish and shellfish, were not included in the survey. The evaluation mitigated this aspect by gathering stakeholder views and opinions through interviews.

Timing of Surveys

Both SEP employee and SEP stakeholder surveys were conducted between the months of July and September, a period where many employees are away on annual vacation and when stakeholders are busy working in the field.  In order to ensure as high a response rate as possible, a notification e-mail was sent to all potential respondents (employees and stakeholders) in June prior to the survey launch; and during the implementation, two reminders were also sent.  In addition, the survey was left open for several weeks, allowing enough time for respondents to participate.

Site visits

The selection of site visits was based on the possibility for the evaluation team to get exposure to different aspects of BCARP activities. Weather conditions affected the planned itinerary and as a result some locations could not be visited. This was mitigated by BCARP staff demonstrating to the evaluators how the equipment functions and showing videos taken from previous visits.

Availability of Performance Data and Program Documentation

Although the program collects large amounts of information, the data was not readily accessible in a way that aligns with outcomes.  Very little performance data was gathered by the program and did not cover the entire range of program activities. The evaluation did not find a program logic model or a performance measurement strategy in place. In addition, the program’s approach to reporting data through annual Departmental Performance Reports changed from year to year, making comparisons difficult. To mitigate this issue, the evaluation team collected data from various sources in order to help assess the results of the program.

4. MAJOR FINDINGS


This section combines information from all lines of evidence used and presents the findings according to the broad evaluation issues of relevance and performance, the latter covers effectiveness, efficiency and economy.

4.1 RELEVANCE

There are three aspects covered under the issue of relevance:

  • Is there a continued need for BCARP?
  • To what extent is BCARP aligned with Government of Canada and DFO priorities?
  • Is BCARP aligned with federal roles and responsibilities?

Is there a continued need for BCARP?


Findings: BCARP is needed as it regulates and ensures that BC aquaculture industry is compliant with regulations and legislation which contribute to the viability and sustainability of the aquaculture industry.


Aquaculture is the fastest growing segment of the global fishing industry and one of the best potential sources for expanding economic growth and prosperity in coastal and rural Canada. BCARP, which regulates the aquaculture industry in British Columbia, contributes to its viability and sustainability.

Key informants indicated that the program is ensuring that the BC aquaculture industry is in compliance with the Fisheries Act and the Pacific Aquaculture Regulations. These are in place to ensure the industry is managed responsibly and sustainably. As well, respondents mentioned that BCARP contributes to ensuring the conservation and protection of fish and fish resources in BC.

BCARP is also responsive to the economic needs of the industry, stakeholders and First Nations. The aquaculture industry accounted in 2013 for approximately $500 million in Gross Domestic Product (GDP) for BC, which represents 0.25% of BC’s GDP. Figure 2 shows that BC represented 49% of the total Canadian aquaculture production.

Figure 2. Total Aquaculture Production by Province in 2013

total aquaculture production by province 2013

Source: Statistics Canada, Aquaculture Statistics, 2013, Catalogue no. 23-222-X

The contribution of BCARP in the BC economy is highlighted by a PricewaterhouseCoopers LLP report prepared for the province of BC in 20093 which found that aquaculture in BC provides an estimated 6,000 direct and indirect jobs. Similar numbers were presented in a DFO study4 which reported that these direct and indirect jobs generated over $223 million in labour income. DFO’s Departmental performance reports emphasised that sustainable fisheries and aquaculture make significant contributions to Canada’s economy.

In addition, the aquaculture industry is important for Aboriginals in British Columbia. Aboriginal people represent 14.1% of the labour force in the BC aquaculture industry and earn 12.8% of the total labour income5. As the total population of people of Aboriginal ancestry in BC was just over 6% in 2011, there is evidence that aquaculture is an industry which is an important source of employment for Aboriginals.   

Finally, key informants indicated that BCARP helps the aquaculture industry to be viable in terms of ensuring sustainable aquatic ecosystems and providing transparency to the public while enabling market access for seafood products in the European and Asian markets.

Figures 3 and 4 demonstrate that the production of aquaculture in the province of BC is by far the largest in Canada in terms of finfish and second in terms of shellfish. More specifically in BC, salmon represented almost 90% of the aquaculture production with oysters and clams accounting for almost all of the rest of the production in 2013.


3 Cohen Commission of Inquiry into the Decline of Sockeye Salmon in the Fraser River, Volume 1.

4 Socio-Economic Impact of Aquaculture in Canada (2010).

5 A report prepared by the Aboriginal Aquaculture Association in 2011.

Figure 3. Marine Finfish Aquaculture Production by Province (2009 to 2013)

Marine Finfish Aquaculture Production by Province (2009 to 2013)

Source: Statistics Canada, Aquaculture Statistics, 2013, Catalogue no. 23-222-X

Figure 4. Shellfish Aquaculture Production by Province (2009 to 2013)

Shellfish Aquaculture Production by Province (2009 to 2013)

Source: Statistics Canada, Aquaculture Statistics, 2013, Catalogue no. 23-222-X

To what extent is BCARP aligned with Government of Canada and DFO priorities?


Findings: BCARP is aligned with Government of Canada and DFO priorities.


Evidence supports BCARP’s contribution to Government of Canada and DFO priorities through the Budget, Speech from the Throne, and alignment with departmental strategic outcomes of “Economically Prosperous Maritime Sectors and Fisheries” and “Sustainable Aquatic Ecosystems”.

Activities undertaken by BCARP are aligned with Government of Canada priorities as demonstrated through the Budget and Speech from the Throne. For example, the 2013 Speech from the Throne indicated that the Government of Canada will continue to support fisheries by, amongst other initiatives, opening markets worldwide.

BCARP’s activities also contribute to the Government of Canada outcome area of “Strong Economic Growth”. As previously referenced, aquaculture in BC contributes half a billion to British Columbia’s GDP, provides an estimated 6,000 direct and indirect jobs and generates over $223 million in labour income6.

The program directly supports DFO Strategic Outcomes of “Economically Prosperous Maritime Sectors and Fisheries” and “Sustainable Aquatic Ecosystems” by ensuring that the aquaculture industry in BC operates in a sustainable manner, as to protect the marine environment for current and future needs.

Finally, the program also aligns with DFO priorities in terms of aquaculture where the focus is on sustainable production, diversification, market access and improving the predictability of regulations.


6 A 2009 PricewaterhouseCoopers LLP report; and the 2010 Socio-Economic Impact of Aquaculture in Canada.

Is BCARP aligned with federal roles and responsibilities?


Finding: BCARP’s federal role stems from the 2009 BC Supreme Court decision stating that finfish aquaculture is a fishery which falls under federal jurisdiction. BCARP derives its mandated activities from four pieces of legislation: Fisheries Act, Oceans Act, Pacific Aquaculture Regulations and Fishery (General) Regulations.


The regulatory mandate of BCARP is mainly derived from four pieces of legislation; the Fisheries Act, the Oceans Act, Pacific Aquaculture Regulations and Fishery (General) Regulations.

The Pacific Aquaculture Regulations, which is an annex of the Fisheries Act pursuant to section 43 of that Act, stems from a February 2009 British Columbia Supreme Court decision (Morton v. British Columbia (Agriculture and Lands)7) which determined that finfish aquaculture is a fishery in BC and therefore falls under the exclusive jurisdiction of Parliament under s.91 (12) of the Constitution Act, 1867.  The Pacific Aquaculture Regulations lay out the regulatory and management approaches that replace those previously enforced by the Province of British Columbia.

While the court case focused on finfish, the federal government’s jurisdiction applies equally to shellfish and freshwater aquaculture operations with the exclusion of aquatic plant cultivation8. The Pacific Aquaculture Regulations came into force on December 18, 2010. Since then, BCARP is assuming the management and regulations of aquaculture in BC to ensure that the aquaculture industry in BC operates in a sustainable manner, so that the marine environment is protected for future generations.


7 http://www.courts.gov.bc.ca/jdb-txt/SC/09/01/2009BCSC0136err1.htm

8 http://www.parl.gc.ca/content/sen/committee/412/POFO/15EV-51771-E.HTM

4.2 EFFECTIVENESS

The effectiveness section examines the extent of achievement of the program’s immediate and intermediate outcomes. Long-term outcomes were not assessed since the program only began in 2010.

The program is active in five main areas (see logic model in Section 2.4):

  1. Aquaculture Programs;
  2. Communications;  
  3. Aquaculture Resource Management;
  4. Aquaculture Conservation and Protection; and
  5. Aquaculture Environmental Operations.

For clarity purposes, the effectiveness section is organised by these areas, even though, in some cases, multiple activity areas may contribute to one outcome.

There are three sectors of aquaculture in BC – marine finfish (mostly salmon), shellfish (mostly oysters and clams) and freshwater/land-based. In 2013, 89% of BC aquaculture production was salmon. The shellfish and freshwater sectors do not present the same level of risks as the marine finfish sector. Due to higher risks associated with the marine finfish sector and public concern, BCARP’s efforts have been concentrated on this sector.

AQUACULTURE PROGRAMS


Finding: Tools and advice provided by Aquaculture Programs are used to support decision-making. Public reporting has improved since BCARP undertook the management and regulation of the aquaculture industry in BC.


The Aquaculture Programs unit is generally achieving its immediate outcome – Tools and advice are used in management and decision-making. Policies, procedures and advice provided by Aquaculture Programs are used to support decision-making. As the program evolves, there are gaps that still exist in terms of policies and advice. For example, policies/frameworks for new aquaculture species, like geoduck and sea cucumbers, are at different stages of development. In terms of advice, key informants noted that there could be better linkages and engagement between DFO Science and BCARP.

With respect to the intermediate outcome – Transparency and accountability in DFO’s management of the aquaculture sector in BC – the Aquaculture Programs unit has made good progress. On the whole, stakeholders agreed that public reporting has improved under BCARP. However, there seems to be divergent views in terms of timeliness and frequency of reporting between program staff on one hand, and industry and ENGO’s on the other. From a program perspective, quarterly reporting allows a good balance between transparency and reporting burden. However, from an industry and ENGO perspective, a higher frequency of reporting was deemed to be useful. For example, from an industry perspective, certification requirements necessitate posting of public reports within 30 days.   

Over the last five years, the Aquaculture Programs unit has improved the working relationship with ENGOs. At the start of the program there was resistance from ENGOs in being engaged by the BCARP. Many ENGOs did not attend the Marine Finfish Aquaculture Management Advisory Committee (AMAC) meetings or participated only as observers. Lately, with possible modifications to the terms of reference of the AMAC being considered, ENGOs are now reconsidering their position.

COMMUNICATIONS


Finding: Information on BC aquaculture sites is available and accessible to the public. This publicly accessible information contributes to transparency and accountability in DFO’s management of the aquaculture sector in British Columbia.  


Communications achieved its immediate outcome – Information on BC aquaculture sites is available and accessible to the public – through DFO’s Pacific Region internet site.  

This is attributable in part to efforts made by Aquaculture Environmental Operations and Communications that ensured that information on BC aquaculture sites is available and accessible to the public.

Public information available includes: Benthic Monitoring; Compliance Assessment Activities; Escapes; Introductions and Transfers; Fish Health Management; Incidental Catch; Marine Mammal Interactions; Salmon Egg Imports; Sea Lice; and Use of Lights9.

This publicly accessible information contributes to the intermediate outcome – Transparency and accountability in DFO’s management of the aquaculture sector in British Columbia.


9 http://www.pac.dfo-mpo.gc.ca/aquaculture/index-eng.html

AQUACULTURE RESOURCE MANAGEMENT


Finding: Licenced operators have for the most part the necessary information, oversight and guidance required for their operations. BCARP’s contribution to Aquaculture industry growth and economic opportunities was limited due, in part, to external factors.


Aquaculture Resource Management (ARM) staff have put in place procedures for licensing, consultations and engagement as well as procedures and operational policies for the preparation of Integrated Management of Aquaculture Plans. There are still gaps in terms of procedures for resource management, for example, procedures for certain shellfish and freshwater aquaculture policies still need to be developed.

Aquaculture Resource Management and Aquaculture Environmental Operations staff are working toward achieving the immediate outcome – Licenced operators have the necessary information, oversight and guidance required for their operations. The achievement of this outcome is well advanced for the marine finfish sector, but not to the same extent for the shellfish and freshwater sectors. Aquaculture Conservation and Protection staff are also contributing to this outcome through site inspections and education.

BCARP has taken a while to adapt to the shellfish sector’s specificities such as the wide variety of operations ranging from large companies to small family run operations which create difficulties in providing guidance and oversight. Conditions of licence are sometimes not applicable for all type of shellfish aquaculture operations. As well, the shellfish sector holds the largest number of licences (average of 462 licences issued in the last five years).

Some progress in the last four years has been made in the freshwater sector. Freshwater hatcheries (mostly salmon), operated by DFO’s Salmonid Enhancement Program (SEP) employees as well as small hatcheries operated by volunteers have benefited from guidebooks prepared by the program to ensure compliance of these facilities with the Pacific Aquaculture Regulations. Survey respondents expressed their satisfaction with the guidebooks and mentioned that, as a result, they are more vigilant with respect to fish health protocols. 

Resource Management was successful in issuing aquaculture licences when BCARP became responsible for regulating and managing aquaculture in BC in December 2010. Licences have to be issued every year except for freshwater licences which are issued every two years. Since BCARP’s inception, aquaculture operators do not have to pay a fee to hold a licence.

As shown in figure 5, the number of aquaculture licences in BC has been relatively stable over the last five years. This is as a the result of a decision by DFO to not allow any production increases as well as a moratorium on issuing new marine finfish licences. The moratorium and production limitations were lifted in October 2013 except in the Discovery Islands area where the moratorium will be maintained until September 2020. It should be noted that shellfish licences were issued for a year and a quarter in December 2010, which explain the absence of shellfish licences issued in 2011-12. As well, no freshwater licences were issued in 2011-12 and in 2013-14 since freshwater licences are issued every two years.

Figure 5. Number of Aquaculture Licences Issued by BCARP, by Sector (Marine Finfish, Shellfish and Freshwater) from 2010-11 to 2014-15

Aquaculture Licences Issued by BCARP, 2010-11 to 2014-15

Source: BCARP administrative data

BCARP’s contribution to Aquaculture industry growth and economic opportunities was limited due, in part, to external factors. BCARP fosters aquaculture industry growth by providing regulations and enforcement which contribute to public confidence. The “federal brand” is more visible than a provincial one and therefore was beneficial in terms of product recognition in export markets. There was no consensus amongst key informants with regards to the role of DFO for market access of BC aquaculture products. ENGOs perceive that DFO is in a position of conflict of interest being both the regulator and promoter of aquaculture growth.

BC represents half of the aquaculture production in Canada.  The value of BC aquaculture has stayed fairly constant since 2009 (see figure 6).

Figure 6. British Columbia Aquaculture Value, 2009 to 2013

British Columbia Aquaculture Value, 2009 to 2013

Source: Statistics Canada, Aquaculture Statistics, 2013, Catalogue no. 23-222-X

Opportunities for industry growth can be observed by the number of requests for new aquaculture licences and for amendments to existing licences in 2013-14 and 2014-15 (see figures 7 and 8).

Figure 7. Number of Requests for New Aquaculture Licences by Sector (Marine Finfish, Shellfish and Freshwater) in 2013-14 and 2014-15

Requests for New Aquaculture Licences by Sector

Source: BCARP administrative data

Figure 8. Number of Requests for Aquaculture Licence Amendments by Sector (Marine Finfish, Shellfish and Freshwater) in 2013-14 and 2014-15

Requests for Aquaculture Licence Amendments by Sector

Source: BCARP Administrative Data

Around half of all BC aquaculture exports, led principally by cultured salmon, go to the United States while most of BC shellfish aquaculture products are exported principally to China and Japan. A limitation to growth and economic opportunities was the appreciation of the Canadian dollar between 2011 and 2013, especially when it was above parity with the US dollar. The consequence of the strong currency had a negative impact on exports and sales. Other limitations were DFO’s decision to freeze aquaculture production at existing levels and the moratorium on issuing new marine finfish licences that were in place between December 2010 and October 2013, as a precautionary approach during the Cohen Commission of Inquiry into the Decline of Sockeye Salmon in the Fraser River.

AQUACULTURE CONSERVATION AND PROTECTION


Findings: Non-compliance issues are addressed and the marine finfish sector is compliant with regulations and legislation for the most part.


The Aquaculture Conservation and Protection  (C&P) unit has made good progress related to the immediate outcome – Non-compliance issues addressed, and intermediate outcome – Aquaculture industry is compliant with regulations and legislation for the marine finfish sector. Non-compliance issues between 2011 and 2014 consisted mainly of warnings from Fishery Officers. The number of convictions has been very low, only three, and a constant decrease in the percentage of non-compliant operators between 2011 and 2014 has been observed.

Figure 9 demonstrates a good inspection coverage (percentage of sites that had a general inspection out of total licenced facilities) of marine finfish sites and an increase in coverage in shellfish sites. The freshwater sector is in the very early stages of programming by C&P.  It should be noted that C&P changed its inspection methodology which impacted its statistics. The decreases of coverage in the percentage of marine finfish sites inspected in 2012 and 2014 are due to a change in inspection methodology.

Figure 9. Percentage of Annual Coverage of Aquaculture Sites Covered Under General Inspections, by Sector (Marine Finfish, Shellfish and Freshwater) 2011 to 2014

Percentage of Annual Coverage of Aquaculture Sites

Source: Conservation and Protection Administrative Data

The rigorous biosecurity protocols that were developed by BCARP in order to avoid the cross-contamination of aquaculture facilities have been a challenge in terms of program delivery. The special needs stemming from these biosecurity protocols complicate field work. For example, DFO vessels inspecting facilities are more limited in terms of the number of sites they can visit in one outing. Another example is the impossibility for C&P units to work interchangeably between commercial fisheries and aquaculture. 
   
Figure 10 demonstrates that C&P general inspections are delivered according to the number of planned inspections.  As shown in the graph, the percentage of general inspections completed against planned is around 95% for all three aquaculture sectors. The drop in marine finfish inspections in 2012 is due to 23% of active marine finfish sites being off limits due to biosecurity concerns.  C&P data also showed a 100% rate of complaints investigated against complaints received for both the marine finfish sector and the shellfish sector since 2011.

Figure 10. Percentage of General Inspections Completed Against Planned by Sector (Marine Finfish, Shellfish and Freshwater) from 2011 to 2014

ercentage of General Inspections Completed Against Planned by Sector

Source: Conservation and Protection Administrative Data

Figure 11 shows a sharp decrease in the percentage of non-compliant operators in the marine finfish sector. Indeed, the number of warnings in the finfish sector decreased from 102 in 2011 to eleven in 2014. Although non-compliance issues are addressed for the shellfish sector, the percentage of operators that were non-compliant has remained stable but this is reflective of the increase in the number of inspections in this sector.

Figure 11. Percentage of Non-Compliant Operators, Marine Finfish and Shellfish Sectors between 2011 and 2014

Percentage of Non-Compliant Operators

Source: Conservation and Protection Administrative Data

Key informants were of the opinion that marine finfish operators are aware of their regulatory obligations to a considerable extent, shellfish operators to a moderate extent and freshwater operators to a limited extent. This is due to the fact that C&P efforts have been largely allocated to the marine finfish sector.

Finally, according to key informants, the finfish sector is compliant with regulations and legislation to a considerable extent. As more Conservation and Protection resources shift to the shellfish and freshwater sectors, a more comprehensive understanding of compliance and/or gaps in these sectors will be available.

AQUACULTURE ENVIRONMENTAL OPERATIONS


Findings: BCARP is informed about the health of BC aquaculture stock and the environmental performance of sites. Aquaculture Environmental Operations do not play a significant role in notifying CFIA of reportable diseases.


The Aquaculture Environmental Operations (AEO) group is working toward achieving the immediate outcome – Program informed about the health of BC aquaculture stock and the environmental performance of sites.

There is evidence that fish health inspections have covered every active marine finfish facility during a three-year period (between 2011-12 and 2013-14). In terms of environmental monitoring, AEO undertake three types of inspections for the marine finfish sites: benthic survey; harvest operations; and conditions of licence. These three types of inspections are complementary and help ensure compliance on all facets of the environment aspects of a marine finfish aquaculture licence. There is also evidence of environmental monitoring for shellfish and freshwater sites as approximately 10% of the active sites in each sector are inspected annually. 

Some challenges related to inspecting and monitoring the shellfish sector are the variety of operations and the number of operators. Also, operators are not always present on shellfish production sites, which create difficulties for the monitoring of sites.

Additionally to field work, AEO inputs and analyzes the data collected from its audit, surveillance and monitoring activities, as well as data submitted by industry and prepares quarterly and annual reports that are posted on BCARP internet site. AEO also ensures constant communications and feedback to industry as well as providing advice to Aquaculture Resource Management regarding licence applications and amendments.

The evaluation did note a discrepancy in terms of AEO’s intermediate outcome – CFIA notified of potential incidences of reportable diseases. This intermediate outcome is specific to only one aspect of AEO and is not representative of the array of activities undertaken by AEO. The Canadian Food Inspection Agency (CFIA) has the legal mandate to monitor and manage cases of reportable diseases through the Health of Animals Act. Within this context, evidence collected for this evaluation found that BCARP plays a very minor role in terms of informing CFIA about reportable diseases. In the marine finfish sector, industry does its own testing and reports directly to CFIA. From a key informant perspective, BCARP testing is not seen as having any added value.  In the shellfish sector, the monitoring of reportable diseases is overseen by CFIA through the National Aquatic Animal Health Program. Finally, BCARP does not test freshwater hatcheries for reportable diseases. CFIA does selective sampling of commercial hatcheries to determine if they are disease free.

4.3 EFFICIENCY AND ECONOMY


Findings: BCARP demonstrates efficiencies but certain improvements could be made in terms of streamlining the decision making process, human resources allocation, improving information management/information technology, clarification of roles and responsibilities as well as gathering performance data.  


Since 2010, building on the work done by the government of British Columbia, BCARP developed and refined numerous policies, processes and design and delivery mechanisms, which created efficiencies; for example, reviewing conditions of licence, developing licence applications screening processes, preparing a monitoring and compliance strategy and establishing industry self-reporting requirements.

BCARP will introduce multi-year licences for federally regulated aquaculture operations in British Columbia. From an industry and staff perspective, implementing multi-year licences will save time, money and promote industry growth.

During the first years of the program, conditions of licence changed from year to year. However, changes to conditions of licence have become less frequent as the conditions stabilized in the last few years. Further streamlining the decision making process would help minimize costs for the review of licence applications. For example, are consultations necessary when modifying a licence from round pens to square pens? From a Conservation and Protection perspective, streamlining conditions of licence in order to have the right regulations and good licensing conditions would help reduce inspection costs.

The evaluation team observed that staff tend to work in silos (within their activity areas and by aquaculture sector) and could be more versatile. This combined with an uneven work balance in some activities resulted in inefficiencies. For example, more communications between Aquaculture Environmental Operations and Aquaculture Resource Management would help with the file review during the licence application or review process. Also, the number of requests for new licences or licence amendments has been much more important in the shellfish sector than the marine finfish or freshwater sectors.  If not addressed, silos and lack of versatility will hamper the program’s efforts in terms of proper allocation of human resources to deliver the program efficiently.

There are other human resources allocation issues that should be reviewed as the program priorities and activities have evolved. For example, Communications has been successful in establishing public confidence in aquaculture in BC. Communication needs for BCARP have decreased over the years and is now similar to other DFO Programs. Therefore, perhaps BCARP could rely on the Pacific Region’s Communications Directorate as a typical internal services rather than directly funding two dedicated FTEs in the Pacific Region’s Communications Directorate.

Other upcoming changes with possible impacts in terms of human resources allocation are related to the introduction of multi-year licences and the intended transfer of responsibilities from DFO to the Canadian Food Inspection Agency regarding regulatory activities for interprovincial carrying of live salmonid fish and their eggs to prevent the spread of infectious disease of concern.  Although the pending transfer of responsibilities is well known by both departments, a plan for addressing workload and resourcing has not been developed nor have the timelines surrounding the transfer been clearly determined10.

Although the responsibility for the transfer of responsibilities to CFIA presently sits at a broader level in DFO than the BCARP, it will nonetheless have impacts on its operations. For example, BCARP’s involvement with regards to monitoring fish health and notably auditing for reportable diseases as well as information to be publicly available on BCARP internet site will have to be reviewed to avoid duplication.

Another challenge that BCARP faces is ensuring that the program has the requisite information management and information technology (IM/IT) infrastructure to efficiently deliver the program. Staff noted that the following IM/IT tools are difficult to work with: the Aquaculture Integrated Information System (AQUIIS), Citrix, and internet and cellular network connections unavailable in some areas of Vancouver Island. AQUIIS centralizes and standardizes the Licensing and Compliance Management Processes. AQUIIS was designed and developed for BCARP but it took longer than planned. As well, the functionality of some of AQUIIS components is problematic. The delays encountered in developing AQUIIS resulted in staff developing their own means to capture data, for example with Excel files. Therefore, there is some duplication of staff time and effort spent with respect to data management. There are commitments to continue to improve and upgrade AQUIIS which should become the national DFO database for aquaculture in the coming years.

Compliance and Protection staff mentioned two areas that would create efficiencies during site inspections. It would be beneficial for Fishery Officers to access their database in real-time during site inspections and it would be beneficial if Fishery Officers could have the ability to ticket offences. Currently, Compliance and Protection only have the ability to issue warnings. Ticketing offences for minor violations would create greater program credibility while helping to reduce the trial time and costs associated with pursuing charges under the Fisheries Act.

Overall, the working relationship between BCARP interests and stakeholders (other DFO programs, other Federal Government Departments/Agencies, Government of British Columbia, First Nations, Industry and ENGOs) is satisfactory. There are still opportunities to improve some processes as describe below.

The BCARP Service Level Agreement between the Pacific Region and the National Headquarters has helped to clarify roles and responsibilities. The program should consider revising this Service Level Agreement which expired in 2014 with a view to: increasing engagement of internal and external committees; streamlining decision-making; and strengthening and improving coordination and buy-in across sectors.

Relationships with other DFO programs (Science, Salmonid Enhancement Program, Aboriginal Strategy and Governance, Resource Management) are developing but could be improved. For example there could be better engagement of DFO Science with regards to policies pertaining to sea cucumbers.

There is a formal written agreement between BCARP and two Ministries of the provincial government. There are also regular committee meetings which provide the opportunity to clarify operational issues such as roles and responsibilities with regard to siting requirements and licensing.

Some key informants mentioned that in some cases the delays required in the harmonized application review process between the various stakeholders are significant. Further harmonizing the Front counter BC application process for aquaculture submissions would be helpful. BCARP and the province of BC are continuing their efforts to make the process more efficient.

Finally, key informants suggested that consultations with First Nations are processes that could be better harmonized between federal departments as well as with the province of BC. For example, if a certain site requires FN consultations by both DFO and Transport Canada, it would be useful that only one department would handle consultations for both departments rather than having two separate consultation processes. Similarly, from a provincial government perspective, it was suggested that both the federal government and the provincial government use the same legal approach when consulting FN so that consultations can be done with one process.


10  http://www.dfo-mpo.gc.ca/ae-ve/evaluations/14-15/6B164-AAH-eng.html

4.4 PERFORMANCE MEASUREMENT

Although the program collects large amounts of information, very little performance data is gathered by the program and it does not cover the entire range of program activities. The evaluation did not find a program logic model or a performance measurement strategy in place. In 2010, some efforts were made by the program to develop performance information which included a number of indicators. However, the document was not finalized and the data was not gathered. This gap in performance measurement information would need to be addressed as section 6.2.1 of the Directive on the Evaluation Function11 notes that program managers are responsible for “developing and implementing ongoing performance measurement strategies for their programs […]”.

The program needs as a first step to refine the logic model developed for the evaluation. For example, although AEO undertakes an array of activities related to fish health monitoring, the intermediate outcome of “CFIA notified of potential incidences of reportable diseases” might be more suitable as an output than an outcome and a new intermediate outcome should be determined. As well, the wording of some outcomes, for example “accountability” and “accurate” are hard to measure which creates difficulties when assessing the degree of achievement of those outcomes. Once the logic model is refined, the program would need to develop performance indicators that are specific, measurable, attainable, relevant and trackable. The last step is to collect data systematically with a consistent approach as much as possible.

Developing robust performance measurement tools will enable the program to improve decision-making as well as reinforce the accountability of the program by providing the necessary data to demonstrate its performance story.


11 http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681

4.5 BEST PRACTICES AND/OR LESSONS LEARNED

The guidebooks developed for major and small fish hatcheries to ensure compliance with the Pacific Aquaculture Regulations were considered a good practice and found very helpful. The guidebooks help operators to remain vigilant with respect to fish health protocols. They were also noted as a useful tool to aid with knowledge transfer when a new operator comes on board.

5. CONCLUSIONS AND RECOMMENDATIONS


5.1 RELEVANCE

The British Columbia Aquaculture Regulatory Program was implemented by DFO in December 2010 following a BC Supreme Court decision which ruled that finfish aquaculture is a fishery and therefore falls under federal jurisdiction. BCARP is needed as it regulates and ensures that BC aquaculture industry is compliant with regulations and legislation which contribute to the viability and sustainability of the BC aquaculture industry. BCARP aligns with two DFO strategic outcomes: “Economically Prosperous Maritime Sectors and Fisheries” and “Sustainable Aquatic Ecosystems”. BCARP also aligns to the whole of government framework priority of “Strong Economic Growth”. BCARP derives its mandated activities from four pieces of legislation: Fisheries Act, Oceans Act, Pacific Aquaculture Regulations and Fishery (General) Regulations.

5.2 EFFECTIVENESS

There are three sectors of aquaculture in BC – marine finfish (mostly salmon), shellfish (mostly oysters and clams) and freshwater/land-based. In 2013, 89% of BC aquaculture production was salmon. The shellfish and freshwater sectors represented around 11% of BC aquaculture production and do not present the same level of risks as the marine finfish sector. Due to higher risks associated with the marine finfish sector and public concern, BCARP’s efforts have been concentrated on this sector.

The evaluation assessed the extent to which the program has achieved its intended outcomes. Overall, BCARP has made good progress on most immediate and intermediate outcomes. Long term outcomes were not assessed since the program has been in existence for only four years.

The evaluation found that tools and advice provided by Aquaculture Programs are used to support decision-making. Public reporting has improved since BCARP undertook the management and regulation of the aquaculture industry in BC. Information on BC aquaculture sites is available and accessible to the public. This publicly accessible information contributes to transparency and accountability in DFO’s management of the aquaculture sector in British Columbia.BCARP took over the responsibilities of regulating and managing the aquaculture in British Columbia from the province in December 2010. BCARP was successful in replacing aquaculture licences that were previously issued by the Province of British Columbia. Licenced operators have, for the most part, the necessary information, oversight and guidance required for their operations.

BCARP’s contribution to Aquaculture industry growth and economic opportunities was limited due, in part, to external factors, such as a moratorium on issuing new marine finfish licences and the appreciation of the Canadian dollar.

Conservation and Protection efforts have been mostly allocated to the marine finfish sector. Non-compliance issues are addressed and the marine finfish sector is compliant with regulations and legislation for the most part. As more Conservation and Protection resources shift to the shellfish and freshwater sectors, a more comprehensive understanding of compliance and/or gaps in these sectors will be available.

Finally, BCARP is informed about the health of BC aquaculture stock and the environmental performance of sites. Fish health inspections have covered every active marine finfish facility (100%) during a three-year period. Environmental monitoring also occurs for shellfish and freshwater sites – approximately 10% of the active sites in each of these sectors are inspected annually. BCARP’s Aquaculture Environmental Operations do not play a significant role in notifying the Canadian Food Inspection Agency of reportable diseases.

5.3 EFFICIENCY AND ECONOMY

BCARP demonstrates efficiencies but certain improvements could be made in terms of streamlining the decision making process, human resources allocation, improving information management/information technology, clarification of roles and responsibilities as well as gathering performance data.

BCARP will introduce multi-year licences for federally regulated aquaculture operations in British Columbia that should save time, money and promote industry growth. Changes to conditions of licence have become less frequent as the conditions have stabilized over the last few years. Further streamlining the decision making process would help minimize costs for the review of licence applications.

There are human resources allocation challenges that were observed by the evaluation team. For example, staff tend to work in silos (within their activity areas and by aquaculture sector) and could be more versatile. This combined with an uneven work balance in some activity resulted in inefficiencies. As the program evolved, the alignment of staff with respect to current program priorities and activities and upcoming changes should be examined to ensure optimal allocation of resources. The importance of having versatile employees in a program as diverse as BCARP would be a clear advantage.

There is duplication of staff time and effort spent with respect to data management. The Aquaculture Integrated Information System that was developed for BCARP is continuing to evolve to better meet program needs.

Strong coordination between National Headquarters and BCARP would: foster engagement of all stakeholders; streamline decisions; and secure buy-in across DFO sectors. The Service Level Agreement between NHQ and BCARP, which expired in 2014, could be re-examined to address these issues.

Finally, BCARP should ensure that it has the appropriate performance measures in place to assist in program decision-making and accountability.

5.4 RECOMMENDATIONS

Recommendation 1: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, reassess BCARP’s human resources allocation to ensure alignment of staff with respect to current program priorities and activities and upcoming changes.

Recommendation 2: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, develop BCARP’s Performance information tools (refined logic model and develop performance indicators). Once the tools are developed, ensure that data is collected and reported consistently.

Recommendation 3: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, develop an agreement for BCARP between Ecosystems and Fisheries Management and the Pacific Region.  The agreement should clarify roles and responsibilities between National Headquarters and the Pacific Region related to improving the AQUIIS database as well as increasing engagement of internal and external committees, streamlining decision-making and strengthening coordination and buy-in across sectors.

ANNEX A: MANAGEMENT ACTION PLAN


Management Action Plan
Recommendation

Rationale: As the program has evolved, human resources allocation would benefit from a realignment to optimize resources with current program priorities and activities as well as upcoming changes.

Recommendation 1: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, reassess BCARP�s human resources allocation to ensure alignment of staff with respect to current program priorities and activities and upcoming changes.

Strategy
The Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region will formally request an assessment of human resource allocation by the respective Director Generals / Directors / Managers under the BCARP. Once complete, this assessment will be used to support discussions and actions to ensure alignment of staff with respect to current and projected future program priorities and activities.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Reason for Change in Due Date Output
Hold a DG / Director level discussion to determine an action plan for the assessment of BCARP human resource allocation. Target the identification of current program priorities and activities and upcoming changes and their impacts on Human resource allocation. July 2015    
Directors and Managers under BCARP identify current program priorities and activities and upcoming changes. September 2015    
In light of current program priorities and activities and upcoming changes, Directors and Managers under BCARP conduct an assessment of human resource allocation. November 2015    
Where required, Director Generals / Directors / Managers develop action plan to re-align staff with respect to current program priorities and activities and upcoming changes. January 2016    
DG level presentation of action plan to re-align staff with respect to current program priorities and activities and upcoming changes to ADM/RDG EFM/Pacific Region. February 2016    
Implement approved action plan to re-align staff with respect to current program priorities and activities and upcoming changes. March 2016    
Management Action Plan
Recommendation

Rationale: Developing robust performance measurement tools will enable the program to improve decision-making as well as reinforce the accountability of the program by providing the necessary data to demonstrate its performance story.

Recommendation 2: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, develop BCARP’s Performance information tools (refined logic model and develop performance indicators). Once the tools are developed, ensure that data is collected and reported consistently.

Strategy
The Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region will formally request the development of robust performance measurement tools (refine logic model and identify indicators). The development and data collection of performance measurement information will enable the program to improve decision-making as well as reinforce the accountability of the program by providing the necessary data to demonstrate its performance story.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Reason for Change in Due Date Output
Hold a DG / Director level discussion to ensure the development of robust performance measurement tools (refine logic model and identify indicators) in relation to BCARP. July 2015    
Directors and Managers under BCARP refine BCARP logic model. September 2015    

Directors and Managers under BCARP:

Analyse existing performance measurement information; and

Develop indicators for outcomes identified in the logic model; and

Collect data on at least an annual basis.

November 2015    
ADM/RDG EFM/Pacific Regional approval of performance measurement tools. January 2016    
Management Action Plan
Recommendation

Rationale: An agreement between the Ecosystems and Fisheries Management Sector and the Pacific Region would reinforce working relationships.

Recommendation 3: It is recommended that the Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region, develop an agreement for BCARP between Ecosystems and Fisheries Management and the Pacific Region. The agreement should clarify roles and responsibilities between National Headquarters and the Pacific Region related to improving the AQUIIS database as well as increasing engagement of internal and external committees, streamlining decision-making and strengthening coordination and buy-in across sectors.

Strategy
The Senior Assistant Deputy Minister, Ecosystems and Fisheries Management, in collaboration with the Regional Director General, Pacific Region will formally request a review of the existing agreement and develop an action plan to put in place a renewed Agreement for the BCARP.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Reason for Change in Due Date Output
Hold a DG / Director level discussion to determine an action plan to establish an agreement. July 2015    
Directors / Managers of BCARP draft an agreement. September 2015    
ADM/RDG EFM/Pacific Region approval of agreement. November 2015    

ANNEX B: EVALUATION MATRIX


Evaluation Matrix
Question Indicator Interviews Survey Docu- ment Review Admin Data Data Source
Program Stakeholder
1.0 RELEVANCE
1.1 Is there a continued need for BCARP? Views on the importance/need of the BCARP checkmark checkmark checkmark      
Evidence to support continued need       checkmark    
Presence / absence of other programs that compliment or duplicate the objectives of the BCARP       checkmark    
1.2 To what extent is BCARP aligned to departmental and federal government priorities? Degree of alignment of BCARP with current federal government objectives and priorities checkmark     checkmark    
Degree of alignment of BCARP with current departmental strategic outcomes checkmark     checkmark    
1.3 Is BCARP consistent with federal roles and responsibilities? BCARP mandate aligned with federal government jurisdiction       checkmark    
Views on the appropriateness of federal roles and responsibilities checkmark checkmark checkmark      
2.0 EFFECTIVENESS
2.1 To what extent has BCARP achieved its immediate outcomes?
2.1.1 To what extent are tools and advice used in management decision- making? Views of the effectiveness tools in management decision-making. checkmark checkmark   checkmark    
Views of the effectiveness advice in management decision-making. checkmark checkmark   checkmark    
2.1.2 To what extent are licenced operators aware of regulatory obligations? Views on the effectiveness of BCARP in providing necessary information, oversight and guidance to aquaculture sites. checkmark checkmark   checkmark    
Views of BCARP staff and stakeholders on the effectiveness of management tools (IMAPs, policies & procedures) for managing aquaculture sites in BC. checkmark checkmark   checkmark    
2.1.3 To what extent are non-compliance issues addressed?

C&P performance indicators:
% of sites covered under general inspections
% of general inspections completed against planned
% of complaints investigations against complaints received
Number of convictions
Number of repeat offences
Number of severe offences
% of operators offending

        checkmark British Columbia Aquaculture Compliance & Protection Strategy document
Views of BCARP staff and stakeholders in addressing non-compliance issues. checkmark checkmark   checkmark    
2.1.4 To what extent is BCARP informed on the health of BC aquaculture stock and the environmental performance of aquaculture sites? Views of BCARP staff and stakeholders on information related to the health of BC aquaculture stock and environmental performance of sites. checkmark checkmark   checkmark    
2.1.5 To what extent is information about aquaculture accurate, available and accessible to the public? Views of BCARP staff and stakeholders on availability and accessibility of aquaculture information. checkmark checkmark   checkmark    
2.2 To what extent has BCARP achieved its intermediate outcomes?
2.2.1 To what extent is the aquaculture industry compliant with regulations and legislation % of British Columbia aquaculture sites which are compliant with regulations and legislation.         checkmark AQUIIS
Views of BCARP staff and stakeholders on compliance of licence holders with regulations and legislation. checkmark checkmark   checkmark    
2.2.2 To what extent does BCARP facilitate industry growth and economic opportunities? Variance in the production of aquaculture companies in British Columbia         checkmark CANSIM
Variance in the value-added of aquaculture companies in British Columbia         checkmark CANSIM
Views of BCARP staff and stakeholders on aquaculture industry growth in BC and economic opportunities checkmark checkmark   checkmark    
2.2.3 To what extent is CFIA notified of potential incidences of reportable diseases? % of potential incidences of reportable diseases communicated to CFIA         checkmark AQUIIS
Views of BCARP staff and stakeholders on BCARPs ability to notify CFIA on potential incidences of reportable diseases. checkmark checkmark   checkmark    
2.2.4 To what extent is there transparency and accountability in DFO's management of the aquaculture sector in BC? Views of BCARP staff and stakeholders on the degree of transparency of DFO's British Columbia�s aquaculture information checkmark checkmark   checkmark    
2.3 Are there any internal or external factors and/or challenges that may have impacted the results of BCARP? For internal factors: Evidence of risks that have influenced the achievement of program outputs. checkmark checkmark   checkmark    
For internal factors: Evidence that BCARP has established systems in place to monitor and mitigate risks. checkmark checkmark   checkmark    
For external factors: Evidence that the program is aware of threats and/or opportunities to the program, and that these factors are being monitored. checkmark checkmark   checkmark    
2.4 Are there any unintended outcomes, positive or negative, that can be attributed to BCARP? Identification and assessment of unintended positive and negative outcomes reported by staff, stakeholders and/or observed by the evaluator (high profile or recurring items or issues)            
3.0 EFFICIENCY
3.1 Is the BCARP design and delivery appropriate for achieving program results? Clearly defined and understood governance structure, including program processes, roles and responsibilities       checkmark    
Program resources/capacity are adequate for achievement of expected results       checkmark    
Views on the appropriateness of program activities, processes and governance structures checkmark checkmark        
3.2 Are appropriate performance data being collected and stored? If so, is this information being used to inform senior management / decision makers? Existence of effective performance measurement plan; evidence of data collection and a performance data system       checkmark    
Extent to which performance data informs BCARP and departmental wide decision- making processes       checkmark    
Views on the extent to which the performance measurement strategy can be improved checkmark          
3.3 Is BCARP undertaking activities in the most efficient manner?
  • How could the efficiency of the BCARP’s activities be improved?
  • Are there alternative, more efficient, ways of delivering the program?
Comparison of program activities and products delivered by other similar programs       checkmark    
Administrative costs of BCARP       checkmark    
Views on how the efficiency of program activities could be improved (operational efficiency) checkmark checkmark checkmark checkmark    
Views on how the allocation of program resources could be improved (allocative efficiency) checkmark checkmark checkmark checkmark    
3.4 Does BCARP demonstrate use of best practices and/or lessons learned in the design and implementation of their activities? Evidence that program outputs have incorporated best practices and/or lessons learned. checkmark checkmark   checkmark    
4.0 ECONOMY
4.1 Is BCARP achieving its intended outcomes in the most economical manner? Evidence /views on whether there is overlap / duplication with other programs checkmark checkmark   checkmark    
Views on the adequacy of program resources checkmark checkmark        
Evidence of approaches that could assist in more efficient management of program inputs checkmark checkmark   checkmark