EVALUATION OF THE OCEAN FORECASTING PROGRAM

EVALUATION DIRECTORATE


Table of Contents

ACKNOWLEDGEMENTS AND ACRONYMS


ACKNOWLEDGEMENTS

The Evaluation Directorate acknowledges and thanks all individuals who gave of their time and input for this evaluation.

ACRONYMS


List of acronyms
ACCASP Aquatic Climate Change Adaptation Services Program
AZMP Atlantic Zone Monitoring Program
CONCEPTS Canadian Operational Network of Coupled Environmental Prediction Systems
DFO Fisheries and Oceans Canada
DND Department of National Defence
ISDM Integrated Science Data Management
IT Information Technology
NSDC National Science Directors Committee
OCB Oceanography and Climate Branch
OSD Oceanography and Scientific Data
PMF Performance Measurement Framework
PM Performance measurement

EXECUTIVE SUMMARY


Introduction

This report presents the results of an evaluation of the Ocean Forecasting Program. The evaluation assessed the Program’s relevance and performance, the latter including effectiveness, efficiency and economy, in accordance with Treasury Board’s Policy on Evaluation. The Evaluation Directorate of Fisheries and Oceans Canada (DFO) conducted the evaluation between May 2013 and February 2014.

The evaluation covers all activities undertaken by the Ocean Forecasting Program during the period from 2008-09 to 2012-13.  In 2011-12 and 2012-13, the Ocean Forecasting Program was given the mandate to oversee the creation of a new program, the Aquatic Climate Change Adaptation Service Program (ACCASP).  As of 2013-14, this program is now captured under a separate program on the department’s Program Alignment Architecture (PAA).  For this reason, ACCASP was not covered by this evaluation.

Program Profile

As a maritime nation bordered by three oceans and an extensive network of inland waters, Canada has a requirement to understand oceans processes to enable prediction of ocean conditions and their influences on our environment, ecosystems, and coastal communities. This is accomplished through research and long-term monitoring of key ocean parameters via space-based, aerial, autonomous vehicles, and vessel-based observations and the management of data to ensure its integrity and accessibility. The Ocean Forecasting Program is the foundation for marine information, including ocean prediction products and services used to support emergency preparedness, adaptation to climate change, mitigation of oils spills, and at-sea operations such as fisheries.

Evaluation Methodology

This evaluation adopted a “realist evaluation approach”, whereby measurement of the Program’s success in achieving its expected outcomes took into account the complex and dynamic nature of what is meant by success.  As such, evaluation questions sought to address the circumstances in which outcomes are achieved, recognizing that the meaning of success can differ across individuals, organizations, functions, and time. Furthermore, the evaluation adopted a client perspective, where the measures of success were based on the needs of the various stakeholders along the Program’s science production chain. Key information sources included internal and external key informants, program documents, and web literature on other countries’ ocean forecasting functions.

Evaluation Findings

There is an increasing need for the Ocean Forecasting Program due to changing climate conditions which create the need for real time and accessible ocean data. The Program’s science efforts are also directed towards the assessment of potential impacts on marine environments, ecosystems, fish and marine mammal populations.

The Ocean Forecasting Program is well aligned with federal priorities and contributes to all departmental outcomes. It is the only federal program that has the organization and infrastructure to collect and generate the basic oceanographic data that underpins departmental research in fisheries, ecosystems and oceans management on a long term basis and the ability to provide neutral advice to the federal government on ocean variability. The Canadian model for Ocean Forecasting is consistent with the worldwide trend to have these services managed by government jurisdiction. Moreover, the interconnectedness of the oceans and the requirement for international cooperation in the management of the oceans means that it is a role that can only be fulfilled by the federal government, making it the most indicated entity to deliver the Program.

The Ocean Forecasting Program is internationally recognized and clearly contributes to a number of ocean prediction systems and collaborations, such as Argo, the Atlantic Zone Monitoring Program (AZMP), the Canadian Operational Network of Coupled Environmental Prediction Systems (CONCEPTS), etc.  However, there is unclear definition of intended products and services, and limited capacity to address specific clients’ needs.

The key overarching limitation of the Program is the insufficient integration of its multiple and diverse activities. At the time of evaluation, Program activities were conducted in relatively isolated silos in DFO regions and between different directorates, such as the Canadian Hydrographic Service and the Ecosystem Science directorates.   For one, the Program uses multiple and at times incomparable data and platforms.  Quality, precision and timeliness of ocean science data are inconsistent from one region and one function to another. The Program’s fragmentation also limits its ability to identify and meet the needs of all potential clients, there being no single point of entry to the Program for internal or external partners and clients.

In addition, the development of an “operational oceanography” function at DFO is perceived to be weak by Program partners and clients.  Varying definitions of operational oceanography were encountered during data collection.  For the purpose of this evaluation, operational oceanography was defined as a service-oriented approach where the needs of well-identified target users inform decisions on the development and dissemination of specific science products, while factoring in resource and technical limitations. This particular definition was chosen because it speaks directly to the recognized need for better identification of clients, client needs and of specific articulations between those needs and program activities (i.e., is the program producing useful products and are all intended and potential users aware of and benefiting from these products). In a context of limited federal government resources (i.e., successive economic reduction exercises such as strategic and operational reviews), it is important for the program to be able to document and justify allocation of resources to specific activities and a clear articulation of intended products and target clients is a key element of such justification.

Whereas such an approach has been firmly established in the international and Canadian meteorological services community for several years, it is, by all accounts, comparatively new for the world of ocean sciences and DFO is only just starting to explore this orientation. Nevertheless, international comparisons show that other countries are more advanced in this field, if only due to stronger integration of their oceanographic and meteorological science functions (e.g., within the same organisation).

The evaluation found examples showing that some Program information and products are being used as intended by a variety of users including science researchers, the Canadian Coast Guard, other government departments, and the media. It however remains unclear whether all Program outputs are being fully used and for what purpose.  Evidence suggests that use is somewhat limited by potential users’ insufficient awareness of these outputs and by insufficient tailoring of these outputs to intended users’ needs.

Finally, key concerns have been raised about a perceived erosion of the Program’s cutting edge positioning and capacity, which is seen to affect its ability to interact with international partners and to “pull its weight” on the international oceanography scene.

A number of contextual factors, including resource shortages, limited Information Technology (IT) capability and support, and ageing vessels are impacting the effectiveness of the Program.  For example, the work performed by the Ocean Forecasting Program entails heavy reliance on information technology, be it storage capacity for data, server capacity to process large quantities of data, or specialized software to model, analyze, or share the data within and outside the department. According to various Program representatives, their specialised science IT needs are not as well met since the Department’s computing and data infrastructure were consolidated under Shared Services Canada (SSC).

There is also a need for national coordination and integration of the Program’s governance structure.  Currently, daily Program transactions rely on ad hoc, informal decision-making by Program staff, with an over-reliance on interpersonal relationships between regions and headquarters.  There is no formal mechanism to coordinate cross-functional collaboration at the Program working level.  However, Program managers have taken the lead in developing a national service framework in order to address the need for national coordination and integration.  To date, an informal network of senior scientist in various regions have been engaged in ongoing discussions, a survey of Program clients has been conducted, and a pre-framework document is being drafted.

According to the majority of senior managers and Program representative from all sectors and regions, the Program is operating with increasingly diminishing resources as a result of attrition.  However, there is clear evidence that the Program is leveraging significant additional resources from multiple external sources. In addition to international collaborations, the most significant of these sources is the Program’s participation in CONCEPTS, whereby DFO gains access to Environments Canada’s supercomputer and meteorologists and oceanographers’ expertise, as well as the Department of National Defence (DND)’s marine vessels for ocean data collection. 

A number of other avenues might be explored to further improve Program efficiency, such as stronger national integration, more dedicated IT support, the pursuit of additional opportunities for collaboration with stakeholders, co-locating of some activities with Environment Canada’s client service function, the provision of competitive funding envelopes to influence the orientation of academic research, and the implementation a cost-recovery fee structure for some of its services. The feasibility and potential benefits of such opportunities will be more easily identifiable and marketable once a clearer definition of the Program is developed. Ultimately, the Program will be in a better position to identify means to reduce costs and maximize Program impacts once it has more precisely identified its target clients. This will enable development of tools and products that can benefit more than one user and an improved targeting of user needs.

Recommendations

With the above findings in mind, the evaluation has made the five following recommendations.

Recommendation 1:  It is recommended that the Program pursue and finalize the development of a national service framework.

Recommendation 2:  Either as part of the framework or of subsequent concept papers, it is recommended that the Program develop a vision and mandate, clear accountabilities and decision-making structure, and formalized roles and responsibilities for the delivery of all the Program’s activities at national headquarters and in the regions.
Recommendation 3:  Either as part of the framework or of a subsequent exercise, it is recommended that the Program identify its target clients and their needs, as well as the products and services the Program intends to and/or currently offers to address those needs.

Recommendation 4:  Either as part of the framework or of a subsequent exercise, it is recommended that the Program develop a risk management strategy, with concrete mitigation actions to address risks related to limited IT support and capability, the ageing of research vessels, Program resource limitations, and expert knowledge transfer.

Recommendation 5:  It is recommended that, as part of the development of a national service framework, the Program also update its PM strategy, including a revised description of activities, outputs, expected outcomes, performance indicators, targets and data collection strategy.

1. INTRODUCTION


This evaluation report presents the results of the evaluation of the Ocean Forecasting Program. In accordance with the Treasury Board Policy on Evaluation (2009), the evaluation assessed the core issues of relevance and performance, including effectiveness, efficiency and economy.

This evaluation covered the period from 2008-09 to 2012-13 and was undertaken between May 2013 and February 2014 by Fisheries and Oceans Canada’s (DFO) Evaluation Directorate.

1.1 CONTEXT OF THE EVALUATION

As identified by the Treasury Board Policy on Evaluation, all direct program spending must be evaluated every five years. The evaluation of the Ocean Forecasting Program was identified in the DFO’s 2012-13 multi-year departmental evaluation plan and focuses on the core issues in assessing value for money as defined by the Treasury Board Policy on Evaluation. These core issues include: relevance (demonstrable continued need, alignment with government priorities, and alignment with federal roles and responsibilities) and performance (effectiveness, efficiency and economy).

The data gathered by the Ocean Forecasting Program are the foundation for ocean prediction products, services, and information that is used to inform safe navigation, emergency preparedness (e.g., tsunami warnings, storm surges), adaptation to climate change, search and rescue, the mitigation of oil spills, and at-sea operations such as offshore oil and gas. Clients of the Program include internal users such as the Canadian Coast Guard, other federal government departments and agencies (e.g., Environment Canada, Department of National Defence (DND), Transport Canada, Public Safety Canada), various maritime industries (e.g., commercial shipping, off-shore oil and gas, fishing industry), the Canadian and international marine science community, and interested Canadians.

1.2 SCOPE OF THE EVALUATION

The evaluation covers activities undertaken by the Ocean Forecasting Program during the five-year period from 2008-09 to 2012-13 and includes the National Capital Region and all six DFO regions (Pacific, Central and Arctic, Quebec, Maritimes, Gulf, Newfoundland and Labrador).  In 2011-12 and 2012-13, the Oceanography and Climate Branch (OCB) was given the mandate to create a new program, the Aquatic Climate Change Adaptation Services Program (ACCASP).  As of 2013-14, this program is now captured as a separate program on the department’s Program Alignment Architecture (PAA).  For this reason, ACCASP was not covered by this evaluation.  A horizontal evaluation of the Helping Canadians Adapt to Climate Change initiative, which includes DFO’s ACCASP, will be led by Environment Canada in 2015-16.

1.3 REPORT STRUCTURE

The executive summary provides a brief summary of the evaluation findings, conclusions and recommendations. Section 2 presents an overview of the Ocean Forecasting Program’s mandate, activities and resources. Section 3 describes the evaluation methodology, followed by a discussion of the main findings in section 4 and conclusions and recommendations in section 5. Annex A presents the recommendations and the management action plan. Annex B presents the evaluation matrix, i.e., key issues, questions, indicators and data sources used for the evaluation. Annex C includes the Program logic model.

2. PROGRAM PROFILE


2.1 CONTEXT

As a maritime nation bordered by three oceans and an extensive network of inland waters, Canada has a requirement to understand ocean processes to enable prediction of ocean conditions and their influences on our environment, ecosystems, and coastal communities.

Activities and elements of the current Ocean Forecasting Program have been present at DFO for decades. Data series pertaining to oceanographic parameters were being collected in years prior to 1940. Over time, DFO ocean science related activities have come to be known as the Ocean Forecasting Program. Today scientists working under the umbrella of the Ocean Forecasting Program develop ocean forecast products, services, and research which support a wide range of initiatives such as: ecosystem-based management approaches; climate change adaptation strategies; emergency preparedness for marine natural hazards; search and rescue operations; oil spill mitigation; and at-sea operations such as fisheries and offshore energy.

2.2 MANDATE

Under Section 41 of the Oceans Act, DFO has a legislative mandate to deliver services for: the safe, economical and efficient movement of ships in Canadian waters; search and rescue in the marine environment; and, marine pollution response. These mandated services depend strongly on operational oceanography to supply information products on which decisions are based. Mandated services in this area supports two of DFO’s Strategic Outcomes, namely 1) Economically Prosperous Maritime Sectors and Fisheries and 2) Safe and Secure Waters. Strategic Outcomes are integral to the Department’s PAA, which is part of the Government of Canada’s Management, Resources and Results Structure, the foundation of a government-wide approach aimed at strengthening the management and accountability of public expenditures and clearly demonstrating results for Canadians. 

Furthermore, under Section 42 of the Oceans Act, in performing the duties of the Fisheries Act, the Minister may, among other:

  • collect data for the purpose of understanding oceans and their living resources and ecosystems;
  • conduct hydrographic and oceanographic surveys of Canadian and other waters; 
  • conduct marine scientific surveys relating to fisheries resources and their supporting habitat and ecosystems;
  • conduct basic and applied research related to hydrography, oceanography and other marine sciences, including the study of fish and their supporting habitat and ecosystems; and,
  • participate in ocean technology development.

Mandated services in this area support DFO’s third Strategic Outcome: Sustainable Aquatic Ecosystems.

Finally, under Section 43 of the Oceans Act, the Minister is responsible for coordinating, promoting and recommending national policies and programs with respect to fisheries science, hydrography, oceanography and other marine sciences.

2.3 GOVERNANCE AND ORGANISATIONAL STRUCTURE

The organisational structure of the Ocean Forecasting Program has changed several times over the evaluation period of 2008-09 to 2012-13.  Table 1 provides a brief overview of the Programs’ past and current structure. 

Table 1. Evolving Organisational Structure of the Ocean Forecasting Program

Evolving Organisational Structure of the Ocean Forecasting Program
Period Name of Program PAA Branch Directorate Sector
September 2008
to
April 2010
Ocean Climate Oceanography and Climate Ocean Sciences and Canadian Hydrographic Service Science
Integrated Science Data Management
April 2010
to
June 2011
Ocean Climate Oceanography and Climate Ocean Sciences and Canadian Hydrographic Service Oceans and Science
Integrated Science Data Management
June 2011
to
October 2013
Ocean Forecasting Oceanography and Climate Ecosystem Science Ecosystems and Oceans Science
Integrated Science Data Management Canadian Hydrographic Service
October 2013
to
present
Ocean Forecasting Oceanography and Scientific Data Canadian Hydrographic Service Ecosystems and Oceans Science

At the national level, the Assistant Deputy Minister, Ecosystems and Oceans Science, has the overall accountability for the Program development and delivery.  At the regional level, the Program follows the Department’s matrix management model.  Regional Directors of Science are responsible for the Program delivery in the regions.  They have a line reporting relationship to Regional Director Generals.

Although the Program’s position has shifted within the department, its activities have remained the same.  The Program relies heavily on regional involvement, where the majority of activities take place. Scientists of the Ocean Forecasting Program are mainly located in five regional science institutes:  the North West Atlantic Fisheries Centre in St. John’s, Newfoundland; the Bedford Institute of Oceanography in Dartmouth, Nova Scotia; the St. Andrews Biological Station in St. Andrew’s, New Brunswick; the Maurice Lamontagne Institute in Mont-Joli, Quebec; and, the Institute of Ocean Sciences in Sidney, British Columbia.

The Program also relies heavily on Integrated Science Data Management (ISDM), which is responsible for managing and archiving in situ ocean data collected by DFO, or acquired through national and international programs.  Figure 1 provides an overview of the Program’s organisational structure that was in effect during the time the evaluation was conducted.  It is important to note that as of October 2013, the OCB and ISDM branch have been merged together as the Oceanography and Scientific Data branch of the Canadian Hydrographic Service.

Figure 1. Organizational Structure of the Ocean Forecasting Program

Ocean Forecasting Program

ocean forecasting program

2.4 EXPECTED RESULTS

The objectives of the Ocean Forecasting Program are:

  • Contribute to the greater understanding of Canada’s three Oceans and significant inland waterways through the monitoring and analysis of ocean parameters, the generation of ocean forecasts, and the management and dissemination of data.
  • Provide scientific advice to support decision-making in areas pertaining to variability and change in Canada’s oceans and significant inland waterways.

2.5 PROGRAM ACTIVITIES

The main activities of the Ocean Forecasting Program help support the generation of information and knowledge on Canada’s oceans and major waterways that supports DFO operations and management functions.

According to the program’s Performance Measurement Strategy, the main intended activities include: ocean observation and monitoring; modeling; analysis; research and development; data management; collaboration with partners; and, provision of advice.

Ocean Observation and Monitoring

The Ocean Forecasting Program collects data on selected ocean parameters through multiple national and international initiatives, using a variety of technologies such as in situ measurement from ships, buoys and moorings; autonomous floats; and remotely sensed images from satellites. Selected marine parameters include sea surface temperature, currents, salinity, sea level height, and phytoplankton blooms.  A few key examples of observation and monitoring initiatives are:

  • The Argo Monitoring System: Argo is the largest ocean climate monitoring program in the world. It consists of over 3,500 free-drifting floats that collect data on ocean temperature and salinity. DFO is a strong contributor to the international Argo project.  Since 2001, the department has launched over 350 floats, 100 of which are still operating.1 Argo data are publically available and are used for improving weather and ocean models, as well as for research.
  • Line P Time Series: The Pacific Region has been monitoring and collecting ocean variables on the Line P Time Series since 1956. The long-term nature of this program allows for the identification of significant oceanographic changes due partly to global climate change.

The Atlantic Zone Monitoring Program (AZMP): This program was developed in 1998 to complement data series already maintained on the Atlantic Ocean since the beginning of the 20th century.  Specific stations have been identified for the collection of long time series. The data are used to analyze the biological, chemical and physical state of Eastern Canadian waters.

Modelling

The Ocean Forecasting Program develops ocean forecasting models that predict the conditions of Canada’s oceans or a region within an ocean in three dimensions and projects its evolution over time. The ocean forecasting models are based on marine data collected by earth observation technology and direct measurement of conditions such as temperature, salinity, currents and wave heights.

There is increasing recognition that accurate atmospheric and oceanic forecasts require “coupled” (i.e., atmosphere-ice-ocean) models, as well as atmospheric and oceanographic data. Based on this integrated information, the ocean forecasting component can be used to predict various aspects of ocean conditions which can then be used operationally to assist, for example, in emergency preparedness planning for coastal infrastructure threatened by storm surges.

Real-time operations for at-sea search and rescue, navigation purposes, effective and efficient offshore industry operations (e.g., offshore oil and gas, fishing fleets, etc.) require ocean weather forecasting advice. This advice is supported by operational oceanography forecasts that are generated through the analysis and predictions of ocean variables such as currents, temperatures, salt content, and ocean waves. This information is provided to a variety of organizations, including the Canadian Coast Guard and fishery fleet managers and is used for tracking and aiding navigation of marine vessels.

Analysis

The Program receives oceanographic data from different sources (e.g., satellite data, Argo, etc.) This information is aggregated and analyzed by the Program.

Research and Development

Scientists within the Department conduct detailed research activities on many aspects of oceanography to examine oceans in systematic, scientific and minimally invasive ways. This research seeks to better understand questions such as: why oceans are becoming increasingly acidic, how the saltwater/freshwater outflow from the Canadian archipelago is changing ocean conditions in the Northwestern Atlantic, and how oxygen-depleted zones will impact marine ecosystems. Research is conducted into a wide range of oceanographic issues in Pacific, Arctic and Atlantic waters. 

New approaches, such as advanced sea floor observatories, new autonomous underwater vehicles, earth observation technologies, and other observation tools have the potential of allowing scientists supporting the Ocean Forecasting Program to increase scientific understanding of Canada’s oceans. This in turn can enable better forecasting of the ocean.

Data Management

Ease of access to ocean observations, both historical and recent, is essential for determining status and trends in chemical and physical marine environments, as well as for providing timely inputs and validation for modelling and forecasting activities.  The preservation and  management of  information and long-term data series (e.g., line P time series) falls under the purview of government to document the history of Canada’s environment, provide a basis for evidence based decision making and set baselines against which to measure change.

The data management function includes observational data as well as data generated by models and forecasting systems.   Data accessibility is essential to the public for full transparency and public trust.

The Ocean Forecasting Program provides management and stewardship for the oceanographic information collected by DFO through its regional and zonal programs. The Program mandate also includes acquiring and managing foreign sources of similar data and information. All information and data are to be assembled, processed and disseminated through Science and Data Centres operating in DFO’s regional offices.

The ISDM branch (now part of the OSD branch), in the National Capital Region maintains archives of in situ physical, chemical, and biological observations, coordinates data management activities in the regions through DFO’s National Science Data Management Committee, and acts as the focal point for international cooperation in data and information exchange.

Collaboration with Partners

The Ocean Forecasting Program relies on many collaborative initiatives with academia, other government departments, and the international science community, in order to help leverage and enhance the utilization of Program resources.

Canadian academics conduct leading edge oceanographic research on all three coasts of Canada. Collaboration between DFO and universities provides excellent synergies to advance oceanographic research. For example, the Ocean Forecasting Program has established a partnership with Université Laval’s ArcticNet, a Network of Centres of Excellence of Canada that brings together scientists and Inuit organizations to study the impacts of climate change in the coastal Canadian Arctic. Ocean Networks Canada at the University of Victoria is another example of academic partnership. Ocean Networks Canada maintains two cabled undersea observatories in the Pacific Ocean. Oceanographers from DFO’s Pacific Region are engaged in aspects of this initiative.

DFO oceanographers also take part in intergovernmental initiatives such as CONCEPTS. This cooperative agreement between Environment Canada, DFO, and DND allows for significant improvement in the analysis and forecasting of ocean features, including waves, currents, temperature, and pollutant transport for DFO’s operational oceanography program, to allow for a greater understanding and better management of Canadian marine ecosystems.

International collaborations are also an important part of the Ocean Forecasting Program. Oceanographic data are exchanged amongst countries and are incorporated into global ocean forecasting models in order to improve accuracy. Ongoing DFO oceanographic science is also supporting Canada’s contribution to the Intergovernmental Panel on Climate Change and the Intergovernmental Oceanographic Commission.

Provision of advice

DFO oceanographers generate advice for both Canadian and international decision-makers on issues such as emergency preparedness, operational oceanography, and health of the oceans.

Coastal communities rely on ocean forecasting advice related to the risk of storm surges and tsunamis.  Real-time data are also used for at-sea search and rescue missions as well as for offshore industry operations (e.g., offshore oil and gas, fishing fleets, etc.).  The Ocean Forecasting Program provides data to support advice on ocean management activities such as designing and managing marine protected areas, habitat management, and fisheries resource management. With global climate change, it is also increasingly important to monitor the state of the health of Canada’s oceans in order to detect changes in marine ecosystems. This type of research will be further pursued under the new ACCASP.


1 Argo Canada : http://www.meds-sdmm.dfo-mpo.gc.ca/isdm-gdsi/argo/index-eng.html

2.6 PROGRAM STAKEHOLDERS

Clients of the Ocean Forecasting Program include internal users such as the Canadian Coast Guard and Integrated Oceans Management Program, other federal government departments and agencies (e.g., Environment Canada, DND, Transport Canada, Public Safety Canada), and various maritime industries (e.g., commercial shipping, off-shore oil and gas, fishing industry).

The beneficiaries of the Ocean Forecasting Program are public and private sector organizations, and, in general, Canadians at large, who utilize Program information, forecasts and advice to navigate safely in Canada’s oceans and significant inland waterways. These beneficiaries primarily use information pertaining to currents, water levels, tide heights, and ice movements to assist them to safely navigate, as well as to support activities related to shipping, search and rescue, and maritime security. The Ocean Forecasting Program also enhances the economic well-being of most of the above Program beneficiaries. Based on Program information, actions are taken that reduce costs and/or maximize the revenue of commercial entities and individuals. 

Information on the state of  oceans and waterways conditions, including water temperature, and chemical composition of the water, serve as the foundation from which Canada’s scientists predict the impact of these conditions on the biology of water bodies and ecosystems both in terms of the overall health and productivity of Canada’s oceans. With this knowledge, the government can better assess fish stocks, and manage species at risk, aquatic habitat, and our networks of marine protected areas to the benefit of all Canadians.

Academia, the private sector, and the international science community regularly work with DFO’s oceanographers in collaborative initiatives, which help generate new knowledge, and create the models, information and forecasts that are key to the delivery of the Ocean Forecasting Program.

2.7 PROGRAM RESOURCES

The Program’s budget and expenditures for the five-year evaluation period are shown in Table 2.  The table includes information regarding salaries, operations and maintenance (O&M), other sources of funding such as grants and contributions, and full-time equivalents (FTEs).

Table 2.  Program Resources

Table of Program Resources
  Budget ($million) Expenditures ($million) FTEs
Salaries O&M Other Total Salaries O&M Other Total
2008-09 8.0 5.3 0.2 13.4 8.9 5.5 0.2 14.6 130.26
2009-10 7.2 4.3 0.02 11.5 9.0 4.0 0.02 13.0 124.62
2010-11 7.0 3.0 0.08 10.0 8.8 2.3 0.08 11.2 120.45
2011-122 8.3 3.5 0.5 12.3 9.5 3.1 0.5 13.1 117.19
2012-132 11.2 3.5 0.4 15.1 11.6 3.3 0.4 15.3 141.10

Source: Program Data


2 The figures provided for fiscal years 2011-12 and 2012-13 also include special funding ($2.2M and $5.18M respectively) allocated to DFO for the Aquatic Climate Change Adaptation Services Program (ACCASP), which is excluded from the scope of this evaluation. This funding could however not be subtracted from the totals shown here because the precise amounts allocated to the Ocean Forecasting Program (as opposed to other DFO program activities) for the ACCASP are unknown, and the specific breakdown across salary, O&M and other categories of budget and expenditures is unavailable.

2.8 PROGRAM RISKS

The Ocean Forecasting Program identified3 five key risks that have the potential of affecting the attainment of Program objectives:

  • Human Resources: There is a risk that Science will lack skilled knowledge workers, particularly in specific key areas such as Arctic Ocean science.
  • Physical Infrastructure: There is a risk that Science may not have sufficient resources to invest in the maintenance and control of the infrastructure (i.e. vessels, small crafts, Information Technology infrastructure, labs and equipment) that is necessary to achieve the Ocean Forecasting Program objectives.
  • Information Technology: There is a risk that the increasing complexity and volume of scientific data that must be stored and managed will lead to an inability of Science to fulfill its information and Information Technology needs.
  • Arctic: There is a risk that a lack of resources will result in Science not being able to deliver on the required evidence-based advice, information and services related to the Arctic. The impact of this risk on Canada’ credibility will be magnified if Canada is not able to show leadership on a number of Arctic files notably as it assumes its role as Chair of the Arctic Council until 2015 (since 2013). 
  • Partnerships: There is a risk that Science’s growing reliance on partnerships may lead to deficiencies in in-house expertise.

These risks impact several programs under the Oceans and Ecosystems Science Sector, therefore a shared mitigation initiative has been undertaken with other DFO science programs. The evaluation team reviewed these risks, which helped to inform various aspects of the evaluation, both in terms of context and further areas for exploration as part of the evaluation.


3 Ocean Forecasting Program Performance Measurement Strategy, May 2012.

3. METHODOLOGY


3.1 EVALUATION APPROACH AND DESIGN

This evaluation adopted a realist evaluation approach4, whereby measurement of the Program’s success in achieving its expected outcomes took into account the complex and dynamic nature of what is meant by success.  As such, evaluation questions sought to address the circumstances in which outcomes are achieved, recognizing that the meaning of success can differ across individuals, organizations, functions, and time.  Specifically, this meant asking the questions “what works?”, “for who?”, and “in what circumstances?” rather than simply asking “does the program work?”  Furthermore, this evaluation adopted a client perspective, where the measures of success are based on satisfaction of the needs of the various stakeholders along the Program’s science production chain. The Ocean Forecasting Program science production chain is illustrated5 in Figure 2.

Figure 2. Ocean Forecasting Program - Science Production Chain

ocean forecasting program illustration

* The Client Services Function currently has limited visibility.

Each participant in the science production chain is both a client and a supplier of scientific information and products to other participants within and outside the chain.  Outcomes are achieved when the intended clients for the Program’s various scientific information and products consider that their needs are met and when the information and products are used as intended by their target clientele. These outcomes largely reflect those identified in the Program’s logic model (Annex C), which was developed as part of the Program’s performance measurement (PM) strategy.


4 Realist evaluation is a type of theory-driven evaluation method particularly useful for evaluating complex programs. See Ray Pawson and Nick Tilley (2004). Realist Evaluation. http://www.communitymatters.com.au/RE_chapter.pdf

5 This illustration was developed for the purpose of this evaluation and validated by the various key informants interviewed. It is intended to illustrate the current functioning of the Program.

3.2 KEY ISSUES AND EVALUATION QUESTIONS

The evaluation matrix in Annex B presents the key issues investigated (relevance, effectiveness, efficiency and economy), the specific evaluation questions addressed, and the corresponding lines of evidence used for each question. The evaluation team developed the questions based on Treasury Board’s Policy on Evaluation, a review of Program documents, and evaluation planning discussions with key Program personnel.

3.3 DATA SOURCES AND COLLECTION

The evaluation drew on the following lines of evidence to gather data:

Review of Program documents

A document and website review covered material from the Ocean Forecasting Program, DFO and other federal government sources.

Scan of web literature

A scan of documents available on the internet was conducted to document how the ocean forecasting function is structured and delivered in other countries – Australia, France, United Kingdom and United States – and to identify studies demonstrating the societal impacts of ocean forecasting products.

Interviews

A total of 32 semi-structured interviews were conducted, either by phone or in person, with key informants from the following groups:

  • Directors General responsible for the Program (n=2)
  • Regional Directors of Science (n=5)
  • Program representatives (from National Headquarters, Pacific, Newfoundland, Maritimes and Quebec Regions) (n=13)
  • Program clients/partners (internal DFO clients, other government departments and international partners) (n=12)

It must be noted that Program partners could not be distinguished from Program clients because in most cases the same individuals and organisations played both roles.

3.4 METHODOLOGICAL LIMITATION AND MITIGATION STRATEGY

The Ocean Forecasting Program is a relatively new (2011-12) program activity in DFO’s PAA. Rather than a well-defined and circumscribed set of activities, it includes a complex array of science research activities conducted in multiple organisations throughout the department (i.e., Hydrographic Products and Services, ISDM, OCB, and DFO regions). The Program’s operating context is characterized by collaborative, interdependent, and multi-sectoral working networks. Although a PM strategy has been developed for this Program, it is still missing a precise articulation of how the various activities are expected to lead to intended outcomes, and a description of the roles and responsibilities of the various organisational units involved.

In the absence of a fully articulated program theory, it was difficult for the evaluation to determine the outcomes against which to measure program performance and the means by which to measure such outcomes. To mitigate this limitation, this evaluation used a realist evaluation approach which put particular emphasis on identifying the Program’s main delivery mechanisms. The Program’s performance was measured by examining the extent to which the needs of various users of the Program’s outputs and outcomes were satisfied and the extent to which the outputs and outcomes were used as intended.

4. MAJOR FINDINGS


4.1 CONTINUING NEED FOR THE PROGRAM


Key Findings: There is a continuing need for the Program as marine activities and marine use are increasing, and considering emerging climate change issues.


This Program is the foundation for marine information, including ocean prediction products, services, and research which support a wide range of adaptation strategies that are used to support emergency preparedness (e.g., tsunami warnings, storm surges, ice formation, etc.), adaptation to climate change, search and rescue operations, the mitigation of oil spills, and at-sea operations such as fisheries and offshore energy.

Marine activities and marine use have been increasing over the past decade. Between 2006 and 2008, marine transportation in Canada increased by 41%.  Maritime sector employment also grew by 4% in the same period. Overall, marine transportation remains the backbone of international trade with over 80% of world merchandise trade by volume being carried by sea6. As a maritime nation bordered by three oceans, Canada has a requirement to understand ocean processes in order to provide prediction that will influence and support safe navigation.

The Program’s science efforts are also directed towards the assessment of potential impacts on marine environments, ecosystems, fish, and marine mammal populations. The Program derives ocean and waterway state information in all forms, including water temperature, and chemical composition of the water, and serves as the foundation from which Canada’s scientists predict the impact of these conditions on the biology of water bodies and ecosystems both in terms of the overall health and productivity of Canada’s oceans. With this knowledge, steps can be taken to better assess stocks, manage species at risk, habitat, and plan networks of marine protected areas to the benefit of all Canadians.

The Ocean Forecasting Program also addresses ongoing and emerging climate change issues. Understanding of the role of oceans on the earth’s climate has advanced significantly through years of monitoring, modeling and research. However, emerging issues like ocean acidification and oxygen depletion (dead zones) threaten ecosystems and fisheries, and rising sea levels threaten the economic base of coastal communities (erosion, flooding). Canada’s northern communities will be among the first to be significantly impacted by climate change.

According to several key informants, there is an increasing need for the Ocean Forecasting Program due to changing climate conditions which create the need for real time and accessible ocean data.  For example, the Program is relevant to a wide range of Canadians because climate change is resulting in the relocation of fish populations and altering ecosystems, and may result in the relocation of coastal communities.  The combined value of commercial and recreational fisheries and aquaculture exceeds $5.5 B annually7, and without the knowledge to allow them to adapt, their contribution to Canada’s gross domestic product could be devalued. Further, coastal communities, subject to rising sea levels and erosion, need the Department to develop a capacity for predictions to help community planners adapt.

While ocean monitoring and research have been ongoing since the creation of DFO, a move towards ecosystem-based management and emerging climate related issues has required realignment of priorities.


6 Report on Economic Benefit of Hydrography (cited in the evaluation of Hydrographic Products and Services, 2012-13).

7 http://www.dfo-mpo.gc.ca/stats/stats-eng.htm

4.2 ALIGNMENT WITH FEDERAL PRIORITIES AND DEPARTMENTAL OUTCOMES


Key Findings: The Ocean Forecasting Program is well aligned with federal priorities and contributes to all departmental strategic outcomes.


The Ocean Forecasting Program contributes to the Government of Canada’s Innovative and Knowledge Based Economy outcome and is directly linked to DFO’s strategic outcome of Safe and Secure Waters. The Program’s provision of accurate and timely information pertaining to current conditions and forecasts of Canada’s oceans and significant inland waterways ensures the safety of marine users.

DFO also has legislative mandate, under Section 41 of the Oceans Act, to deliver services for: the safe, economical and efficient movement of ships in Canadian waters; search and rescue in the marine environment; and, marine pollution prevention and response. These mandated services depend strongly on operational oceanography to supply information products for decision-making in support of the departmental strategic outcome of Safe and Secure Waters.

Moreover, the Program also has a measurable impact and contribution to other departmental strategic outcomes. The ability to safely navigate and plan for forecast ocean and waterway conditions plays an important role in enhancing the economic well-being of mariners, fishers, natural resource companies and all those communities that depend on ocean and inland waterways for their livelihood. Based on the Program information, actions can be taken that reduce costs (economic and environmental) and/or maximize the revenue of commercial entities and individuals, thereby demonstrating a clear linkage and contribution of the Ocean Forecasting Program to the strategic outcome of Economically Prosperous Maritime Sectors and Fisheries.

In addition, ocean and waterway state information in all forms including water temperature, and chemical composition of the water, serve as the foundation from which Canada’s scientists predict the impact of these conditions on the biology of water bodies and ecosystems both in terms of the overall health and productivity of Canada’s oceans. With this knowledge, steps can be taken to better manage species at risk, habitat, and plan networks of marine protected areas. Various departmental reports on the state of the oceans8 are important vehicles for disseminating key elements of this information. This contribution by the Ocean Forecasting Program therefore contributes to some degree to the DFO strategic outcome of Sustainable Aquatic Ecosystems.


8 See, for example:, http://www.dfo-mpo.gc.ca/oceans/publications/index-eng.html; http://www.dfo-mpo.gc.ca/Library/345310.pdf; and http://www.dfo-mpo.gc.ca/oceans/publications/soto/index-eng.asp).

4.3 ALIGNMENT WITH FEDERAL ROLES AND RESPONSIBILITIES


Key Findings: The federal government is the most indicated entity to deliver the Ocean Forecasting Program.


Basic oceanographic data underpin departmental research in fisheries, ecosystems, and oceans management. The Ocean Forecasting Program is the only federal program that has the organization and infrastructure to carry out this type of work on a long term basis, or the ability to provide neutral advice to the federal government on ocean variability.  The Canadian model for ocean forecasting is consistent with the worldwide trend to have these services managed by government jurisdiction.

The federal mandate for the Ocean Forecasting Program is found in the Oceans Act. The interconnectedness of the oceans and the requirement for international cooperation in the management of the oceans means that it is a role that can only be fulfilled by the federal government. The Program collaborates extensively with other governments, other departments, and academia.  For example, according to a DFO scientist, 5% of Canadian Argo buoys are launched by organizations or departments other than DFO or the Canadian Coast Guard. DFO scientists rely on expertize from around the world and leverage data from other countries.

Over recent decades there has been a significant increase in well-funded advocacy science at the industry and environmental non-governmental organization (ENGO) levels. The role for government science is to ensure there is impartial, factual information and advice for decision-making. No one is better positioned to do this work, due to long-term research and monitoring demands. There is strong agreement across DFO senior managers and Program staff, as well as external partners and users, that the federal government is the most indicated entity to deliver this program. The private sector is not a feasible alternative to the federal government delivering this program because ocean forecasting activities are too costly for the expected commercial returns and too complex to deliver independently.

In addition, Canadian provinces have limited capacity and mandate, other than in narrow coastal zones. Universities are increasing capacity and provide strong research, but because their programs are not ongoing, they are limited in terms of long-term monitoring, data management, and advice. 

The federal role for activities conducted by Ocean Forecasting Program is clear, with the Oceans Act (1996, c. 31) outlining the mandate to generate knowledge about what is happening in the oceans.

In Section 40, the Minister shall
“…encourage activities necessary to foster understanding management and sustainable development of oceans and marine resources and the provision of coast guard and hydrographic services to ensure the facilitation of marine trade, commerce and safety in collaboration with other ministers of the Government of Canada.”

In section 42, the Minister may
“collect data for the purpose of understanding oceans and their living resources and ecosystems;…(e) carry out investigations for the purpose of understanding oceans and their living resources...”

In section 43 , the Minister
“(a) is responsible for coordinating, promoting and recommending national policies and programs with respect to fisheries science, hydrography, oceanography and other marine sciences;…(c) may provide marine scientific advice, services and support to the Government of Canada and, on behalf of the Government, to the governments of the provinces, to other states, to international organizations and to other persons.”

The Ocean Forecasting Program also plays an important role with respect to the federal response on climate change. There are emerging issues arising which require the Program to assess, predict and adapt to climate change. For example, the rapid developments in Canada’s North rely heavily on ocean data due to the urgent and ongoing need to understand the changing ocean climate.

4.4 PERFORMANCE – EFFECTIVENESS

The performance of the Program was examined in terms of the extent to which the needs of the various stakeholders were satisfied. Recognizing that a program of this complexity cannot be expected to function well for all players all of the time, the evaluation sought to identify which aspects of the Program have worked well, for which groups, and under which circumstances. Similarly, the evaluation identified which aspects of the Program have not worked as well.

4.4.1 ACCESS TO NEEDED OCEAN FORECASTING INFORMATION AND PRODUCTS


Key Findings: The Ocean Forecasting Program has international recognition and clearly contributes to a number of systems.  However, there is unclear definition of products and services, and limited capacity to meet clients’ needs.


What works well?

Overall satisfaction with informal data exchange across and within regions

According to interviews with Regional Directors of Science and Program representatives in the regions, although not perfect, exchange of needed ocean science information between and within regions appears to be generally sufficient to enable regional scientists to perform their duties. Most Regional Directors of Science interviewed are generally satisfied with the support and input they receive from partners within DFO for the conduct of Ocean Forecasting activities in their region. Similarly, findings suggest that regional scientists are satisfied with their ability to obtain data from colleagues through informal channels.

Recognition of expertize and responsiveness of Program staff

The expertise and responsiveness of Program staff is also recognized by Program partners and clients. In particular, international key informants were very positive in their praise for the expertise of their DFO counterparts and the contribution made by DFO to international data sharing and development efforts.

“No problems with getting data from Canada. It is world class, really good.” (International partner/client)

ISDM staff is recognized as being competent and willing to help, and challenges encountered in obtaining needed support from ISDM are attributed to resource and Information Technology (IT) support limitations rather than to the Program staff.

“We want to increase our collaboration in the years to come. We have a trust relationship with ISDM based on our knowledge of the players, their established credibility, and their demonstration of having the tools and capacity to deliver.” (International partner/client)

Canadian Operational Network of Coupled Environmental PredicTion Systems

The collaboration maintained by DFO with Environment Canada and DND through CONCEPTS yields several positive benefits. In partnership with Environment Canada, key DFO scientists are developing the capacity to produce coupled forecasts that combine atmosphere, ice and ocean data. This will contribute to improving the precision and accuracy of both ocean and meteorological forecasts. Several external partners/users of the Program, including DND, are eagerly anticipating the availability of these improved products.

The CONCEPTS partnership also enables the department to have some data collection buoys launched from DND vessels as part of their missions. This allows DFO to expand the occasions and locations where buoys are launched beyond DFO science missions, thereby augmenting the amount and diversity of available ocean data.

In addition, CONCEPT’s early engagement of clients such as members of the oil industry interested in data on the movement of icebergs is used by the Program as a platform to build the Program’s emerging client-service focus, a new orientation that receives strong support from external partners/clients.

“CONCEPTS is the right direction in which to go.” (Program partner/client)

Argo

Canada’s contribution to the Argo ocean climate monitoring system is also among DFO’s flagship initiatives. Recognized as the largest ocean climate monitoring system in the world, Argo consists of an array of over 3,500 free-drifting floats that collect data on ocean temperature and salinity. Data are transmitted in near real-time via satellite. Since 2001, DFO has launched over 350 Argo floats of which approximately 100 are still operating. It has also made a major contribution to the design and management of the international program. The data are used for various purposes such as assessing climate change, improving weather forecasts and developing ocean models. It is also an important source of data for various departmental reporting on the state of the oceans.9

Atlantic Zone Monitoring Program

The data monitoring and observation work of the AZMP, led by DFO’s four Atlantic regions (Newfoundland and Labrador, Gulf, Maritimes and Quebec), is recognized by many as being of high quality. One Regional Director of Science explained that this is attributable to the implementation of a more systematic approach to data collection, improved informatics for modelling, and a larger amount of data collected.

Launched in 1998, the AZMP is a “comprehensive environmental monitoring program” for the purpose of increasing DFO’s capacity “to understand, describe, and forecast the state of the ocean environment and marine ecosystem”.10 Biological, chemical, and physical data are collected and analysed, and the resulting data and products (e.g., science advisory reports, hydrographic data, climate indices) are made available to various clients on the AZMP website, alongside data from other programs (meteorological data, water levels, remote sensing).11

Long Time Series Observation at Ocean Station Papa in Pacific Region

Ocean observations at Ocean Station Papa now span more than 50 years, constituting the world’s premiere open ocean time series. It allows opportunity for more complex measurements (carbon dioxide, particle flux, air/sea gas exchange, acidification) that cannot yet be made by robotic floats. This time series is critical for validation of global climate models and frequently attracts international collaborators. In particular, it informed DFO’s participation in the Metadata Federation Project, which earned participating members the 2009 North Pacific Marine Science Organization (PICES) Ocean Monitoring Service Award (POMA).12

St. Lawrence Global Observatory

The St. Lawrence Global Observatory’s mission is to “promote and facilitate access, dissemination and exchange of electronic data and information about the St. Lawrence system by fostering a networking of data producers, in order to meet their needs and those of their clients, to improve knowledge and to assist decision making.”13

Governed by representatives from academia, non-profit organisations, and various government agencies and departments (including DFO), the Observatory gathers a wide range of data on the St. Lawrence’s ecosystem (e.g., ice coverage, water levels, wind, and water salinity) as well as navigation information such as tides and currents, and provides observations, forecasts, predictions, and data archives. This information is all available on the Observatory’s website in both official languages. The data provided through this website are of quite high quality according to an internal DFO user.

Climate Change Science Initiative

The assessment of the Climate Change Science Initiative, that took place between 2008 and 2012, revealed that in addition to the use of scientific knowledge by internal client program areas within DFO, a number of notable results were also achieved such as: regional climate models being used in the Atlantic and Pacific Oceans; improved knowledge of hypoxia and ocean acidification being applied for management of snow crab; leveraging from other activities within DFO, Environment Canada, academia and international partnership; etc.14 Overall, the Climate Change Science Initiative has succeeded in advancing the Departmental understanding of climate change impacts on Canada’s aquatic resources through the provision of relevant and timely science.

What does not work so well?

Insufficient integration

The key overarching limitation of the Program is the insufficient integration of its multiple and diverse activities. Program activities are conducted in relatively isolated silos in DFO regions, Hydrographic Products and Services, OCB, and ISDM. For one, the Program produces multiple and at times incompatible data and platforms. Quality, precision, and timeliness of ocean science data are inconsistent from one region and one function to another. For example, the Bedford Institute of Oceanography and the Institute of Ocean Sciences have different formats for data: when rolled up, the data format is inconsistent. Also, some data are not accessible to all who may need it: some regions/functions hesitate to share their data for lack of trust in how the data will be managed and the quality maintained. As a result, data produced by the Program are not all available in a central location.

The Program’s fragmentation also limits its ability to identify and meet its client’s needs, there being no single point of entry to the Program for internal or external partners and clients.

“It would be best if the numerical and physical ocean science at DFO were under the same leadership [...] It is fragmented, with no national program, and several different models. It makes it difficult to collaborate effectively because there is no lead contact.” (External partner)

“Collaboration with academics would be greatly improved if we had a clearer definition of the Program (it is currently too fuzzy for potential partners to see the benefits and opportunities for partnering).” (Senior manager)

Furthermore, several Program partners/clients expressed the need for a “central place” or “warehouse” to access all relevant ocean data (bathymetric, satellite, etc.).

“What we need is a national service to be able to work from the same tool set [...] We would like DFO to provide one-stop shopping for physical ocean science analysis, high resolution data that describe the environment, the currents, the best navigational routes.” (Program partner/client)

In contrast, France, the United States, the United Kingdom, and New Zealand, among other countries, all manage their numerical forecast science nationally.

“There is no reason for the science to be any different from one part of a country to another. Regional expertize is important for tailored applications but the systems themselves should be the same across.” (Program partner)

Globally, the Program’s fragmentation affects the Department’s ability to fulfill its responsibility for coordinating national policies and programs with respect to oceanography.

Weak operational oceanography function and capacity

Although a relatively new phenomenon in the field of ocean science, the development of an “operational oceanography” function at DFO is perceived to be weak by Program partners and clients.

Varying definitions of operational oceanography were encountered during data collection.  For the purpose of this evaluation, operational oceanography is defined as a service-oriented approach where the needs of well-identified target users inform decisions on the development and dissemination of specific science products, while factoring in resource and technical limitations. This particular definition was chosen because it speaks directly to the recognized need for better identification of clients, client needs and of specific articulations between those needs and program activities (i.e., is the program producing useful products and are all intended and potential users aware of and benefiting from these products). In a context of limited federal government resources (i.e., successive economic reduction exercises such as strategic and operational reviews), it is important for the program to be able to document and justify allocation of resources to specific activities and a clear articulation of intended products and target clients is a key element of such justification.

Whereas such an operational oceanography approach has been firmly established in the international and Canadian meteorological services community for several years, it is, by all accounts, comparatively new for the world of ocean sciences and DFO is only just starting to explore this orientation. Nevertheless, international comparisons show that other countries are more advanced in this field, if only due to stronger integration of their oceanographic and meteorological science functions. For example, the United States’ National Oceanic and Atmospheric Administration combines atmospheric and ocean predictions.  In this global context, Program partners and clients have expectations of the Program in terms of service-oriented products that it is not yet able to meet.

DFO’s Ocean Forecasting Program does not currently have a clear definition of products or services designed for and offered to users. On the one hand, it has, by Program representatives’ own admission, a limited knowledge and understanding of user needs. On the other hand, several potential users are largely unaware of the Program’s role, utility, and potential, as illustrated by the fact that several of the key informants approached for this evaluation were unaware that they were participants in and/or users of Ocean Forecasting science information products or services.

“Without any clear understanding of what the Ocean Forecasting Program is providing, I don’t believe we’re making use of it.” (Program client)

This limitation is attributed in part to the absence of a client-service approach and dedicated function, but also to the Program’s limited integration.

According to Program interviewees, the number of Program clients is consistently increasing as the need for oceanographic information continues to grow.  There has been a long history of data exchange with a number of Program clients such as academia and other government departments.  However, new and emerging clients such as fishery policy developers and fisheries management are now turning to the Ocean Forecasting Program for information and advice. The diversity of potential users of ocean data exerts pressure on the Program to adapt and improve its data collection and modelling systems in order to meet at times conflicting or competing needs. For example, some changes might be made to improve the precision of some data elements of interest to fisheries management but at the expense of information that is of particular interest to the Canadian Coast Guard. In a context of limited resources and computing power for modelling, the Program must carefully consider which improvements to make to which modelling system because any given change can affect multiple users in different ways.

DFO is perceived by external partners and clients as having the required personnel and expertise to produce physical forecasts but not the required infrastructure and staff capacity to analyse the data and disseminate forecasts and data products. Compared to other countries like France and the United States, DFO is perceived to be missing the computing and data management capacity to implement more advanced models into forecast systems.

It must however be noted that work is underway in the department to develop such capacity, such as the prediction systems being developed within the CONCEPTS Memorandum of Understanding (MOU).

Erosion of Program’s cutting edge positioning and capacity

The above finding might be interpreted in the context of the gradual reduction of resources devoted to ocean science across DFO. This translates into an erosion of DFO’s ability to meet the challenges and opportunities of advancements in technology and innovation in the area of ocean forecasting. A strong majority of senior managers interviewed consider that DFO is either no longer at the front of the pack on ocean science research or currently at risk of losing ground.

“We are no longer competitive on the ocean science research front in terms of our equipment, infrastructure and technology.” (Senior manager)

“We are behind most countries in terms of data networks and high computing systems.” (Senior manager)

This sentiment is also shared by interviewed Program staff, who generally agree that Canada no longer compares favourably to other countries when it comes to ocean science.

“We are no longer considered on the cutting edge. Countries like Norway, Australia, and the United States are far more advanced than Canada.” (DFO Program staff)

Similarly, among some external partners and users there is a perception that successive resource cuts have limited DFO’s ability to keep up with new tools and technologies. This limitation is attributed by Program representatives to diminishing Program resources but also to IT barriers to the adoption of new technologies.

According to several interviewees including external stakeholders, this erosion affects the Program’s ability to interact with international partners and to “pull its weight” on the international oceanography scene. Given the Program’s necessary reliance on other countries to increase its observational capacity (e.g., as part of the RAPID project on the East coast, the United Kingdom government pays the Program to deploy an observational infrastructure that ultimately benefits both parties), the potential loss of such ability constitutes an important risk.

Limited capacity to meet client needs in the Arctic

To date, the Program has limited ocean science research staff dedicated to collecting, monitoring, modelling, and analysing ocean science data in the Arctic. Monitoring remains very limited in terms of breadth and precision. As a result, Canada is heavily dependent on other countries for oceanographic research in the Arctic.

It must, however, be mentioned that DFO and Environment Canada have signed in 2011 a MOU with Natural Resources Canada’s Environmental Studies Research Fund Management Board to improve the accuracy of short-term ice and ocean forecasts in the Beaufort Sea and surrounding Arctic Region in support of environmental protection relating to oil and gas exploration, production and transportation.


9 http://www.meds-sdmm.dfo-mpo.gc.ca/isdm-gdsi/argo/index-eng.html; DFO (2011). Science (Pacific) and DFO Strategic Objectives, presentation to Senior Management, May 2011.

10 http://www.bio.gc.ca/science/monitoring-monitorage/azmp-pmza-eng.php

11 http://www.meds-sdmm.dfo-mpo.gc.ca/isdm-gdsi/azmp-pmza/index-eng.html

12 DFO (2011). Science (Pacific) and DFO Strategic Objectives, presentation to Senior Management, May 2011.; http://www.dfo-mpo.gc.ca/dpr-rmr/2009-10/awr-pxr-eng.html

13 http://ogsl.ca/en/about/mission.html

14 Climate Change Science Initiative, Final Report - http://www.dfo-mpo.gc.ca/science/oceanography-oceanographie/ccsi-irscc-eng.html

4.4.2 USE OF OCEAN FORECASTING INFORMATION AND PRODUCTS


Key Findings: Several examples illustrate the use of some Program information and products but it remains unclear whether all Program outputs are being fully used and for what purpose.  Evidence suggests that use is somewhat limited by potential users’ insufficient awareness of these outputs and by insufficient tailoring of these outputs to intended users’ needs.


The evaluation assessed the extent and circumstances in which clients of the Ocean Forecasting Program used information and products as intended. Intended clients of the Program include, among others: DFO Fisheries Management, DFO Ecosystems Management, the Canadian Coast Guard, other federal and provincial government departments; academia, including several universities in Canada and around the world; international agencies; fishing industry; engineering consultancy groups; oil and gas industry; etc.

The evaluation found several examples illustrating use of Ocean Forecasting information and products. However, in the absence of a comprehensive inventory of intended Program products and services and of a clear articulation of their intended use, the evaluation was unable to determine what proportion of Program information and products are used as intended.

Evidence from interviews and document review demonstrate that Ocean Forecasting products are used for various ocean science research purposes.   For example, a study was recently carried out to better understand the distribution of organisms and pollutants in British Columbia’s Discovery Passage and the Johnstone Strait. The study used oceanographic data to develop a dispersion model that helped scientists estimate how seal lice originating from salmon aquaculture farms may be adversely affecting juvenile salmon migration from the Fraser River to the Pacific Ocean.15

According to a literature review of articles acknowledging the ISDM branch and/or Argo Canada, program data have been quoted a minimum of 161 times in academic journals between 2005 and the first half of 2013.  In this sample, 40 journal articles were authored primarily by a DFO scientist and 19 articles had a DFO scientist as a secondary author. This demonstrates that academia has access to DFO data and uses it for their research purposes. This also speaks to the credibility of such data. 

The Program also provides information in support of safe and secure transportation.  For example, Argo data feed into ocean circulation models which serve the Canadian Coast Guard in search and rescue efforts and potential oil spill response.16 For instance, the Canadian Coast Guard receives oceanographic data that provide speed and direction of ocean surface currents which can help search and rescue personnel conduct operations with greater accuracy. According to Coast Guard representatives, this data allow for more cost-effective search and rescue efforts, and are especially used on the East coast. 

Ocean Forecasting data are also used to support the Department’s work on adaptation and response to changing climate conditions. For example, in 2011, the Program provided advice to DFO’s Science division in the Pacific Region regarding the transport of debris on the West coast of Canada from the March 2011 earthquake and tsunami in Japan. This information responded to information requests from other federal government departments and agencies, the province of British Columbia, and the media on debris generated by the earthquake that may reach Canadian waters and shorelines.17

Ocean data is also used by Environment Canada and DFO scientists to produce a coupled atmosphere-ice-ocean forecasting system that allows better prediction of weather patterns such as snow squalls. According to an Environment Canada scientist,

“[a]tmospheric models that are “uncoupled” with the ocean – where the system does not account for changing ice cover and water temperatures while the forecast model is being run ─  can predict the presence of snow squalls, but typically fall short of predicting the amount of snow from them. When the atmospheric model is run in a coupled system it uses the extra moisture and heat, created in areas of open water, in its calculations. This allows forecasters to more accurately predict the intensity and actual snowfall amount from snow squalls.”18

Based on evidence presented in Section 4.4.1 regarding the Program’s insufficient integration and weak operational oceanography function and capacity, two key factors are found to limit the use of Program information and products. First, potential users are not all aware of the existence of such information and products or of their potential usefulness for their planning, management, operation, adaptation or research activities. Second, some potential users need the Ocean Forecasting information to be packaged or delivered differently in order to be able to use it. As highlighted by one Program staff, the issue of effective knowledge translation (i.e., effective two-way communication between all contributors to and users of Program information and products) is central to the success of the Program.

The Ocean Forecasting Program is very much aware of the challenges surrounding the use of oceanographic information.  The current development of a national program framework is expected to address most of these gaps.


15 M. G.G. Foreman et al (2012) A Circulation Model for the Discovery Islands, British Columbia, Atmosphere-Ocean, 50:3, 301-316.

16 http://www.dfo-mpo.gc.ca/media/infocus-alaune/2012/argo-eng.htm

17 Canadian Science Advisory Secretariat (2012). Transport of Marine Debris from the 2011 Tōhoku Tsunami to the West Coast of Canada, Canadian Science Advisory Secretariat Science Response 2012/006, http://www.dfo-mpo.gc.ca/Library/345699.pdf

18 http://ec.gc.ca/meteo-weather/default.asp?lang=En&n=8075AA19-1

4.4.3  CONTEXTUAL FACTORS


Key Findings: Resource shortages, insufficient IT capability and support, and ageing vessels impact the effectiveness of the Program.


Achievement of Program outcomes can best be understood when situated in the overall Program context. The following factors have a tangible impact on the Program’s ability to achieve its outcomes.

Increasing resource shortages

In the context of cross-government expenditure reductions, the Program has seen its operating, travel, and conference attendance budgets diminish.  Most internal key informants, across regions and organisational units, as well as some external users and partners, have highlighted resources shortages as a key barrier to the effectiveness of the Ocean Forecasting Program. Resource cuts impact the Program’s ability to remain cutting-edge. The absence of budgets to contribute to partnerships diminishes the ability of the Program to leverage resources from external partners such as other governments and academic networks. Similarly, key informants explained that travel restrictions diminish the Program’s opportunities to maintain strategic international partnerships.

As a consequence of budget reductions and hiring freezes, regions have been unable to conduct succession planning for their cadre of expert scientists and technicians. The Program’s ability to deliver, as well as its international credibility and reputation, are highly reliant on its scientists and technicians, many of whom are nearing retirement. This constitutes a significant risk for the department, as noted in the Program’s PM strategy.

Insufficient IT capability and support

The work performed by the Ocean Forecasting Program entails heavy reliance on information technology, be it storage capacity for data, server capacity to process large quantities of data, or specialised software to model, analyse, or share data within and outside the department. Ocean science is a complex discipline that relies by necessity on a wide range of partners and data sources. As such, the ability to exchange data with these partners is essential to the proper functioning of the Program. Furthermore, the availability of some data in real time is essential for several categories of users such as the Argo data being used for improving weather forecasts and for at-sea search and rescue.

By all accounts, insufficient access to IT capability and support is a significant barrier to the effective functioning of the Program across the department.  For example, the department’s access to data from some of its Argo profilers was blocked because of IT security features (they were transmitted from the instrument by e-mail or direct IP Socket connection, but IT Security prevented such server access from within DFO networks). As a result, ISDM had to hire an external company to retrieve, decode, and post the Argo data on a file transfer protocol site. This measure limits the Program’s ability to manipulate the data and makes it vulnerable to the hired company’s periodic system crashes.

In another instance, the Program was unable to obtain a timely response from IT regarding the needed acquisition of extra data storage capacity. At the end of a five-year joint project with Dalhousie University, the Program was asked whether it would accept to archive the data collected as part of that project in order to make it available to the wider research community. It took 1.5 years to negotiate a solution with IT services.  In the end, it was suggested that the Program pay for extra storage capacity.  Since ISDM did not have such a budget, they could not store the data and it was too late for the researchers to find an alternative. This has the potential to affect their collaboration relationship with researchers.

Since 2011, the department must rely on Shared Services Canada for IT infrastructure. The Government of Canada created Shared Services Canada to centralize management of its IT infrastructure. Staff, technology resources, and assets from 43 federal departments and agencies were transferred to the new department, which is now responsible for delivering e-mail, data centre, and telecommunication services. It also provides other optional services to government departments and agencies on a cost-recovery basis.19 A majority of internal key informants, as well as some external users and partners, have mentioned key limitations related to the support provided by Shared Services Canada when it comes to specialised science IT needs. From internal key informants’ perspective, Shared Services Canada staff do not have the specialised knowledge and skills needed (e.g., they are not familiar with the specialised applications used by the Program). 

DFO’s own Information Management & Technology Service (IM&TS) is responsible for application deployment, management, development and security. IM&TS representatives consulted as part of this evaluation have acknowledged the particular challenges of using IT support and infrastructure for science activities in government. In particular, consideration must be given to IT security requirements and all requests must follow decision-making protocols for the allocation of limited IT support resources and IT infrastructure budgets. They added that, while these considerations can limit IT staff’s ability to respond quickly to ad hoc requests, they could be solved through improved forward planning on the Program’s part. This issue has been discussed at the National Science Directors Committee in 2011 with the intention of establishing procedures and behaviours conductive to problem solving.

Ageing research vessels

The Program depends on Canadian Coast Guard vessels (mainly CCGS John P. Tully on the West coast and CCGS Hudson on the East coast) for the collection of an essential portion of its ocean science data. A number of senior managers have mentioned the ageing of Canadian Coast Guard research vessels as a source of preoccupation for the Program. Vessel rust-out is already impacting data quality to some extent (e.g., when a vessel is non-operational for repairs and is unable to retrieve and replace a defective Argo buoy in a timely manner). While alternative technologies for data collection are being explored (e.g., remote sensing satellites and autonomous underwater vehicles), the Program’s need will remain for dedicated science vessels to conduct in-situ measurements to validate, situate, and interpret the other data. While DND vessels are occasionally being used to collect complementary data, the Program cannot rely solely on such “ships of opportunity” because it cannot control when and where the data collection will occur.

These findings echo the Program’s PM strategy, which identified the risk that “Science may not have sufficient resources to invest in the maintenance and control of the infrastructure (i.e., vessels, small crafts, IT infrastructure, laboratories and equipment) that is necessary to achieve Ocean Forecasting Program objectives”. Coast Guard has received funding for the replacement vessel for CCGS Hudson which is part of the National Shipbuilding Procurement Strategy.

Other factors

In addition to the above factors, the Program is also subject to pressures from client expectations in reaction to emerging new technology and platforms, and the development of ocean science by other countries. For instance, the Joint Commission for Oceanography and Marine Meteorology (JCOMM), formed by the World Meteorological Organisation and the Intergovernmental Oceanographic Commission, is progressing towards unified ocean and weather prediction systems with a client services response framework. The range of users and user needs is also evolving, in particular in the Arctic where marine transportation and activities are increasing (e.g., oil rigs, shipping industry going through the Arctic).

On the other hand, current government priorities may have a beneficial impact on the Program. For example, DFO is engaged in the development of a World Class Tanker Safety System, led by Transport Canada in partnership with several federal departments. This initiative, as a secondary benefit, could lead the department in the direction of developing an operational ocean forecasting system aimed at serving various users.


19 http://www.ssc-spc.gc.ca/pages/mndt-eng.html

4.4.4  GOVERNANCE


Key Findings: There is a need for increased national coordination and integration of Program activities.

What works well?

Strong support for the governance structure developed for the Aquatic Climate Change Adaptation Services Program

In 2011-12, the OCB was given the mandate to create a new program, the ACCASP, that is now captured as a separate program activity under the department’s Economically prosperous maritime sector and fisheries strategic outcome in 2013-14. With this change, responsibility for climate change-related oceanographic science was transferred from the Ocean Forecasting Program to the newly created ACCASP. Several key informants among Regional Directors of Science and Ocean Forecasting Program representatives have praised the governance structure established for the new Program. Reasons given for why this is an example of good governance include the following:

  • the decision-making mechanism (Steering Committee) gives a voice to the regions;
  • the Steering Committee also includes representation from the various clients of the Program which allows for a feedback loop to be established;
  • clear selection criteria were established for funding allocation; and
  • information sharing with stakeholders is timely.

Strong support for current national service approach framework development

Program managers in the Ocean Forecasting Program have taken the lead in developing a national service framework for the Program. To date, an informal network of senior scientists in various regions have been engaged in ongoing discussions, a survey of Program clients has been conducted, and a pre-framework document is being drafted. Although not actively involved, Regional Directors of Science are highly supportive of this initiative and several have commented positively on their staff’s relationship with the initiative’s coordinating team in OCB.  According to one key informant (DFO Program staff), some resistance can be expected from Ocean Forecasting Program scientists to the new national framework approach but several are supportive: they see the need and acknowledge the requirement.

What does not work so well?

Absence of effective formal governance mechanism

The main governance mechanism overseeing the Program is the National Science Directors Committee (NSDC), chaired by the Assistant Deputy Minister of Science, with representation from all the Regional Science Directors. According to senior managers, and as confirmed by a review of meeting minutes, the Ocean Forecasting Program has not been discussed much at NSDC meetings since 2009.

Daily Program transactions, however, rely on ad hoc, informal decision-making by Program staff, with an over-reliance on interpersonal relationships across regions and between regions and headquarters (OCB and ISDM). There is no formal mechanism to coordinate cross-functional collaboration at the Program working level. This gap appears to be particularly problematic for ISDM, which must interact with each region for ocean data management. For example, no mechanism exists to decide which data will be posted by ISDM on the AZMP website. As a result, that decision falls back to the judgement of the data management expert as opposed to the content experts. Similarly, there is no formal consultative mechanism to arrive at a decision on the optimal number of Argo profilers to be launched in a given year. DFO stakeholders impacted by this decision are only consulted informally.

Moreover, in the absence of strong management engagement in this Program, it is possible that needed decisions are simply not being made. One key informant contends that the absence of a clear decision on whether or not to move towards an “operational oceanography” mode is a key example of the Program’s weak governance. Another example of slow decision-making impacting the Program: Environment Canada has engaged in discussions with DFO to secure the provision of quality controlled real-time data needed to improve and validate ocean forecast results. A final decision on whether DFO is willing to proceed or not lagged for several months and the contract work was delayed to the end of fiscal year.

Unclear roles and responsibilities

As previously mentioned, there is currently no precise description of the roles and responsibilities of the various organisational units involved in the Ocean Forecasting Program. As a result, some functions at times step on each other’s toes. For example, ISDM and the Maritimes Region both have some ocean data management responsibilities and decisions on who does what are not always clear. Several Program representatives have pointed to difficulties with a lack of clarity on where ISDM’s mandate starts and ends with regards to Ocean Forecasting. A National Science Data Management Committee was created in 2005 to build a national data management Program for data collected by the department’s Science Sector or for the data that it manages on behalf of others. A National Science Data Management Strategy was subsequently developed in 2006. An updated strategy is expected to become available shortly and it is expected to bring some of the needed clarity around ISDM roles and responsibilities.  The recent merging of the OCB and ISDM should also improve this situation as the branch now has reach into all of Ocean Forecasting, but the new branch thus formed is still within the Canadian Hydrographic Services directorate, and therefore still requires coordination with the regions.

Lack of clarity has an impact on how the Program is perceived by external partners and clients. For example, an international partner shared an illustrative anecdote. Upon hearing, at an international conference, about some interesting ocean science activity being conducted at DFO, he approached the scientist in DFO he knew the best, asking to be introduced to the lead scientist in charge of that activity. His DFO interlocutor was unable to point him in the right direction, being unaware of the activity in question.

Need for national coordination and integration

The departmental matrix management structure imposes some limitations to integrated decision-making for any DFO Program. The great majority of science work for Ocean Forecasting is done in the regions. The main role of the Regional Director Generals is to set regional priorities for Ocean Forecasting Program science. Regional priorities affect the types of projects Program scientists work on (e.g., in Pacific Region, focus is on aquaculture and salmon-related science because those are high regional priorities whereas in Quebec Region focus may be on determining and forecasting water levels in the St. Lawrence River to support shipping). Otherwise, governance of the Program in the regions is ensured by the Regional Director of Science who sits on the NSDC. At National Headquarters, the Hydrographic Products and Services, OCB and ISDM (the latter two having been recently merged) are each responsible for some aspects of the Program. Responsibility thus being split between these various organisational units, there is no formal lead for the Program, which hampers the development of a national approach. It must however be noted that, according to several key informants, this situation has begun to improve at the working level with the development of a national service framework.

No consensus on the ideal governance mechanism

Several senior managers think that the NSDC is an effective and appropriate governance mechanism and that once the Ocean Forecasting Program has a framework, it will be able to engage and use the NSDC effectively for decision-making. Others contend that the dedicated governance structure established for the ACCASP (a steering committee and a technical committee) is a best practice but are reluctant to suggest creating yet another governance mechanism for the Ocean Forecasting Program and think that the NSDC would be sufficient and appropriate. Still, an argument was made for creating a dedicated structure that, like the ACCASP or the Canadian Science Advisory Secretariat, would bring Ocean Forecasting Program clients at the table: there is value in creating a formal connection with Program users. Such formal connection does not exist with the NSDC since users are invited on an ad hoc basis only.

One senior manager pointed out that one must however be careful in expecting the same success from the Ocean Forecasting Program as was obtained with the ACCASP because its success stems not solely from the collaborative approach taken and the type of governance structure established, but also from the powerful lever that comes with dedicated funding. The Ocean Forecasting Program does not have funding to allocate per se. Without it, Program leaders may lack sufficient influence.

4.4.5 Performance Measurement


Key Findings: Due to the absence of a clearly articulated and integrated national framework for the Program, it has been difficult to develop a good performance story.

Current measurement and reporting of the Program’s performance is quite limited. Since the Program’s formal inception in 2011-12 some data has been collected and reported in the Departmental Performance Report but such reporting covers almost exclusively data management products and services.

The following table shows the Performance Measurement Framework (PMF) indicators that have been tracked for subsequent reporting in Departmental Performance Reports.

Table 3. PMF Indicators for Ocean Forecasting

PMF Indicators for Ocean Forecasting
2011-12 Result Statement Performance Indicator
Expected Result Canadians have access to oceanographic data and ocean predictions to inform them on the physical and biochemical state of Canadian oceans. Yearly additions to physical oceanographic archives are distributed weekly to national and international sites.
Output Tidal and water level information for Canada’s waterways % of time that tide and water-level gauges are functioning and transmitting data
Output Oceanographic physical and biochemical data received, processed and disseminated. % of new data added to databases, compared to the volume of data received in the year
Output Oceanographic physical and biochemical data received, processed and disseminated. Percentage of requests for data fulfilled
2012-13 Result Statement Performance Indicator
Expected Result Canadians are informed on the current and future physical and biochemical state of Canada's oceans and waterways Percentage of scientific publications by DFO in the field of oceanography compared to the Canadian total in the same field
Output Oceanographic physical and biochemical data Percentage of new data added to databases, compared to the volume of data received in the year
Output Oceanographic physical and biochemical data Percentage of requests for oceanographic data completed in the time required

According to senior Program managers, available data do not cover the entire range of Program activities and is not a good reflection of the overall Program’s performance. The issue is not with data collection but with the articulation of appropriate and sufficient performance measures/indicators. Technical difficulties in defining and measuring adequate performance measures are compounded by the absence of a clearly articulated and integrated national framework for the Program. Without such a framework, it has been difficult for the Program to develop a good performance story.

“It has been hard to represent the breadth of Ocean Forecasting Program activities using so few indicators – the current ones are partial and we are only just now starting to collect data.” (Senior manager)

ISDM has had more success in measuring and reporting on performance than other components of the Program. With the exception of the proportion of scientific publications by DFO in the field of oceanography, all performance indicators recently tracked by the Program pertained to data acquisition and sharing activities.  

The Program has developed in 2012-13 a PM strategy that constituted a first attempt at defining Program activities, intended outputs and outcomes, and how the achievement of these results will be measured on an ongoing or cyclical basis. As noted previously, this Strategy is still missing a precise articulation of how the various activities are expected to lead to intended results, and a description of the roles and responsibilities of the various organisational units involved. And while this Strategy contains more detailed and comprehensive performance indicators than those captured as part of the Departmental PMF, data has yet to be collected and reported on such performance.

4.5 PERFORMANCE – EFFICIENCY AND ECONOMY


Key Findings: The Ocean Forecasting Program leverages additional resources from multiple external sources and uses innovative approaches to increase efficiency.


Current practices

As a result of the work surrounding the development of a national service framework, the OCB was able to document Program budgets and expenditures in each region20.

According to a majority of senior managers and Program representatives from all sectors and regions, the Program is operating with increasingly diminishing resources as a result of attrition.

“The Program is already at a minimum. The reality is that we are already doing as well as we can, given what we have.” (Program representative)

A review of program budgets since 2008-09 has shown that Program resources (both budgets and FTEs) have steadily decreased between 2008-09 and 2010-11. Budget increases are subsequently shown in 2011-12 and 2012-13 but they might possibly be attributed to the allocation of special ACCASP funding. If this hypothesis is true, then budgets have generally remained the same since 2010-11.

Although the actual value has not been calculated, there is clear evidence that the Program is leveraging significant additional resources from multiple external sources.

In addition to international collaborations, the most significant of these sources is the Program’s participation in CONCEPTS, whereby DFO gains access to Environment Canada’s supercomputer and meteorologists and oceanographers’ expertise, as well as DND’s marine vessels for ocean data collection. Considering the high cost of Canadian Coast Guard vessel for research purposes, the use of DND’s boats alone is of significant value. An MOU between Environment Canada, DFO and DND was established as a way to share efforts and resources and to avoid duplication.

Program representatives from both the West and the East coasts mentioned that an unusual (compared to other science sectors) amount of O&M Program funds come from sources external to the department: academic networks, other federal departments, international partners, etc. For instance, the Maritime Region only has A-base funding for monitoring and observation activities; all other ocean science activities are funded from Ottawa science research funding programs (e.g., International Governance Strategy, Climate Change, Ecosystems Science), academia (through research networks), and other federal government funding programs (e.g., Program of Energy Research and Development (PERD)), although the later source of funding has diminished with recent budget reduction exercises.

Similarly, Pacific Region has had, by necessity, to take an opportunistic approach to funding and planning its research activities, by using certain funding streams and other government programs. For example, the Program is leading research conducted at the academically-funded NEPTUNE underwater observatory (Ocean Network Canada). In this enterprise, a DFO scientist who is an adjunct professor at University of Victoria is leading a team that is using money allocated to the University to advance research that is also important to DFO. The project, entitled Towards Real-time Observations and Modeling of Tsunamis and Other Infra gravity Waves off the West Coast of North America, aims at enhancing knowledge and predictive capability for long waves such as tsunamis and other large-scale sea level disturbances in the ocean.21

An example of innovative approach to increase efficiency was provided by the Program in the Newfoundland Region, which collaborated with another DFO sector to put electronic sensors on seals in order to gather data on temperature, depth, salinity, as well as location and seal behaviour. This initiative has the merit of leveraging multiple efforts in order to serve multiple users.

The Argo program is another example of investment that enabled the Program to reduce costs. Estimates indicate that it is much less expensive to use an Argo buoy than to do the same work on a boat, though its sampling is limited to the deep ocean (equal or deeper than 2000 meters) as per program configuration and so cannot substitute coastal monitoring methods.

“Argo is not only a more efficient way of collecting ocean data, it is also extremely cost-effective - about 25 times cheaper per profile than by ship,” says Fisheries and Oceans Canada researcher Dr. Denis Gilbert, who is the national director of Argo Canada and a member of the international Argo steering team. “The program also covers far more ocean than is possible by ship.”22

Possible further improvements

There was no consensus across key informants as to which improvements or which alternative delivery approaches the Program might envisage in order to improve its economy and efficiency. The following are a few avenues mentioned by key informants that might be explored by the Program to further reduce costs and augment returns. The feasibility and potential benefits of implementing these various options could not be assessed within the scope of this evaluation. They are nevertheless reported here as “food for thought” for senior management, should they be interested in exploring such alternatives.

First, stronger national integration of the Program would help reduce potential duplication. While actual evidence of duplication could not be documented as part of this evaluation, some key informants alluded to cases of duplication due to inefficient coordination.

Second, more dedicated IT support would reduce the efforts spent by Program staff in circumventing and compensating for the various data access limitations they face in their daily work. Combined with strengthened data management support services, this would free up regional scientists to focus more on data analysis and less on data management. A few key informants noticed that DFO researchers are drawn into administrative tasks for lack of support infrastructure and that this deters them from their scientific work. In contrast, Environment Canada is perceived to have a better organization to support research because it has dedicated support staff “to take care of operational (non-scientific) tasks.”

Third, the Program’s already extensive leveraging of outside resources might be further increased through the pursuit of additional opportunities for collaboration with stakeholders, however keeping in mind that some seed money might be needed to engage in active networking activities, that the pursuit of some collaborative partnerships may require a minimum financial contribution on the part of DFO, and that such endeavours must be managed to avoid further fragmenting the Program.

All senior managers agree that there are opportunities for increasing exchanges with academics, other federal departments (e.g., Environment Canada) and other sectors within DFO. In its July 19, 2013 draft Ocean Services Pre-Framework, the Program confirmed this observation by concluding that “government, industry, and academia have not worked together as effectively in Canada as in other countries to develop and implement Operational Oceanography. Within Canada and internationally, meteorological agencies are increasingly engaged with oceanographers to deliver better meteorological forecasts.”23

One cost-effective means of offering an ocean science client-service support to users might be to co-locate it with Environment Canada’s client service function in each of its regional offices. 

The Program might also explore means to influence the alignment of academic research with the research needs and priorities of federal government, such as the provision of competitive funding envelopes.

Alternatively, the Program might consider how it might implement a cost-recovery fee structure for some of its services, much as the Canadian Hydrographic Service and the Canadian Coast Guard do. Such opportunities will be more easily identifiable and marketable once a clearer definition of the Program is developed.

Ultimately, the Program will be in a better position to identify means to reduce costs and maximize Program impacts once it has more precisely identified its target clients. This will enable development of tools and products that can benefit more than one user and an improved targeting of user needs.

According to an international key informant, increased collaboration between all Canadian ocean science and meteorological forecasting functions would be beneficial in terms of resource sharing, joint development of instruments, quality control, cost sharing of telecommunications, archiving, etc. This would result in increased access to data and in data that are more responsive to user needs. It would also facilitate adherence to international standards.

Beyond increased integration of current functions, one senior manager contends that the future sustainability of the Program requires that DFO find ways to cluster Program activities in ways similar to Environment Canada’s science laboratories model.


20  However, program budgets and expenditures include special ACCASP funding for years 2011-12 and 2012-13 which can’t be carved out from the regular Ocean Forecasting budget.

21 http://www.neptunecanada.ca/research/research-projects/project.dot?inode=11208

22 http://www.dfo-mpo.gc.ca/science/publications/article/2012/12-13-12-eng.html

23 DFO (2013), Ocean Services Pre-Framework, DRAFT July 19, 2013.

5. CONCLUSIONS AND RECOMMENDATIONS


As a maritime nation bordered by three oceans, Canada has a requirement to understand ocean processes in order to provide prediction that will influence and support safe navigation. As discussed in the relevance section of this report, there is an increasing need for the Ocean Forecasting Program due to changing climate conditions which create the need for real time and accessible ocean data. The Program’s science efforts are also directed towards the assessment of potential impacts on marine environments, ecosystems, fish and marine mammal populations.

The Ocean Forecasting Program is well aligned with federal priorities and contributes to all departmental outcomes. It is the only federal program that has the organization and infrastructure to collect, generate, and manage the basic oceanographic data that underpin departmental research in fisheries, ecosystems and oceans management on a long term basis and the ability to provide neutral advice to the federal government on ocean variability. The Canadian model for Ocean Forecasting is consistent with the worldwide trend to have these services managed by government jurisdiction. Moreover, the interconnectedness of the oceans and the requirement for international cooperation in the management of the oceans means that it is a role that can only be fulfilled by the federal government.

The Program’s performance was difficult to evaluate in the absence of a fully articulated Program theory. Nevertheless, the evaluation found evidence of some high quality outputs produced across the regions, such as CONCEPTS, Argo, Long-term Line P series, AZMP, etc.

The evaluation also found clear evidence that there is a need for stronger national integration of the Program. During the period covered by the evaluation, the Program was fragmented across the various regions and the organizational units responsible for separate components of Program delivery (i.e., OCB, ISDM and Hydrographic Products and Services). It lacks a clear national vision and mandate, clear accountabilities and decision-making structure, and formalized roles and responsibilities. Current decision-making is largely ad hoc and informal. Better integration would lead to more consistent outputs, better outcomes, and increased efficiencies through more effective leveraging of internal resources and external partnerships.

Furthermore, there is a demonstrated need for strengthening the Program’s ability to meet its various clients’ needs in a more service-oriented fashion. The Program currently has an incomplete picture of its potential clients and their needs, and some users of Ocean Forecasting products are either unaware that they are currently using Program outputs, unaware of the potential usefulness of such outputs, or find that the outputs are not presented in a format that is the most useful or accessible to them. Development of a more service-oriented, operational oceanography approach would enable better targeting and streamlining of the department’s efforts in the field of ocean forecasting, and a resulting improved ability for the Program to demonstrate the usefulness of its activities.

Program management is already aware of these needs and an initiative is already underway to develop a national ocean services framework. Successful implementation of this framework must however take into account the contextual factors that limit the Program’s ability to achieve its intended results, namely, insufficient IT capability and support, ageing research vessels, increasing resource shortages, and the resulting difficulties in securing transfer of knowledge as key Program experts retire.

Recommendation 1:  It is recommended that the Program pursue and finalize the development of a national service framework.

Recommendation 2:  Either as part of the framework or of subsequent concept papers, it is recommended that the Program develop a vision and mandate, clear accountabilities and decision-making structure, and formalized roles and responsibilities for the delivery of all the Program’s activities at national headquarters and in the regions.

Recommendation 3:  Either as part of the framework or of a subsequent exercise, it is recommended that the Program identify its target clients and their needs, as well as the products and services the Program intends to and/or currently offers to address those needs.

Recommendation 4:  Either as part of the framework or of a subsequent exercise, it is recommended that the Program develop a risk management strategy, with concrete mitigation actions to address risks related to limited IT support and capability, the ageing of research vessels, Program resource limitations, and expert knowledge transfer.

Recommendation 5: It is recommended that, as part of the development of a national service framework, the Program also update its PM strategy, including a revised description of activities, outputs, expected outcomes, performance indicators, targets and data collection strategy.

ANNEX A. MANAGEMENT ACTION PLAN

management action plan
Recommendation 1

Rationale: The evaluation found clear evidence that there is a need for stronger national integration of the Program and for strengthening the Program’s ability to meet its various clients’ needs in a more service-oriented fashion. Program management is already aware of these needs and an initiative is already underway to develop a national ocean services framework.

Recommendation 1: It is recommended that the Program pursue and finalize the development of a national service framework.

Strategy
The Oceanographic Services Framework is with the Director of OSD and the DG of CHS for review.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Reason for Change in Due Date Output
Final Draft of the Ocean Services Framework March, 2014    
Approval by NSDC June, 2014    
Presentation to Deputy Minister Policy Committee for information and validation September, 2014    
Recommendation 2

Rationale:  The evaluation found that the Program lacks a clear national vision and mandate, clear accountabilities and decision-making structure, and formalized roles and responsibilities across the various regions and the organizational units responsible for separate components of Program delivery. Current decision-making is largely ad hoc and informal. Better integration would lead to more consistent outputs, better outcomes, and increased efficiencies through more effective leveraging of internal resources and external partnerships.

Recommendation 2: Either as part of the framework or of subsequent concept papers, it is recommended that the Program develop a vision and mandate, clear accountabilities and decision-making structure, and formalized roles and responsibilities for the delivery of all the Program’s activities at national headquarters and in the regions.

Strategy
Vision and mandate to be included in the Framework (see Recommendation 1).  Accountabilities and decision-making structure will follow with the development of an Implementation Plan.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Revised Date and Reason for Change Output
Develop accountabilities and decision-making structure as the first components of the Implementation Plan April, 2014    
Presentation of the proposed governance structure to NSDC for approval August, 2014    
Recommendation 3

Rationale:  There is a demonstrated need for strengthening the Program’s ability to meet its various clients’ needs in a more service-oriented fashion. The Program currently has an incomplete picture of its potential clients and their needs, and some users of Ocean Forecasting products are either unaware that they are currently using Program outputs, unaware of the potential usefulness of such outputs, or find that the outputs are not presented in a format that is the most useful or accessible to them. Development of a more service-oriented, operational oceanography approach would enable better targeting and streamlining of the department’s efforts in the field of ocean forecasting, and a resulting improved ability for the Program to demonstrate the usefulness of its activities.

Recommendation 3: Either as part of the framework or of a subsequent exercise, it is recommended that the Program identify its target clients and their needs, as well as the products and services the Program intends to and/or currently offers to address those needs.

Strategy
Preliminary client survey is part of the Framework but future work will be part of the Implementation Plan.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Reason for Change in Due Date Output
Add to the Implementation Plan a section on continual client engagement as well as the maintenance of client contacts, follow up on their needs and satisfaction with the products and services they use May, 2014    
Presentation to NSDC  for information and validation September,  2014    
Recommendation 4

Rationale:  Successful implementation of the national ocean services framework must take into account the contextual factors that limit the Program’s ability to achieve its intended results, namely, insufficient IT capability and support, ageing research vessels, increasing resource shortages, and the resulting difficulties in securing transfer of knowledge as key Program experts retire.

Recommendation 4:  Either as part of the framework or of a subsequent exercise, it is recommended that the Program develop a risk management strategy, with concrete mitigation actions to address risks related to limited IT support and capability, the ageing of research vessels, Program resource limitations, and expert knowledge transfer.

Strategy
Risk analysis for Ocean Forecasting presently under review but a full mitigation plan will be ready in the fall of 2014.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Revised Date and Reason for Change Output
Develop a draft Risk Management Plan, to mitigate risks, including IT risks, as an appendix or chapter to the Implementation Plan October, 2014    
Obtain final approval on Risk Management Plan from NSDC January, 2015    
Recommendation 5

Rationale:  The Program has developed in 2012-13 a PM strategy that constituted a first attempt at defining Program activities, intended outputs and outcomes, and how the achievement of these results will be measured on an ongoing or cyclical basis. This Strategy is still missing a precise articulation of how the various activities are expected to lead to intended results, and a description of the roles and responsibilities of the various organisational units involved.

Recommendation 5: It is recommended that, as part of the development of a national service framework, the Program also update its PM strategy, including a revised description of activities, outputs, expected outcomes, performance indicators, targets and data collection strategy.

Strategy
An update of the existing PM strategy will be undertaken concurrently with the development of the national ocean services framework.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Revised Date and Reason for Change Output
Propose a new Performance Measurement Strategy as a chapter or appendix to the Implementation Plan January, 2015    
Present and have approved by NSDC March 2015    
Presentation of the complete Implementation Plan to Deputy Minister Policy Committee for information and validation June, 2015    

ANNEX B. EVALUATION MATRIX


evaluation matrix
Issue/Question Indicators Data Sources
1. Relevance
1.1 Is there a continued need for the Ocean Forecasting Program?
  • Evidence/demonstration that there is continuing need for the Program
  • Program stakeholders attest to the importance of the Ocean Forecasting Program
  • Document review
  • Key informant interviews
1.2 Is the Ocean Forecasting Program aligned with Government of Canada priorities and DFO strategic outcomes?
  • Degree of alignment of Program objectives with Government of Canada priorities
  • Degree of alignment of Program objectives with DFO strategic outcomes
  • Document review
1.3 Is the Ocean Forecasting Program aligned with federal roles and responsibilities?
  • Demonstrated link with federal legislation or policies
  • Evidence of exclusive or shared federal jurisdiction for the Program
  • Program stakeholder opinions on whether some components of the Program could or should be conducted by other organizations or levels of government
  • Document review
  • Key informant interviews
2. Performance: Achievement of Expected Outcomes
2.1 To what extent and in what circumstances do Ocean Forecasting Program stakeholders have access to the information/products they need in order to fulfill their work responsibilities? (Direct outcome)
  • Stated and documented needs of various Ocean Forecasting Program stakeholders
  • Satisfaction of Program stakeholders with the extent to which their needs are being met
  • Examples of circumstances in which needs are met or not met
  • Document review
  • Key informant interviews
2.2 To what extent and in what circumstances do Ocean Forecasting Program clients use Program information/products as intended? (Intermediate/ultimate outcomes)
  • Program stakeholders attest to which information/products they use and don’t use
  • Program stakeholders attest to which information/products they produce and for what purpose(s)
  • Congruence between actual and intended use
  • Examples of circumstances in which Ocean Forecasting Program information/products are used or not used as intended
  • Document review
  • Key informant interviews
2.3 What factors (internal & external) influence the capacity of the Ocean Forecasting Program to achieve its intended outcomes?
  • Examples from documents and key informants
  • Document review
  • Key informant interviews
2.4 To what extent and in what circumstances is the Ocean Forecasting Program governance effective?
  • Evidence of decision‑making mechanisms being in place
  • Program stakeholder opinions on the adequacy and effectiveness of decision‑making
  • Document review
  • Key informant interviews
2.5 To what extent and in what circumstances is monitoring of the Ocean Forecasting Program’s performance adequate to inform management decisions and accountability reporting?
  • Presence of results statements and performance indicators
  • Availability of performance information
  • Quality of performance information
  • Evidence of reporting on performance
  • Opinions of Program stakeholders regarding quality and usefulness of performance information
  • Document review
  • Key informant interviews
3. Performance: Demonstration of Efficiency and Economy (assessment of program resource utilization)
3.1 To what extent and in what circumstances could the efficiency and economy of Ocean Forecasting Program activities be improved?
  • Description of Program budgets, expenditures and number of FTEs, by year, by component of the Ocean Forecasting Program science production chain
  • Alternatives to current Program activities, organization and/or governance that could yield reduced costs, increased outputs and/or improved outcomes
  • Document review
  • Key informant interviews
  • Literature scan

ANNEX C. PROGRAM LOGIC MODEL


Oceans Forecasting Program Logic Model

oceans forecasting program logic model