EVALUATION OF THE NAVAREA INITIATIVE

FINAL REPORT
JANUARY 23, 2015

EVALUATION DIRECTORATE


Table of Contents

ACKNOWLEDGEMENTS AND ACRONYMS


ACKNOWLEDGEMENTS

The Evaluation Directorate acknowledges all individuals who provided input for this evaluation of the NAVAREA (Arctic Navigational Warning Services) Initiative. In particular, the Directorate thanks key informants in the Coast Guard and Environment Canada who took the time to share information during site visits, teleconferences, and in-person or telephone interviews. We especially appreciate that managers fully explained the technologies and processes involved in broadcasting navigational warnings, provided data, and produced new graphics to illustrate various aspects of the Initiative.

ACRONYMS


List of acronyms
C&A Central and Arctic
GRT Gross Register Tonnage
ITS Integrated Technical Services
MCTS Marine Communications and Traffic Services
METAREA Meteorological Area
NAVAREA Navigational Area

EXECUTIVE SUMMARY


INTRODUCTION

This report presents the results of an evaluation of an initiative intended to meet the international commitment by Canada to broadcast warnings of maritime navigational hazards to designated areas in the Arctic. In accordance with Treasury Board’s Policy on Evaluation, the relevance and performance of the initiative was examined.

The Evaluation Directorate in the Department of Fisheries and Oceans conducted the evaluation. The study covers a four-year period from the launch of the initiative in fiscal year 2010-11 to the end of fiscal year 2013-14.

PROGRAM PROFILE

The NAVAREA Initiative, implemented by the Canadian Coast Guard, provides broadcasts of navigational warnings to vessels in the Arctic Ocean. Broadcasts cover two designated navigational areas, NAVAREAS XVII and XVIII, between western Alaska, eastern Greenland and the North Pole. Warnings are broadcast to ships via the INMARSAT C satellite service. As some areas in the Arctic are out of range of that service, warnings are also broadcast by radio via a high-frequency transmitter in Iqaluit, Nunavut.

Implementing the initiative involved creating and staffing a 24-hour NAVAREA desk at the Marine Communications and Traffic Services (MCTS) Centre in Prescott, Ontario. Officers at this NAVAREA desk receive notices of Arctic marine hazards through a variety of channels, prepare messages about the hazards in an internationally prescribed format, send them to a service provider for uploading to INMARSAT C, and ensure that messages have been broadcast.

The Initiative also aimed to upgrade the high frequency radio transmitting equipment at the MCTS Centre in Iqaluit and signal towers at Resolute, Nunavut.

EVALUATION METHODOLOGY

After considering the levels of funding and risk associated with the NAVAREA Initiative, a small-scale evaluation was deemed appropriate. Rigorous and efficient study of the Initiative proceeded in three stages: evaluation planning, data collection, and analysis.

Planning involved an initial review of key documents, consultations with managers of the Initiative, iterative development of an expanded logic model and preparation of an evaluation matrix. The matrix included the issues to be considered, specific questions to be addressed, sources of information, and data collection methods. Interview guides and other evaluation instruments were prepared.

Data collection involved a site visit to the NAVAREA desk at the MCTS Centre in Prescott, telephone or in-person interviews with six Coast Guard officials and an interview with the coordinator of a parallel initiative in Environment Canada, the METAREA Initiative, which delivers broadcasts of Arctic maritime weather warnings. Information was also obtained from documents, websites and databases.

Analysis involved extracting themes and messages from interview data, preparing descriptive statistics and graphs, and summarizing relevant points from documents. Throughout the process the evaluation team worked closely with managers of the Initiative to exchange information, observations, questions and ideas.

EVALUATION FINDINGS

Relevance

There is a growing need for navigational warnings in the Arctic. As the polar ice cap diminishes and open water increases, vessel traffic in the Arctic appears to be is on the rise and larger vessels are using the Northwest Passage.

The NAVAREA Initiative meets an international commitment by the Government of Canada to provide navigational warnings in Arctic NAVAREAS XVII and XVIII. It also aligns with government priorities with respect to Canadian Arctic sovereignty, protection of the northern environment and economic development.

Warning mariners of navigational hazards is an appropriate role for the federal government. The Canadian Coast Guard, with a mandate to ensure safe and secure waters, is the best government agency for delivery of the service.

Effectiveness

The Initiative met international deadlines for providing service to the Arctic NAVAREAS.1 By July 1, 2010 Canada had an initial operational capacity to broadcast navigational warnings to the Arctic NAVAREAS via satellite. Service was fully operational by the deadline of June 1, 2011. Broadcasts have also been delivered by high frequency radio from Iqaluit since 2010 to ensure coverage of any sections of the two Arctic NAVAREAS not reached by satellite signal.2

Warnings of marine navigational hazards by the Canadian NAVAREA service conform to international standards with respect to format and frequency of broadcast.

Efficiency and Economy

In the first four years of the five-year initiative, the 24-hour NAVAREA service has been delivered within budget.

Upgrading of the radio tower at Resolute, Nunavut, has been completed but the purchasing of new high frequency radio transmitters for installation at Iqaluit MCTS has been delayed.

Key informants indicated that value-for-money was considered throughout the implementation of the NAVAREA desk and coordination function. For the major equipment upgrading, value-for-money has been a major consideration as the Coast Guard and Public Works and Government Services Canada have worked together on the procurement process.


1 Deadlines for implementation of service to the five Arctic NAVAREAS by Canada, Russia and Norway were established jointly by the International Maritime Organization and the International Hydrographic Organization.

2 The INMARSAT C satellite service for the broadcast of maritime safety information uses satellites in a geo-stationary position over the equator. Curvature of the earth prevents their signals from reaching all parts of the Polar Regions. The supplementary land-based broadcasts from Iqaluit MCTS use a technology known as High-Frequency Narrow Band Direct Printing. The broadcast navigational warnings are received and printed by a shipboard device.

RECOMMENDATIONS

The Coast Guard managed the NAVAREA Initiative as an extension of its existing navigational warning broadcast service rather than as a discrete major project. The absence of a project management approach put some elements of the project at risk when the Coast Guard underwent a major reorganization in 2012 and key officials changed position or retired. The first recommendation aims to help the Coast Guard strengthen future governance and management of the NAVAREA service.

Recommendation 1

Assign responsibility for the following activities: documenting the governance structure for the NAVAREA service; assessing resource needs; and, producing a monitoring and evaluation strategy that indicates the frequency of reports as well as the positions responsible for providing, collating and communicating data.

Large ocean-going vessels are required to have equipment that will automatically receive broadcasts of maritime safety information. They are also required to carry manuals covering rules, regulations and guidelines that apply in the NAVAREAS they enter. However, as the Internet is frequently used as an alternate source of navigational information, NAVAREA-related postings on the Web should be easy to access and use. Online information on the NAVAREAS managed by Canada is helpful not only to mariners but also to Coast Guard officials, other government departments and the public. Understanding of Canada’s northern waterways is also enhanced if historical data on vessel traffic is readily available for analysis. A second recommendation is intended to focus attention on online information related to the Canadian-managed NAVAREAS.

Recommendation 2

Improve the accessibility of online information on Canadian-managed NAVAREAS and increase access to relevant data in accordance with federal policy on open government.

There has been good cooperation between the Coast Guard and Environment Canada in implementing the NAVAREA and METAREA Initiatives. For example, the Coast Guard monitored METAREA broadcasts for Environment Canada during the development phase of that department’s initiative. Also, Environment Canada uses the high frequency radio broadcast capability at the MCTS Centre in Iqaluit to supplement its satellite broadcasts of weather and ice warnings. A third recommendation aims to consolidate the partnership between the NAVAREA and METAREA service teams and unite their efforts to get feedback from Arctic mariners to improve their respective broadcasts.

Recommendation 3

Consult with Environment Canada on the creation of an interdepartmental NAV-MET committee and on a joint survey of users of Arctic NAVAREA and METAREA broadcasts.

The evaluation study revealed a general lack of understanding about the purpose and usefulness of monitoring programs and projects. A fourth recommendation aims to address the need for greater awareness of the benefits of ongoing monitoring and evaluation.

Recommendation 4

Develop a plan for increasing awareness and use of program and project monitoring within the Coast Guard including the possibility of embedding basic training in measurement and evaluation in the curriculum of the Canadian Coast Guard College.

1. INTRODUCTION


1.1 PURPOSE OF THE EVALUATION

This report presents the results of an evaluation of the NAVAREA (Arctic Navigational Warning Services) Initiative by the Evaluation Directorate in Fisheries and Oceans Canada. The main objective of the evaluation was to determine the extent to which the NAVAREA Initiative was managed effectively and efficiently and achieved the results that were expected.

1.2 PARAMETERS

The evaluation covers a four-year period beginning with the launch of the Initiative in fiscal year 2010-11 and finishing at the end of fiscal year 2013-14.

Work on the evaluation was commenced in March 2014. A first draft of the evaluation report, including recommendations, was produced in June.

The Canadian Coast Guard implemented the NAVAREA Initiative. The main points of contact for the evaluation within the Coast Guard were the Preparedness and Response Directorate in Ottawa and the Regional Directorate, CCG Programs, Central and Arctic, responsible for the delivery of MCTS activities in that region.

1.3 REPORT STRUCTURE

The Executive Summary provides an overview of the evaluation purpose, methodology, findings and recommendations. Section 2 presents information on the NAVAREA Initiative and Section 3 describes the evaluation methodology. Evaluation findings are presented in Section 4 and conclusions and recommendations in Section 5. Annexes provide information to complement that presented in the body of the report.

2. PROFILE OF THE INITIATIVE


2.1 MANDATE

The mandate of the initiative was to meet the international commitment of the Government of Canada to broadcast navigational warnings to vessels in Arctic NAVAREAS XVII and XVIII. Figure 1 positions the two NAVAREAS serviced by Canada in relation to those in the rest of the world.

Figure 1: The World NAVAREAS

Figure 1

The mandate for the Initiative arose as follows. In 2007, with formal approval from the Minister of Fisheries and Oceans and the Deputy Minister of the Environment, Canadian officials informed the international community of Canada’s willingness to take on responsibility for two of five newly defined Arctic navigational and meteorological areas. In October of that year, the International Maritime Organization confirmed Canada as the country responsible for services to Arctic NAVAREAS XVII and XVIII. The international community later set a firm timetable for implementation of service to the Arctic areas. It required the commencement of test services on July 1, 2010, and an official launch on June 1, 2011.

The NAVAREA Initiative, providing warnings of navigational hazards, aligns with the strategic objective of Fisheries and Oceans Canada for safe and secure waters. In 2010, Fisheries and Oceans Canada was provided with a five-year budget for the Initiative of $8.38 million. The Canadian Coast Guard was assigned responsibility for implementation.

2.2 ACTIVITIES

The two main activities in this Initiative were 1) establishing a monitoring and NAVAREA desk, and a coordination function, and 2) expanding high-frequency radio communications coverage.

Monitoring and NAVAREA Desk, and Coordination

A 24-hour NAVAREA desk was established at the MCTS Centre in Prescott, Ontario, to receive and verify information on navigational hazards in the NAVAREAS, prepare navigational warnings in an internationally accepted format, arrange for broadcast of warnings via satellite, and confirm that the broadcasts had been sent.3

A NAVAREA Coordinator was appointed. That position, based in Ottawa, primarily involved liaising between the international body responsible for the World-Wide Navigational Warning System and the management of the NAVAREA desk. A Deputy NAVAREA Coordinator, based at the MCTS Centre in Sarnia, Ontario, facilitated communication between the NAVAREA desk in Prescott and the MCTS Centre in Iqaluit, Nunavut, which provides a supplementary broadcast service for NAVAREA and METAREA warnings via high-frequency radio during the shipping season.4

High-Frequency Radio Communications Coverage

The second major activity was to expand high-frequency radio coverage in the Arctic. This involved upgrading a receiving tower in Resolute, Nunavut, and replacing out-dated transmission equipment, antennas and software at the MCTS Centre in Iqaluit.


3 Annex A provides examples of marine navigational hazards.

4 In NAVAREAS XVII and XVIII the shipping season usually extends from June to mid November.

2.3 EXPECTED RESULTS

Overall, the expected result of the Initiative is that Canada meets its international commitment to provide navigational warnings to appropriate vessels in the two Arctic NAVAREAS.5

This result contributes to safe and secure waters, a strategic objective of Fisheries and Oceans Canada, as follows. When Arctic mariners have access to information about navigational hazards they can take appropriate precautions. By avoiding hazards, they are more likely to have safe and efficient ocean transit, which contributes to safe and secure waters.


5 The International Convention on Safety of Life at Sea prescribes the size of vessel that must carry equipment capable of receiving satellite broadcasts of maritime safety information.

2.4 LOGIC MODEL AND PERFORMANCE MEASUREMENT

The logic model that guided the evaluation study is attached as Annex B. It provides an overview of the logical linkages between inputs, activities, outputs and various levels of outcome.

The Preparedness and Response Directorate produced a one-page draft report on performance measures for the NAVAREA Initiative, showing data for 2010-11 and 2011-12. It presents measures for the following four indicators: number of broadcasts; complaints received from stakeholders; vessel strikings and groundings as a percent of vessel movements; and, shipping accidents as a percent of vessel movements.

2.5 STAKEHOLDERS

Key stakeholders in the Initiative include the Coast Guard officials directly responsible for implementation activities and the managers to whom they report. Officials responsible for the METAREA Initiative in Environment Canada are stakeholders by virtue of its interdependence with the NAVAREA Initiative. Indirect stakeholders include shipping companies that use routes in the Arctic, ship crews and passengers. The International Maritime Organization has a stake in the effective delivery of service to all world NAVAREAS.

2.6 GOVERNANCE

The Initiative was primarily positioned within the Coast Guard’s existing programs rather than as a specific project. MCTS Central and Arctic was the logical location for the NAVAREA desk. The Coast Guard’s Integrated Technical Services (ITS) was the appropriate organization to handle the high-frequency radio communications upgrading activity. The main coordination function was based at Coast Guard headquarters in Ottawa. A supporting coordination role was established at the MCTS Centre in Sarnia. 

In 2012 the Coast Guard underwent a major reorganization. Currently, governance of the Initiative involves the following Coast Guard organizational entities: National Strategies, Operations, MCTS Central and Arctic and ITS.

The Commissioner of the Coast Guard reports to the Deputy Minister of Fisheries and Oceans. The distributed governance of the NAVAREA Initiative flows down from the Commissioner as follows:

  • For MCTS activities, governance is through the Deputy Commissioner, Operations, through various levels, to the Regional Director C&A, to the Superintendent MCTS, and then to the Officer in Charge of MCTS Prescott.
  • For the equipment upgrade project within ITS, governance flows from the Director General, Integrated Technical Services, through various levels, to the Project Manager.
  • For the NAVAREA coordination function at headquarters, governance flows down from the Director General, National Strategies, and Director General, Operations, to the Director, Preparedness and Response, to the Manager, Client Services and Partner Relations, and thence to the NAVAREA XVII & XVIII Coordinator.6

There was no committee specifically charged with oversight of the Initiative. General governance of the Coast Guard is through its Management Board. The Regional Management Board provides oversight of MCTS C&A and the Technical Executive Board oversees major communications equipment procurement projects.

Internationally, the World-Wide Navigational Warnings Services Subcommittee governs NAVAREA services by setting standards, providing quality control and calling for improvements where necessary. That level of governance flows through the Canadian NAVAREA Coordinator.


6 The NAVAREA Coordinator position now resides in Operations, under the Director of Operational Support.

2.7 RESOURCES

The Government of Canada allocated $8.38 million for implementation of the NAVAREA Initiative over a five-year period commencing in 2010-11. Table 1 indicates the resources available in each fiscal year.

Table 1: Resources Available for the NAVAREA Initiative

Table 1
Fiscal Year Full Time Equivalents Salaries Employee Benefits Operation & Maintenance Capital TOTAL
2010-11 8 $677,250 $135,450 $377,300 $0 $1,190,000
2011-12 8 $677,250 $135,450 $237,300 $0 $1,050,000
2012-13 8 $677,250 $135,450 $233,732 $2,013,568 $3,060,000
2013-14 8 $677,250 $135,450 $233,732 $813,568 $1,860,000
2014-15 7 $574,250 $114,850 $530,900 $0 $1,220,000
Five-Year Total $3,283,250 $656,650 $1,612,964 $2,827,136 $8,380,000

3. EVALUATION METHODOLOGY


3.1 SCOPE

In determining the scope of the evaluation, careful consideration was given to the extent of resources that had been invested in the NAVAREA Initiative and the level of risk involved in its implementation. The Evaluation Directorate decided to use a small-scale evaluation approach, which it has developed, tested and implemented for past evaluations. The approach is rigorous and makes efficient use of data from multiple lines of evidence.

A senior evaluation manager engaged by the Evaluation Directorate conducted the evaluation over a period of nine weeks. Experts within the Directorate critically reviewed drafts of key evaluation documents. Coast Guard officials provided essential assistance for the evaluation by identifying major documents, suggesting key informants, providing feedback on evaluation instruments and reviewing draft reports.

3.2 EVALUATION APPROACH AND DESIGN

The overall evaluation approach involved determining the questions that needed to be answered and collecting data to address them.

Determining the appropriate questions involved two steps. First, a clear understanding of the Initiative was sought through background readings, discussions with managers and iterative refinement of a logic model. Then, starting from the basic evaluation issues specified by the federal Treasury Board, questions were framed to address each issue within the context of the Initiative.

Collecting data involved three further steps. The data required to answer each question was identified along with the sources from which it could be obtained and the appropriate data collection methodology. That step was followed by the design of data collection instruments such as interview guides. Then data was collected by interviewing key informants, importing administrative databases and conducting focused research on the Web.

None of the questions required an experimental design. Using data from multiple lines of evidence wherever possible strengthened the validity and reliability of findings with respect to each question.

3.3 KEY ISSUES AND EVALUATION QUESTIONS

Annex C presents the key issues investigated, specific evaluation questions addressed and potential sources of evidence. The key issues were relevance and performance including effectiveness, efficiency and economy. Specific questions addressed the issues in the context of the NAVAREA Initiative.

3.4 DATA COLLECTION METHODS

The evaluation used the following four methods to collect qualitative and quantitative data: key informant interviews; a site visit for direct observation; analysis of databases; and, review of documents. In view of the small-scale evaluation approach, the number of key informant interviews was limited to seven and only one MCTS Centre was site visited.

3.4.1  Key Informant Interviews

Interviews were conducted with five regional Coast Guard staff, one person from the Coast Guard national office and one person from Environment Canada. Informants occupied the following positions:

  • NAVAREA Coordinator, Canadian Coast Guard
  • Deputy NAVAREA Coordinator, Central and Arctic (C&A) Region
  • Officer-In-Charge, Iqaluit MCTS Centre
  • Officer-In-Charge, Prescott MCTS Centre
  • Regional Superintendent, MCTS C&A
  • Project Manager, Expansion of High-Frequency Radio Coverage
  • METAREA Coordinator, Environment Canada

3.4.2  Site Visit for Direct Observation

A site visit to the MCTS Centre in Prescott, Ontario enabled structured direct observation of the personnel, technologies and processes involved in producing broadcasts of navigational warnings to the two Arctic NAVAREAS.

3.4.3  Administrative Data

Administrative databases provided evidence on expenditures, staffing, lines of reporting, broadcasts, Arctic vessel traffic and NAVAREA web page hits.

3.4.4  Document Review

Document and website reviews examined the following areas:

  • Rationale, mandate, plans and resources for the Initiative
  • The World-Wide Navigational Warnings Service, SafetyNet and the Global Maritime Distress and Safety System
  • Satellite configurations for communicating maritime safety information
  • Reports by the Transportation Safety Board on Arctic marine accidents
  • Other areas related to international navigational warnings

3.5 ANALYTICAL METHODS

Analysis of qualitative data involved transcription from notes, organization of text and then extraction of general themes and specific messages with respect to a given question or set of documents. Quantitative data analysis involved cleaning the source databases, rendering data accessible to computational software, graphing and calculating statistics.

3.6 LIMITATIONS AND MITIGATION STRATEGIES

The evaluation team was not able to interview the chairperson of the international committee that oversees NAVAREA services around the world. This lack of direct international input was mitigated by a statement from a key informant that international monitoring of NAVAREA broadcasts around the world has to date yielded no concerns about service to NAVAREAS XVII and XVIII.

Feedback on navigational warnings is generally difficult to obtain. For example, a recent worldwide survey of mariners on the subject, conducted by the international body responsible for NAVAREA services, obtained a response from only one ship in NAVAREA XVIII. To mitigate this problem of limited direct feedback from mariners, the evaluation team identified and interviewed a Coast Guard official who was in direct contact with captains and crews of vessels working in the Arctic.

The evaluation did not include direct input from a neighbouring NAVAREA service about Canada’s cooperativeness in dealing with areas where messaging may overlap. A request to interview the coordinator of two neighbouring NAVAREAS that are serviced by the United States was not answered. To mitigate this lack of external input, a question on international cooperation was posed in multiple key informant interviews.

4. MAJOR FINDINGS


Findings from study of the Initiative will cover relevance and performance with respect to effectiveness, efficiency and economy.

4.1 RELEVANCE

The evaluation examined whether the Initiative addresses an essential need and is aligned with federal and departmental priorities. It also asked whether the mandate for the Initiative is in keeping with federal responsibilities.

4.1.1  Need for the Initiative

Key Findings

There is a continuing need for broadcasts of navigational warnings in Arctic NAVAREAs XVII and XVIII.

As the polar ice cap continues to recede the Arctic shipping season will lengthen and the extent of open water will likely increase. It is reasonable to predict that the volume of traffic in the NAVAREAS will rise.

There is some evidence that larger ships may be using northern sea routes in the future. In 2011, a large drilling ship, the Corvocado, entered Canadian northern waters. With a Gross Register Tonnage (GRT) of 56,610 it was about four times the size of large vessels typically found in Canadian Arctic waters. In 2012, a large passenger vessel, The World, weighing 43,188 GRT and with 508 people on board, transited the Northwest Passage from west to east. In 2013, the Nordic Orion, the largest bulk carrier ever to transit the Northwest Passage, carried a load of British Columbia coal to Finland via a route across the top of North America.7


7 Data in this paragraph was obtained from a key informant and from an Excel database maintained at the MCTS Centre in Iqaluit. Additional information on the Nordic Orion was found in newspaper articles.

4.1.2  Alignment with Federal Government and Departmental Priorities

Key Findings

Canadian responsibility for navigational warnings in NAVAREAS XVII and XVIII aligns with federal government priorities for Arctic sovereignty, protection of the ocean environment, and economic development. It also aligns with a strategic outcome of Fisheries and Oceans Canada, ensuring safe and secure waters.

If Canada had not accepted responsibility for broadcasts of navigational warnings in Arctic NAVAREAS XVII and XVIII, the United States probably would have. If not the US, then Russia, Norway or Denmark might have become involved. While the broadcasting of maritime safety information is apolitical, Canada’s responsibility for navigational warnings in surrounding Arctic international waters is a natural extension of its desire to maintain sovereignty of its northern territories.

Increased vessel traffic and larger cargos in northern waters lead to a higher risk of environmental damage through oil spills or the sinking of ships. Effective broadcasting of navigational warnings helps mitigate that risk.

The availability of maritime safety information contributes to the feasibility of shipping companies using northern transportation routes that are shorter and hence more economical than other routes by virtue of reduced costs for fuel and personnel.

The NAVAREA Initiative contributes to the provision of safe and secure waters, a strategic objective for Fisheries and Oceans Canada.

4.1.3  Alignment with Responsibilities of Government and the Department

Key Findings

Providing mariners with navigational warnings in NAVAREAS is an international public service and an appropriate responsibility for Fisheries and Oceans Canada.

When Canadian mariners sail any ocean they have free access to maritime safety information transmitted by satellite from NAVAREA services operated by governments around the world. It is appropriate that the Canadian federal government, through the Coast Guard within Fisheries and Oceans Canada, contributes to the international effort to provide this essential public service.

4.2 PERFORMANCE

The evaluation examined the performance of the NAVAREA Initiative with respect to effectiveness, efficiency and economy.

4.2.1  Effectiveness

Delivery of Expected Direct Outcomes

Key Findings

The expected direct outcomes of the Initiative were achieved. Information on navigational hazards was made available to mariners in Arctic NAVAREAS XVII and XVIII and broadcasts met international standards in terms of content and timing.

The Canadian Coast Guard met the international expectation that by June 1, 2010, Canada would have an “initial operating capability” to broadcast warnings of navigational hazards to the two NAVAREAS.8 It also met the requirement to have a “fully operational capability” by June 1, 2011.

Table 2 indicates the number of warnings of marine navigational hazards issued from 2010-11 to 2013-214.

Table 2: Warnings of Navigational Hazards

Table 2
NAVAREA Number of Warnings Issued
2010-11 2011-12 2012-13 2013-14
XVII 112 154 133 106
XVIII 143 193 169 124
Total 255 347 302 230

8 The term is quoted from an official announcement of the five new Arctic NAVAREAS in 2010.

Each warning was broadcast a number of times. For example, in 2012-13 the 302 warnings led to 5,147 broadcasts, suggesting about 17 broadcasts per warning. Warnings of navigational hazards are also posted at the Canadian Coast Guard website.

Managers of the NAVAREA service were asked to comment on the variation in number of Arctic navigational hazards from one year to the next. They indicated that as they accumulate experience with the issuing of warnings they are increasingly capable of distinguishing hazards that need be communicated to ocean-going vessels as NAVAREA messages from hazards that may be communicated over other systems as regular Notices to Shipping.

The logic model for the Initiative indicates an expectation that the broadcasts meet international standards. This outcome was achieved by ensuring that a supervising MCTS officer checked all warnings against the international standard before authorizing broadcast. The Coast Guard followed international guidelines with respect to broadcasting warnings twice daily at prescribed times.9

Indirect Outcomes

Key Findings

There is a clear logical link between making navigational warnings available to mariners and safe and efficient vessel transit. Likewise, there is a clear logical ink between safe and efficient vessel transit and safe and secure maritime environments.

International regulations require that ocean-going vessels over a specified tonnage must carry equipment that will automatically receive warnings of navigational hazards broadcast from INMARSAT-C satellites.10 It is thus reasonable to assume that once a warning is broadcast to a NAVAREA by satellite it is received by ocean-going vessels in the NAVAREA. It is also reasonable to speculate that, upon receiving a navigational warning, the bridge crew will read it and make any appropriate adjustments to the planned route, thus increasing the likelihood of safe voyage.

When asked about the likelihood that warnings of navigational hazards are read, one informant mentioned that upon visiting the bridge of a vessel in the Arctic, he saw a printout of a NAVAREA broadcast open and ready for use. Another informant pointed out that a variety of influences help ensure that broadcasts are read and acted upon. Shipping companies, and their insurers, will expect that captains and crews give top priority to maritime safety issues to protect their investments.

Many key informants pointed out that the logical links between warnings of navigational hazards and outcomes such as safe and efficient vessel transit, and safe and secure waters, are obvious. It was also clear that it is virtually impossible to measure the specific contribution of navigational warnings to the indirect outcomes. One can assess the cost of hypothetical marine accidents that might have occurred had a warning not been issued, but direct measures of the impacts of navigational warnings are elusive.

The Initiative also has an indirect impact on Canada’s international presence. By taking on responsibility for two of the world NAVAREAS and effectively delivering service to them, Canada gains recognition in the international maritime community and the potential to contribute to future development of maritime safety information systems. A spin-off from the Initiative is increased knowledge of satellite broadcast systems, which are expected to radically alter the way maritime safety information is delivered in the future.


9 Detailed guidelines for NAVAREA services are produced by the International Maritime Organization.

10 Ships are also required to carry equipment to receive maritime safety information over other systems such as NAVTEX and High-Frequency Narrow Band Direct Printing.

4.2.2  Efficiency

The evaluation examined whether governance and management processes and systems were appropriate, whether roles and responsibilities were clear, and whether resources were used efficiently.

Governance and Management

Key Findings

The NAVAREA Initiative was governed and managed as part of existing Coast Guard programs rather than as a discrete project. The governance and management structure for the NAVAREA Initiative per se was not clearly articulated.

The Coast Guard has had decades of experience providing maritime safety information in northern Canada in a region known as NORDREG. Figure 2 shows the relationship between NORDREG and the two Arctic NAVAREAS.

Figure 2: NORDREG and NAVAREAS XVII and XVIII

Figure 2

Given the experience of the Coast Guard in delivering marine communications and traffic services in NORDREG, it is understandable that the NAVAREA Initiative would be perceived as an extension of Coast Guard activities in the Arctic rather than as a new project.

Information from key informant interviews suggest that from 2010 to 2013, implementing the NAVAREA desk and monitoring activity was primarily managed by a senior official at Coast Guard headquarters working directly with MCTS personnel in the Central and Arctic region. There appeared to be no substantive documentation of an implementation plan or a solid performance monitoring framework to explain how the Initiative was to be governed or managed. Such documentation would have helped personnel who became involved in the project after a major reorganization of the Coast Guard in 2012 and retirement of the principal manager of the Initiative in 2013.

As indicated in the Program Profile, managers of the NAVAREA Initiative produced a one-page draft document on performance measurement. It presented data for 2010-11 and 2011-12 on four indicators. Two of the indicators, related to vessel mishaps, lay outside the locus of control of the Initiative and did not appear to be useful for managing the NAVAREA service. The two potentially useful indicators were number of navigational hazard warnings issued and number of complaints received.

Data on number of warnings issued for 2012-13 and 2013-14 was produced with some delay during the evaluation study. The user satisfaction indicator, number of complaints, was likely of limited use for management of the NAVAREA service as complaints were unspecified and data was not collected directly from users. Potentially useful indicators such as sources of information on hazards, types of hazard, and the types of ships in the NAVAREAS appear not to have been considered.

The lack of a clear governance and management structure may have contributed to a delay in completing the second major activity of the Initiative, upgrading high-frequency radio coverage at Iqaluit MCTS Centre. It had been expected that new equipment would have been procured before the end of fiscal year 2013-14 but procurement is still in progress.

Roles and Responsibilities

Key Findings

Roles and responsibilities for implementing the Initiative could have been more fully documented.

The evaluation team was provided with three organization charts. One showed the reporting relationship for the Deputy NAVAREA Coordinator in Sarnia. Another showed the reporting relationships for NAVAREA desk supervisors, officers on watch and administrative support at the Prescott MCTS Centre. A third showed the reporting relationship for an electronics specialist. There was no organization chart showing the reporting relationships of the NAVAREA Coordinator or the Project Manager responsible for the high-frequency radio communications upgrading.

It would have been helpful to have a single chart depicting both functional and reporting relationships for all participants in the implementation of the Initiative. Interviews with key informants indicated different perspectives on the personnel structure. Challenges in assembling data for the evaluation suggested that the lack of clarity concerning relationships was a hindrance to effective communication.

It is reasonable to speculate that the roles of persons hired specifically for the Initiative were documented. Interviews with key informants suggested that for persons whose work on the Initiative was only a part of their responsibilities, their precise role in the Initiative was unclear.

Efficient Use of Resources

Key Findings

Establishment and operation of the NAVAREA desk and coordinating function has to date been achieved within the planned budget. The upgrading of high frequency radio communications at Iqaluit has been delayed so associated expenditures have yet to be determined. Sharing of NAVAREA-related Coast Guard services with the METAREA service in Environment Canada has been an efficient use of public resources.

Table 3 indicates that expenditures on implementation and operation of the NAVAREA desk at Prescott over the first four years of the five-year funding period have remained within budget. Over the four years, 2010-11 to 2013-14, 96% of available funds have been expended. Of the budget available for expansion of high-frequency radio communications coverage, only 34% of funds available for 2010-11 to 2013-14 have been expended. The remainder of the budget is being held available for the major procurement project.11

Table 3: NAVAREA Allocations and Expenditures, 2010-11 to 2013-14

Table 3
Activity 1: Monitoring/NAVAREA Desk and Coordination
  2010-11 2011-12 2012-13 2013-14 4-Year Total
Allocated $746,930 $614,695 $604,641 $604,641 $2,570,907
Expended $621,769 $624,820 $612,735 $617,500 $2,476,824
Variance         - $94,083
Activity 2: Expansion of High-Frequency Radio Communications Coverage
  2010-11 2011-12 2012-13 2013-14 4-Year Total
Allocated $109,000 $109,000 $2,122,568 $922,568 $3,263,136
Expended $0 $139,785 $661,831 $303,317 $1,104,933
Variance         - $2,158,203

Most key informants within the Coast Guard indicated that activities within their locus of control had been conducted as efficiently as possible. However, there were also indications that efficiency gains may be possible with a reduction in levels of coordination. As the NAVAREA service is now well established, the need for a Deputy NAVAREA coordinator could be reassessed.

The Coast Guard has shared resources with Environment Canada to support the parallel METAREA Initiative. The Coast Guard monitored broadcasts from Environment Canada during the METAREA development phase. Also, the Coast Guard has made available to Environment Canada its existing high frequency radio transmission capability at Iqaluit MCTS so that METAREA warnings delivered via satellite could be supplemented with high-frequency radio broadcasts.12


11 A Request for Tenders on the major equipment project was posted on the Government Electronic Tendering Service in July 2014.

12 As mentioned earlier, these are not audio broadcasts. The radio signals, analogous to fax or telex signals, are picked up by a shipboard device that prints out the message. The technical term is High-Frequency Narrow Band Direct Printing.

4.2.3  Economy

Key Findings

Value-for-money appears to have been given due consideration in implementation of the NAVAREA Initiative.

Key informants indicated that wherever possible, value-for-money was considered. For the expansion of high frequency radio coverage, value-for-money considerations involved due diligence in the letting of contracts for technical advice and for installation of new radio towers at Resolute. It is reasonable to expect that Public Works and Government Services Canada and the Coast Guard will give high priority to consideration of economy in the process of procuring new equipment for installation at Iqaluit MCTS.

INMARSAT is the only satellite telecommunications company approved by the International Maritime Organization to provide international SafetyNET services to world NAVAREAS13. There were thus no options for a value comparison of satellite company.


13 SafetyNET is an international automatic direct-printing satellite-based service for the promulgation of Maritime Safety Information, navigational and meteorological warnings, meteorological forecasts, Search and Rescue information and other urgent safety-related messages to ships. (From page 3 of the International SafetyNET Manual, 2011 Edition, from the International Maritime Organization)

5. CONCLUSIONS AND RECOMMENDATIONS


5.1 CONCLUSIONS

The findings from this evaluation study suggest that the NAVAREA Initiative is relevant and effective.

The Initiative responds to an important and growing need for maritime safety information in the Arctic. It fits with Government of Canada priorities with respect to Arctic sovereignty, environmental stewardship and economic development. Warning mariners of navigational hazards in the oceans is an international public service and the Canadian Coast Guard is the most appropriate agency for delivering broadcasts of navigational warnings to Arctic NAVAREAS XVII and XVIII.

The Initiative effectively met Canada’s international commitment to broadcast navigational warnings via INMARSAT C satellite to vessels in the NAVAREAS. Bulletins were reviewed against international standards with respect to content and format. They were delivered twice daily, year round, from a dedicated 24-hour NAVAREA desk.

The Coast Guard managed the NAVAREA Initiative as an extension of existing programs rather than as a unique project. This approach may have contributed to a lack of clarity with respect to governance and management. Roles and responsibilities could have been more clearly articulated.

To date, the NAVAREA desk and coordination functions have been delivered within the assigned budget. The upgrading of high frequency radio communications coverage at Iqaluit MCTS Centre has been delayed but the Coast Guard and Public Works and Government Services Canada have recently issued a call for tenders.

5.2 RECOMMENDATIONS

The evaluation study leads to recommendations that aim to enhance future management of the NAVAREA service, improve online accessibility of information, focus attention on partnering with the METAREA service provided by Environment Canada, and increase the utility of monitoring and evaluation within the Canadian Coast Guard.

A first recommendation arises from the observation that in the first four years of the Initiative there has been a lack of clarity around governance and management of the NAVAREA service.

Recommendation 1

Assign responsibility for the following activities: documenting the governance structure for the NAVAREA service; assessing resource needs; and, producing a monitoring and evaluation strategy that indicates the frequency of reports as well as the positions responsible for providing, collating and communicating data.

A second recommendation acknowledges the growing use of online maritime safety information and the trend to making publicly funded databases, such as the Coast Guard’s Vessel Traffic database, readily available to the public.

Recommendation 2

Improve the accessibility of online information on Canadian-managed NAVAREAS and increase access to relevant data in accordance with federal policy on open government.

A third recommendation is for a formal recognition of partnership between the NAVAREA service provided by the Canadian Coast Guard and the METAREA service provided by Environment Canada.

Recommendation 3

Consult with Environment Canada on the creation of an interdepartmental NAV-MET committee and on a joint survey of users of Arctic NAVAREA and METAREA broadcasts.

A final recommendation was prompted by observation of a general lack of understanding among key informants in the Coast Guard about using program performance monitoring to enhance program effectiveness. Performance measurement and evaluation appeared to be viewed as a paper exercise rather than as a regular examination of data that is useful for program management.

Recommendation 4

Develop a plan for increasing awareness and use of program and project monitoring within the Coast Guard including the possibility of embedding basic training in measurement and evaluation in the curriculum of the Canadian Coast Guard College.

ANNEXES


ANNEX A: EXAMPLES OF NAVIGATIONAL WARNINGS BY TYPE OF HAZARD

  1. Casualties to lights, fog signals, buoys and other aids to navigation affecting main shipping lanes
  2. The presence of dangerous wrecks in or near main shipping lanes and, if relevant, their marking
  3. Establishment of major new aids to navigation or significant changes to existing ones when such establishment or change might be misleading to shipping
  4. The presence of large unwieldy tows in congested waters
  5. Drifting hazards (including derelict ships, ice, mines, containers, other large items, etc.)
  6. Areas where search and rescue (SAR) and anti-pollution operations are being carried out (for avoidance of such areas)
  7. The presence of newly discovered rocks, shoals, reefs and wrecks likely to constitute a danger to shipping, and, if relevant, their making
  8. Unexpected alteration or suspension of established routes
  9. Cable or pipe laying activities, the towing of large submerged objects for research or exploration purposes, the employment of manned or unmanned submersibles, or other underwater operations constituting potential dangers in or near shipping lanes;
  10. The establishment of research or scientific instruments in or near shipping lanes
  11. The establishment of offshore structures in or near shipping lanes
  12. Significant malfunctioning of radio-navigation services and shore-based maritime safety information radio or satellite services
  13. Information concerning special operations which might affect the safety of shipping, sometimes over wide areas, e.g. naval exercises, missile firings, space missions, nuclear tests, ordnance dumping zones etc. It is important that where the degree of hazard is known, this information is included in the relevant warning. Whenever possible such warnings should be originated not less than five days in advance of the scheduled event and reference may be made to relevant national publications in the warning
  14. Acts of piracy and armed robbery against ships
  15. Tsunamis and other natural phenomena such as abnormal changes to sea level
  16. World Health Organization (WHO) health advisory information
  17. Security related requirements

ANNEX B: LOGIC MODEL FOR THE INITIATIVE


annex b

ANNEX C: EVALUATION ISSUES, QUESTIONS AND DATA SOURCES


annex c
Core Evaluation Issue Broad Question Data Source 14
Relevance Need for the programming Is there a continuing need for broadcasts of Arctic navigational warnings?
  • Management
  • Documentation
Alignment with federal government priorities Does the NAVAREA Initiative continue to align with government priorities?
  • Management
  • Documentation
Alignment with federal responsibilities Is the NAVAREA Initiative consistent with federal and departmental roles and responsibilities?
  • Management
  • Documentation
Performance Achievement of expected outcomes To what extent have navigational warnings been provided to Arctic mariners?
  • Management
  • Documentation
  • Observer – QA
  • Partner – EC
  • User
To what extent have monitoring and broadcasts met international standards?
  • Observer - QA
  • Management
  • Partner – EC
  • User
  • Documentation
How plausible are the links between direct outcomes and anticipated longer-term results?
  • Management
  • Documentation
Have there been any unexpected outcomes from the Initiative?
  • Management
  • Staff – Ops
  • Staff – Tech
Efficiency and economy To what extent has the Initiative been implemented as planned?
  • Management
  • Staff – Ops
  • Staff – Tech
  • Documentation
Was the initiative implemented as efficiently as possible?
  • Management
  • Staff – Ops
  • Staff - Tech
To what extent was value for money considered when making expenditures related to the Initiative?
  • Management
  • Staff – Ops
  • Staff - Tech
Is the governance structure for the Initiative clear and effective?
  • Documentation
  • Management
  • Staff – Ops
  • Staff – Tech
Was the management of the Initiative appropriate?
  • Documentation
  • Management
  • Staff – Ops
  • Staff – Tech
Is there clear documentation of systems, processes and reporting requirements for the Initiative?
  • Documentation
  • Management
  • Staff – Ops
  • Staff – Tech
Have partnerships been managed effectively?
  • Partner – EC
  • Partner - Internat
  • Management
  • Staff – Ops
  • Staff – Tech
  • User (CCG)
Is there a more cost-effective way of providing navigational warnings in NAVAREAS XVII and XVIII?
  • Management
  • Staff – Ops
  • Staff – Tech
  • Partner - EC

14 An explanation of the terms in this column is provided at the end of the table.

Key to Data Sources

annex c
Code Term Explanation
Documentation Extraction of information from program documents or databases.
Management Interviews with staff involved in oversight of the NAVAREA Initiative
Observer - QA Interviews with chair of international NAVAREA oversight committee
Partner - EC Interview with manager of the Environment Canada METAREA Initiative
Partner - Internat Interview with an official responsible for a neighbouring NAVAREA
Staff - Ops Interviews with staff involved with day-to-day operation of the NAVAREA desk
Staff - Tech Interviews with staff involved with upgrading radio communication capacity
User Interview with person connected to Arctic mariners

ANNEX D: MANAGEMENT ACTION PLAN


Management Action Plan:
Evaluation Recommendations

Evaluation of the NAVAREA Initiative

Canadian Coast Guard
Project #: 6B169

Submitted for Approval to DEC:  January 23, 2015
MAP Completion Date: March 2016

Revised Completion Date:

1st Status Update:
2nd Status Update:
3rd Status Update:
4th Status Update:

Overview: The Canadian Coast Guard agrees with the intent of the following recommendations for the NAVAREAS initiative.  Moving forward, with the renewal of funding for this initiative, the management thereof will be incorporated into the overall Marine Communications and Traffic Services (MCTS) program.  Therefore, for future evaluations it will be included with the MCTS program.

Management Action Plan
RECOMMENDATION 1
Rationale: A first recommendation arises from the observation that in the first four years of the Initiative there has been a lack of clarity around governance and management of the NAVAREA service.

Recommendation 1: Assign responsibility for the following activities: documenting the governance structure for the NAVAREA service; assessing resource needs; and, producing a monitoring and evaluation strategy that indicates the frequency of reports as well as the positions responsible for providing, collating and communicating data.
STRATEGY
Preparedness and Response Directorate will review and propose NAVAREAS roles and responsibilities in a short, concise management framework document.  This paper will also seek to clarify the assessment and allocation of resources, the need to develop a monitoring framework and the responsibilities around data collection and dissemination.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Develop a short, concise management framework document that defines the roles and responsibilities as defined in the recommendation. March 2015    
Present this document to the Operations Executive Board (OEB), responsible for the operationalization of the NAVAREAS initiative, for review, comments and ultimately buy-in. March 2015    
Communicate the document and results of the OEB conversation to all key players. March 2015    
RECOMMENDATION 2
Rationale: A second recommendation acknowledges the growing use of online maritime safety information and the trend to making publicly funded databases, such as the Coast Guard’s Vessel Traffic database, readily available to the public.

Recommendation 2: Improve the accessibility of online information on Canadian-managed NAVAREAS and increase access to relevant data in accordance with federal policy on open government.
STRATEGY
The Coast Guard will look to improve the accessibility of online information marine safety information, specifically for the Canadian-managed NAVAREAS, thereby increasing access to relevant data for navigational warning broadcasts.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Discuss issue with the Central and Arctic Region webmaster (Mr. Francois Giguère) to determine feasibility of providing this data and review the way forward (noted below). April 2015    
Review information that is publicly available on the national and regional websites for NAVAREAS and NOTSHIPS in general. June 2015    
Identify the server on which the NAVAREAS information resides and establish if there is sufficient space/capacity. July 2015    
Grant authorization to post the NAVAREAS warnings on an accessible website page (national portal, MarInfo, other). September 2015    
Make NAVAREA data user-friendly and compatible with federal policy on open government. December 2015    
Ensure that past NAVAREAS warnings and active ones are easily accessible with the ability to query by number, year, type of warnings, etc. March 2016    
RECOMMENDATION 3
Rationale: A third recommendation is for a formal recognition of partnership between the NAVAREA service provided by the Canadian Coast Guard and the METAREA service provided by Environment Canada.

Recommendation 3: Consult with Environment Canada on the creation of an interdepartmental NAV-MET committee and on a joint survey of users of Arctic NAVAREA and METAREA broadcasts.
STRATEGY
Preparedness and Response will consult with Environment Canada to explore the creation of an interdepartmental NAV-MET committee to develop a joint survey for users of Arctic NAVAREA and METAREA broadcasts.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Preparedness and Response will hold discussions with Environment Canada to explore the creation of an interdepartmental NAV-MET committee to develop a joint survey to users of Arctic NAVAREA and METAREA broadcasts. September 2015    
Interdepartmental NAV-MET committee will be created and a joint survey for users of Arctic NAVAREA and METAREA broadcasts will be drafted. October 2015    
Draft survey will be presented to Environment Canada and the Canadian Coast Guard's respective governance bodies for review/approval (for this initiative, ie. for the Coast Guard it would be Operations Executive Board and Management Board). November 2015    
Survey will be finalized and delivered to the users of the Arctic NAVAREA and METAREA broadcasts. December 2015    
Survey results will be collected. March 2016    
Survey results will be analyzed to identify potential future improvements to the programs. March 2016    
RECOMMENDATION 4
Rationale: A final recommendation was prompted by observation of a general lack of understanding among key informants in the Coast Guard about using program performance monitoring to enhance program effectiveness. Performance measurement and evaluation appeared to be viewed as a paper exercise rather than as a regular examination of data that is useful for program management.

Recommendation 4: Develop a plan for increasing awareness and use of program and project monitoring within the Coast Guard including the possibility of embedding basic training in measurement and evaluation in the curriculum of the Canadian Coast Guard College.
STRATEGY
In discussions with key players, it was determined that the College would limit the exposure and not really hit the most appropriate audience to fully address this recommendation. Instead, National Strategies (Preparedness and Response and Strategic Business Management) will review and propose a wider approach to increasing awareness of program and project monitoring (performance measurement) within the Coast Guard at large, particularly in light of the Department's need to advance the implementation of, and focus upon, Performance Measurement Strategies. This work will also include educational presentations at the College, but not solely. These presentations could occur at such events as Headquarters (HQ) and Regional Town Halls, Regional Management Committee meetings, HQ Management Committee meetings, Training Weeks, Public Service Week events, et cetera.
MANAGEMENT ACTIONS DUE DATE (BY END OF MONTH) STATUS UPDATE: COMPLETED / ON TARGET / REASON FOR CHANGE IN DUE DATE OUTPUT
Review and revise the existing educational Performance Measurement 101 deck, to ensure currency. 30 September 2014    
With various Coast Guard colleagues, inventory appropriate venues to deliver this awareness presentation. February 2015    
Develop a short feedback survey to be able to measure the overall increase of knowledge of performance measurement amongst participants (pre- and post-emails). March 2015    
As a result of the venues’ inventory, and in partnership with regional IBMS Directors and the Coast Guard College, arrange for the delivery of this to-be on-going awareness presentation. Two (2) awareness presentations by the end of March 2016