Canadian Coast Guard Search and Rescue and Canadian Coast Guard Auxiliary Evaluation Report

Final Report
February 2012
Evaluation Directorate

Table of Contents

List of Acronyms

AU
Australia
CCG
Canadian Coast Guard
CCGA
Canadian Coast Guard Auxiliary
CCGC
Canadian Coast Guard College
CCG SAR
Canadian Coast Guard Search and Rescue
DFO
Department of Fisheries and Oceans
DND
Department of National Defence
FTE
Full Time Equivalent
HMCG
Her Majesty's Coast Guard
IRB
Inshore Rescue Boat
JRCC
Joint Rescue Coordination Centre
MCTS
Marine Communication and Traffic Services
MOU
Memorandum of Understanding
MRSC
Maritime Rescue Sub-Centre
NSP
National Search and Rescue Program
NZ
New Zealand
RMAF
Results-based Management Accountability Framework
RNLI
Royal National Lifeboat Institution
SMMS
Search and Rescue Mission Management System
SISAR
Search and Rescue Program Information Management System
SMS
Search and Rescue Management System
UK
United Kingdom
USA
United States of America

Executive Summary

Introduction

This evaluation report presents the results from the Canadian Coast Guard Search and Rescue program and the Canadian Coast Guard Auxiliary program. These programs were scheduled to be evaluated in 2011 – 2012 as per the Department of Fisheries and Oceans multi-year departmental evaluation plan. The evaluation focuses on the core issues in assessing value for money: relevance and performance, including effectiveness, efficiency and economy as per the Treasury Board of Canada Secretariat's Policy on Evaluation (2009). The timeframe for this evaluation covers the period from 2006-07 to 2010-11. The evaluation was conducted by Department of Fisheries and Oceans Evaluation Directorate and was undertaken between February 2011 and January 2012.

Program Profile

Canadian Coast Guard Search and Rescue Coordination and Response

Canada's National Search and Rescue Program was established in 1986 by the Government of Canada.  The federal government is responsible for responding to aeronautical and maritime search and rescue incidents. Provincial / territorial authorities are responsible for all ground search and rescue response, as well as those that occur within inland waters. The Minister of Fisheries and Oceans, through the Canadian Coast Guard is responsible for the maritime component of the federal search and rescue system program.

The Canadian Coast Guard Search and Rescue program involves distress monitoring, communication, and search and rescue operations. The main activities are coordination response, planning, training and exercises. The Canadian Coast Guard Search and Rescue program is delivered with the support of Canadian Coast Guard's operationally capable and ready Fleet, in addition to communications and alerting services, provided through the Marine Communications and Traffic Services program.

The Canadian Coast Guard's Search and Rescue program has the equivalent of 102 full-time equivalents. The budgeted financial resources from 2006-07 to 2010–11 are described in the table below in thousands of dollars.

CCG budgeted financial resources from 2006-07 to 2010–11 in thousands of dollars
2006-07 2007-08 2008-09 2009-10 2010-11
18,478 18,734 19,056 19,652 19,414

Canadian Coast Guard Auxiliary

The Auxiliary was formed in 1978 in an effort to enhance search and rescue coverage and capabilities, and to better coordinate volunteer efforts. The Auxiliary is organized into six federally incorporated, not-for-profit volunteer organizations.  The Auxiliary is a contribution program that complements the maritime search and rescue capability, providing the Canadian Coast Guard's Search and Rescue program with a balance of dedicated maritime rescue vessels and volunteer vessels.  The main activities of the Auxiliary are business planning, coordination and administration; training exercises, and; maritime search and rescue operations.

For fiscal years 2006-07 to 2010-11, approximately $5 million annually was sourced from Department of Fisheries and Oceans existing departmental reference levels, chargeable to Vote 10, Grants and Contributions.

Clients

Clients include any person in distress operating or occupying a vessel or engaging in activities on Canadian territorial waters, within the area covered by the maritime component of the search and rescue system.

Expected Outcome

People in maritime distress are assisted.

Evaluation Methodology

A non-experimental design was used for this evaluation, in which measurements are taken after the program has been implemented with no comparison group.  Evaluators relied on both primary and secondary sources of data to respond to evaluation questions.  Primary sources included interviews, site visits and an on-line survey.  Secondary sources included program documents and databases.  The evaluation triangulated multiple lines of evidence as a means to ensure the reliability of findings and conclusions.  There were minor limitations with respect to the survey data and program databases.  These did not pose any constraints on the findings and conclusions as strategies were put in place to mitigate these limitations.

Evaluation Findings & Recommendations

Relevance

The Canadian Coast Guard's Search and Rescue program is needed largely due to the size of Canada's coastal search and rescue area (5.3 million km2).  Furthermore the program responds to approximately 6000 maritime incidents per year. The Auxiliary is a critical partner and makes valuable contributions to Canadian Coast Guard's Search and Rescue missions.  The Auxiliary participates in over 22.9% of all search and rescue missions, and is in some cases the sole responder. 

The Canadian Coast Guard's Search and Rescue / Auxiliary program's expected result of "assisting people in maritime distress" aligns with the Department of Fisheries and Oceans Strategic Objective of Safe and Accessible Waterways. The Canadian Coast Guard's Search and Rescue / Auxiliary programs aligns directly with the safety component of the government of Canada outcome area of "A safe and secure Canada" where government department program activities are expected to maintain the safety and security of Canada and its citizens.

Canada is a party to the International Convention on Maritime Search and Rescue adopted under the auspices of the International Maritime Organization, and to the Agreement on Cooperation on Aeronautical and Maritime Search and Rescue in the Arctic, concluded between the eight member States of the Arctic Council. The responsibility of the Minister of Fisheries and Oceans regarding the maritime component of the Canadian Search and Rescue program is provided for in section 41 of the Oceans Act, which is enacted by Parliament pursuant to the exclusive legislative jurisdiction of Parliament over navigation and shipping matters, provided by subsection 91(10) of the Constitution Act, 1867. Sections 130 to 133 of the Canadian Shipping Act, 2001 provide more details on Canadian Coast Guard's role in the maritime component of the search and rescue program. The Canadian Coast Guard Search and Rescue program could also be required to respond to "humanitarian incidents" as provided for in the National Search and Rescue Manual.

Effectiveness

Overall, the Canadian Coast Guard's Search and Rescue / Auxiliary programs are achieving their intended outcome.  The Canadian Coast Guard's Search and Rescue program assisted in almost 25,500 maritime incidents while the Auxiliary did so in over 5800 cases over a five year period. On average, the Canadian Coast Guard's Search and Rescue program contributed to saving 97% of lives at risk and the Auxiliary contributed to saving 96% of lives at risk over the past five years. There were just a few areas where the Canadian Coast Guard's Search and Rescue / Auxiliary programs could make improvements in order to enhance the achievement of outcomes.

While the Canadian Coast Guard's Search and Rescue program does engage in planning activities, there is no official strategic approach to national and regional planning. The Auxiliary produces annual Business Plans however they are not strategic planning documents. A strategic approach to planning is a key activity with respect to preparedness. Identifying activities and areas of need, from a strategic perspective, is critical to the success of the Canadian Coast Guard's Search and Rescue program. Since the Auxiliary is an integral component of the Canadian Coast Guard's Search and Rescue program and makes an important contribution to its overall success, identifying activities and areas of need that align with those of Canadian Coast Guard's Search and Rescue program will only further contribute to the overall success of maritime search and rescue. A strategic approach to planning that aligns two key players, such as Canadian Coast Guard's Search and Rescue program and Auxiliary in the delivery of search and rescue services would enhance the preparedness of the Canadian Coast Guard and the success of maritime search and rescue. 

Recommendation #1: We recommend that the Director, Maritime Safety Systems develop a strategic approach to regional and national planning of the Canadian Coast Guard Search and Rescue program and ensure the Canadian Coast Guard Auxiliary Corporations planning aligns with the Canadian Coast Guard's Search and Rescue program planning.

Canadian Coast Guard's Search and Rescue program has various national training requirements and standards but does not have a national standard or approach for conducting large scale planning exercises or the development and dissemination of After Action Reports. Large scale planning exercises were found to be necessary and useful in preparing for a maritime incident. However, the number of exercises, how they should be conducted and with whom has not been identified or standardized. After Action Reports are intended to identify and share lessons learned and best practices for the purpose of contributing to improving preparedness but a national approach to preparing and disseminating these reports has not been developed. The program has not identified a national standard or approach for large scale planning exercises or the development of a process for the production and dissemination of After Action Reports. Developing a national standard in conjunction with a strategic approach to planning would enhance preparedness.

Recommendation #2: We recommend that the Director, Maritime Safety Systems develop a national standard or approach for conducting large scale exercises including the development of a process for the production and dissemination of After Action Reports.

The Canadian Coast Guard's Search and Rescue program currently has only one service standard to measure timeliness of a response and no measures to determine the appropriateness of a response to an incident. The Auxiliary is an integral component of Canadian Coast Guard's Search and Rescue program however they are not able to measure appropriateness as they do not currently have service standards. There is currently no means of measuring or communicating Auxiliary appropriateness of a response or their contribution to overall maritime search and rescue.  Furthermore, the Canadian Coast Guard's Search and Rescue program would benefit from a review of all of its standards to ensure they are measurable and accurately reflect maritime search and rescue performance.

Recommendation #3: We recommend that the Director, Maritime Safety Systems review the current service standards for the Canadian Coast Guard Search and Rescue program for relevance and measurability and determine whether additional service standards are required to appropriately measure the success of the program. In addition, we recommend the development of service standards for the Canadian Coast Guard Auxiliary  that align with the Canadian Coast Guard Search and Rescue program service standards.

Efficiency

Overall, the Canadian Coast Guard's Search and Rescue / Auxiliary programs are appropriately designed to produce outputs.  There were a few areas where the Canadian Coast Guard's Search and Rescue / Auxiliary programs could make improvements in order to enhance the production of outputs.

Roles and responsibilities between Canadian Coast Guard's Search and Rescue program and key delivery partners are not always clearly defined and well understood, particularly with Fleet. Fleet is the primary Canadian Coast Guard on-water responder for maritime search and rescue incidents1. There is currently no clear definition or comprehensive document that clearly defines the reciprocal expectations of the Canadian Coast Guard's Search and Rescue program and Fleet with regards to the roles and responsibilities of each in the delivery of maritime search and rescue.

Recommendation #4: We recommend that the Commissioner of the Canadian Coast Guard formally define Search and Rescue roles and responsibilities between Fleet and Maritimes Services.

The Canadian Coast Guard's Search and Rescue program and Auxiliary performance data do not always support decision-making as a result of limited and inconsistent performance data collection and reporting. Since the Auxiliary is an integral component of the Canadian Coast Guard's Search and Rescue program and makes an important contribution to its overall success, it is imperative that Auxiliary performance data and reporting efforts are aligned with the Canadian Coast Guard's Search and Rescue program. Given the programs have very similar outcomes, the type of information collected should also be very similar. Aligning performance data collection and reporting will only strengthen the overall performance story of the maritime search and rescue system.

Recommendation #5: We recommend that the Director, Maritime Safety Systems revise the Canadian Coast Guard Search and Rescue program Performance Measurement Strategy to accurately measure program performance. In addition, we recommend the Canadian Coast Guard Auxiliary performance data collection and reporting efforts align with the Canadian Coast Guard Search and Rescue program.

Economy

Overall, the Canadian Coast Guard's Search and Rescue / Auxiliary programs are operating in a way that minimizes the use of resources to achieve its intended outcomes.  The Coast Guard's Search and Rescue program leverages volunteers and equipment to deliver maritime search and rescue services generating substantial savings to the federal government.  Historically and as the current evaluation confirms, the Auxiliary makes an important contribution through the provision of thousands of volunteer hours and vessels funded through the contribution agreements at a cost to the federal government of approximately $5 million dollars per year.  There was one area where the Canadian Coast Guard's Search and Rescue / Auxiliary programs could make improvements in order to minimize the use of resources to achieve its intended outcomes.

As for alternative delivery approaches and achieving Canadian Coast Guard's Search and Rescue program results more economically, the evaluation conducted a comparative analysis of the maritime Canadian Coast Guard's Search and Rescue program delivery model to those of the United States of America, the United Kingdom, Australia and New Zealand.  The evaluation found that while all of these countries employ variations of the government funded / volunteer based model, some countries achieve greater degrees of leveraging from the volunteer sector than the Canadian Coast Guard's Search and Rescue program model. 

In recent years, the Canadian Coast Guard has experienced decreased funding and increased operational costs.   It may be beneficial to further examine potential alternative approaches and best practices in other countries, and adapting these to further leverage resources from the volunteer sector.  This may assist the Canadian Coast Guard's Search and Rescue program to achieve search and rescue outcomes more economically. 

1Fleet maintains an operationally ready fleet of 116 vessels, which includes 41 dedicated search and rescue lifeboats stationed strategically throughout the country.

1. Introduction

1.1 Context of the Evaluation

This evaluation report presents the results from the Canadian Coast Guard Search and Rescue Coordination and Response (CCG SAR) program and the Canadian Coast Guard Auxiliary (CCGA) program. As stated in the Policy on Evaluation, all direct program spending such as the CCG SAR program must be evaluated every five years. In accordance with the Transfer Payments Policy and the Policy on Evaluation, the CCGA is defined as an on-going grant and contribution and as such, must be evaluated every five years as well. The Terms and Conditions for the CCGA are scheduled for renewal in fiscal year 2012-2013, and an evaluation is required in 2011-2012 by Treasury Board as part of the renewal process.
These programs were scheduled to be evaluated in 2011 – 2012 as per the Department of Fisheries and Oceans (DFO) multi-year departmental evaluation plan. The evaluations were planned and conducted separately but presented in a combined report to avoid duplication as they share many similar outcomes. Recommendations stemming from the main findings are formulated to allow for improvements to the program where necessary and to inform future decision-making.

1.2 Scope

The evaluation focuses on the core issues in assessing value for money: relevance and performance, including effectiveness, efficiency and economy. It assesses the extent to which the CCG SAR program and CCGA have achieved their outcomes stemming from activities outlined in their respective logic models. The timeframe for this evaluation covers the period from 2006-07 to 2010-11. It is inclusive of the National Capital Region as well as five Regional offices.  The evaluation was conducted by the DFO's Evaluation Directorate and was undertaken between February 2011 and January 2012.

1.3 Structure of Report

This report combines the CCG SAR program and CCGA evaluations into five sections. Section 1 is the introduction which describes the context and scope of the evaluation, followed by Section 2 which describes each of the programs and the context in which their activities take place. Section 3 describes the methodology that is used throughout the evaluation process. In Section 4, main findings and recommendations of the evaluation are presented, while Section 5 draws overall conclusions.

2. Program Profile

2.1 Canadian Coast Guard Search and Rescue Coordination and Response

2.1.1 Background

Canada's National Search and Rescue Program (NSP) was established in 1986 by the Government of Canada. The objective of the NSP is to prevent the loss of life and injury through search and rescue alerting, responding and aiding activities using public and private resources, and reducing the number of incidents through preventative measures. The federal government is responsible for providing primary response to aeronautical distress incidents and maritime search and rescue incidents in the Canadian area of responsibility on the oceans and in Canadian waters of the Great Lakes and St. Lawrence River system. The Minister of National Defence is the lead for the overall search and rescue program. Provincial / territorial authorities are responsible for all ground search and rescue response, as well as those that occur within inland waters.

The Canadian Coast Guard (CCG) is responsible for the maritime component of the federal Search and Rescue (SAR) system which accounts for 85% of SAR incidents in Canada. Services involve searching for, and providing assistance to, people, ships or other craft that are, or are believed to be, in imminent danger.   It should be noted that the CCG's SAR prevention and boating safety program was transferred to Transport Canada's Office of Boating Safety on December 12, 2003.  As a result, the CCG no longer actively engages in SAR prevention activities.

In Canada there are three SAR regions, each associated with a Joint Rescue Coordination Centre2 (JRCC), which are jointly operated by the Department of National Defence (DND) and Canadian Coast Guard personnel, and complemented by two Maritime Rescue Sub-Centres3 (MRSC), which are operated solely by CCG personnel. The JRCC is responsible for promoting the efficient organization of Search and Rescue services and for coordinating the conduct of SAR operations within an associated SAR region. The maritime responsibilities of the MRSCs are the same as the JRCCs, albeit on a smaller scale. The MRSC coordinators must call on either JRCC Trenton or JRCC Halifax to request the assistance of air resources to respond to an incident4. The program provides coordination and response services to increase the likelihood of survival of people at risk of loss of life or injury in the maritime environment by: coordinating and delivering on-water response to maritime search and rescue cases and supporting the safety of life at sea; assisting DND in providing response capacity to aeronautical cases; and managing partnerships essential for the efficient coordination of response services.

The CCG component of the CCG SAR program includes:

  • Primary SAR units (comprised of large CCG vessels, SAR lifeboat stations, and Inshore Rescue Boats); and
  • Secondary SAR units (larger Fleet vessels which have another program as their primary mandate, such as science and a secondary mandate for the provision of SAR).

In addition to the primary and secondary SAR units, CCG also relies on aeronautical resources from DND, the Canadian Coast Guard Auxiliary volunteer response units, and vessels of opportunity.

2The JRCCs are located in Halifax (Maritimes Region), Trenton (Central and Arctic Region) and Victoria (Pacific Region)

3The MRSCs are located in Quebec City (Quebec Region) and St. John's (Newfoundland and Labrador Region). It was announced on June 6, 2011 that the MRSCs would be consolidated into JRCC Halifax and JRCC Trenton in a phased approach throughout 2012-2013.

4While each JRCC/MRSC has the same responsibilities and provide the same services, each must be responsive to the unique demands of their region and the clients therein (geography, climate, seasonality, remoteness, etc.)

2.1.2 Program Activities

The main activities of the CCG SAR program are response coordination; training; exercises and planning (see CCG SAR logic model).

Regional SAR plans are prepared to ensure that the SAR system operates at maximum efficiency. Planning ensures that capable resources are readily available when and where they are most likely to be needed, based on risks in each SAR area. 

To respond to SAR cases, personnel must be prepared to deal with a variety of SAR challenges found in the Canadian maritime environment. Specialized training courses are offered through the Canadian Coast Guard College (CCGC) that allow coordinators to acquire the skills and knowledge needed to effectively operate the rescue centers and ultimately to achieve the core SAR program objective: to save and protect lives.  Operational exercises ensure a high level of readiness and proficiency of the program within the national SAR system, by providing opportunity for crews, rescue centres and key partners (e.g. CCGA, local police) to practice essential skills.

With respect to response coordination activities, when an alert is communicated, a CCG SAR coordinator assumes the role of Search and Rescue Mission Coordinator and is responsible for planning the search and coordinating the rescue. Overall, response coordination involves collecting and distributing essential information concerning a distress situation, assessing environmental factors, tasking resources to vessels or persons in distress, developing search plans, addressing next of kin concerns, communicating with the media and ensuring the successful resolution of maritime incidents where lives are at risk.

2.1.3 Program Partners, Stakeholders and Clients

For the purpose of this evaluation, program partners, stakeholders and clients are defined below.

CCG SAR key delivery partners are those who provide services that assist in the delivery of the CCG SAR program. These include: Fleet Operational Services; Marine Communication Traffic Services (MCTS); CCGC; DND and CCGA.

CCG SAR Stakeholders are entities outside of DFO / CCG that provide support in the delivery of the CCG SAR program. Stakeholders include: other Government Departments, notably DND; Provincial, Municipal and Regional authorities; the United States Coast Guard; and the United States Coast Guard Auxiliary.

Finally, CCG SAR clients include any person in distress operating or occupying a vessel or engaging in activities on the water, within the area covered by the maritime component of the SAR system. Examples include foreigners in Canadian waters and Canadians in foreign waters. Several maritime user groups are dependent on the program for assistance. These include recreational boaters, as well as commercial clients, including shipping, fishing, ferries, and the offshore oil and gas industry as well as the cruise ship industry.

2.1.4 Program Resources/Budget

As per the organizational chart, the CCG SAR program has the equivalent of 102 full-time equivalents (FTE) distributed throughout headquarters and the five Regions during the period of 2005-06 to 2010-115.  The CCG SAR budgeted financial resources6 are described in the table below.

Table 1.0: CCG SAR program expenditures from 2006-07 to 2010-11 ($'000)
2006-07 2007-08 2008-09 2009-10 2010-11
18,478 18,734 19,056 19,652 19,414

5There are an additional 144 FTEs if the seasonal Inshore Rescue Boat program students are included to make the total 246 FTEs

6It is important to note that the FTEs and program expenditures include CCG SAR program resources and does not include the resources provided through Fleet to provide trained crew and equipped vessels to respond to incidents.

2.1.5 Program Logic Model

For the purpose of this evaluation the following logic model was developed.

The CCG SAR Logic Model

2.2 Canadian Coast Guard Auxiliary

2.2.1 Background

The CCGA is a contribution program that supports search and rescue response activities within the CCG SAR program. The CCGA is a non-profit organization dedicated to search and rescue and safe boating activities. Its mission is to provide a National Volunteer Maritime Rescue Service. The CCGA is organized into six federally incorporated, not-for-profit volunteer organizations that parallel the five CCG Regions, and one national organization. Executives from each of the five Regional organizations comprise a sixth federally incorporated national corporation. The national corporation is responsible for supporting the five Regional corporations in the delivery of search and rescue response. Each Regional CCGA corporation is a unique, incorporated entity that delivers services to meet the national CCGA objectives within the Region. The CCGA was formed in 1978-79 in an effort to enhance search and rescue coverage and capabilities, and to better coordinate volunteer efforts. A formal contribution agreement is held with each of the six CCGA Corporations. The Terms and Conditions of the current contribution agreements are in effect until March 2013.

The overall administration and management of the CCGA contribution agreements is managed through the Director General, Maritime Services. Funding from these contribution agreements permits the CCGA to assist CCG in SAR operations as well as cover organizational and administrative costs, training and insurance.

The majority of CCGA members are commercial fishers and recreational boaters who own their vessels and volunteer their time and vessels to assist the CCG SAR program. The remaining CCGA members are volunteers from local communities who crew community-owned dedicated response vessels 24 hours a day, 7 days a week. The CCGA complements the maritime SAR capability, providing the CCG SAR program with a balance of primary maritime rescue vessels and volunteer vessels.

CCGA vessels are authorized to respond to SAR tasking activities by a JRCC or a MRSC. The JRCCs/MRSCs are responsible for the planning, co-ordination and conduct of SAR operations. Once a JRCC/MRSC is notified that a vessel or person(s) is in danger, the SAR coordinator begins to plan and manage the logistics of the rescue. This is when a CCGA member would be tasked if seen as the most effective means of responding to an incident.

2.2.2 Program Activities

The main activities of the CCGA are maritime search and rescue operations, training exercises and business planning, coordination and administration. Business planning, coordination and administration encompass all of the tasks that are required to properly manage the five Regional organizations and the National organization. This includes the development of annual business plans, the recruitment and retention of members, the management of paid staff and the submission of annual reports. Training exercises include the training of volunteers in all aspects of search and rescue. The CCG SAR program cooperates with the CCGA in the provision of training aids, facilities and equipment and in some instances, instructors and participation in joint exercises. Finally, the CCGA responds to maritime SAR incidents when tasked by a JRCC or a MRSC.

2.2.3 Delivery Partners/Clients/Beneficiaries

Delivery partners of the CCGA include the Royal Canadian Mounted Police; the CCG (CCG SAR program and Fleet Operational Readiness) and the DND. Potential clients for CCGA services include domestic and international recreational boaters, commercial vessels, fishing vessels, oil/gas platforms, aircraft, provincial ferries and other persons in distress.

2.2.4 Program Resources/Budget

For fiscal years 2006-07 to 2010-11, approximately $5 million annually was sourced from DFO's existing departmental reference levels, chargeable to Vote 10, Grants and Contributions. Of the current $4.9 M funding level, approximately $3.5 M is for insurance premiums, administrative and organizational costs. Additional costs encompass reimbursement for SAR operations (including insurance deductibles), and training.

Table 2.0: The budgeted financial resources ($ �000s) are described in the table below:
Total Funding Allocation ($000s) 2006-07 2007-08 2008-09 2009-10 2010-11
Total CCGA Funding 4,901 4,901 4,888 4,920 4,920

2.2.5 Program Logic Model

For the purpose of this evaluation the following logic model was developed.

Schema of the CCGA logic model

3. Methodology

This section outlines the scope and methods of our approach, the evaluation design, evaluation questions, analytical methods as well as the limitations of the evaluation.

3.1 Project Management

The evaluation was conducted by an evaluation team within the Evaluation Directorate at DFO. The team engaged the CCG SAR program personnel in preparing a list of documents to review, identifying key informants and stakeholders and reviewing and providing feedback on various documents such as the evaluation plan, interview guides and survey questionnaire.

3.2 Evaluation Approach & Design

A non-experimental design was used for this evaluation, in which measurements are taken after the program has been implemented with no control group. This design was chosen for two reasons; the CCG SAR program and CCGA are national in scope and as such, SAR services cannot be reasonably withheld from any area or region by virtue of its safety oriented intended outcomes; and a control group cannot be reasonably established for the purposes of making comparisons pre- and post-program nor within or across areas or regions. This design is appropriate to examine issues of relevance, efficiency and economy. The evaluation employed a variety of methods (e.g. interviews, document review) where the evidence was triangulated to arrive at valid findings and conclusions.

3.3 Key Issues & Evaluation Questions

The CCG SAR program and CCGA evaluation questions cover both relevance and performance (including effectiveness, efficiency and economy) and can be found in Annex A. The evaluation matrix includes the use of multiple lines of evidence and research methods as a means to ensure the reliability of information and data to be collected.

3.4 Data Sources

3.4.1 Secondary Data Sources

Secondary data sources used in the CCG SAR program and CCGA evaluation include the review of documents and databases. The documents reviewed include public and internal program documents, inclusive of third party studies. International program documentation was also reviewed to make a comparative assessment of comparable SAR delivery models from other countries. The Search and Rescue Program Information Management System (SISAR) along with other CCG SAR and CCGA administrative data were used to capture statistics relating to SAR cases to inform demand for the program services and the achievement of outcomes.

3.4.2 Primary Data Sources

Primary data sources used in the CCG SAR program and CCGA evaluation include key informant interviews and a survey of program personnel and key delivery partners. CCG SAR interviews were conducted with 12 program personnel and 18 partners. Partners included Fleet (5); DND (4); CCGA (6); and MCTS (3). CCGA interviews were conducted with 12 program personnel and 8 CCG SAR partners. In-person interviews were conducted at the National Capital, Pacific and Maritimes Regions. All other interviews were conducted by telephone. Interviews consisted primarily of open-ended questions.

Two online surveys were administered, one to all CCG SAR program personnel and DND partners located at the JRCCs/MRSCs and the other to CCGA volunteers. The surveys were launched in September 2011 and closed in October 2011. A total of 44 CCG SAR and DND personnel completed the survey of an estimated 74 personnel7 for a response rate of approximately 59%. A total of 728 of 3979 CCGA volunteers responded to the CCGA survey for a survey response rate of 18%.

The purpose of the key informant interviews and surveys was to gain a better understanding of the perceptions and opinions of individuals who have a significant role or experience in the design and/or delivery of the CCG SAR and CCGA programs. Interviewees were to self-assess aspects of their program�s relevance and efficiency, as well as assess aspects of the other program�s performance. As part of the analysis of the interview data, the scales in the table below were used to report on the proportion and frequency of responses from respondents offering a particular perspective.

Table 3.0: Proportion and frequency of responses
Proportion Terms Frequency Terms Percentage Range
All Always 100%
Almost all Almost always 80-99%
Many Often, usually 50-79%
Some Sometimes 20-49%
Few Seldom 10-19%
Almost None Almost never 1-9%
None Never 0%

7The total number of CCG SAR and DND personnel who received the survey is unknown as a result of using grouped email addresses for certain JRCCs/MRSCs.

3.5 Methodological Limitations and Challenges

Non-experimental design

The CCG SAR program and CCGA evaluation utilized a non-experimental design. When using this design, it is difficult to clearly measure the net effects of program performance. Since there are no measurements prior to the implementation of the program, or a comparison group against which to assess other plausible causes for the outcome, it is difficult to attribute net effects to the program. The scientific rigour of this design was increased by describing activities, outputs and outcomes through a logic model, enabling evaluators to make causal linkages and logically argue that results can be attributed to the program.

CCGA Survey results

84% of all respondents for the CCGA survey stemmed from two regions of which 62% were from the Pacific Region (n=433) and 22% from the Central and Arctic Region (n=151). The remaining 16% came from Quebec, Maritimes, and Newfoundland and Labrador Regions (n=111). Reporting on the total of all respondents� views may present findings that reflect more the views of the Pacific Region and therefore potentially biased. In order to mitigate this situation, reporting of select survey findings were separated out into three groupings (West, Central and East) so as to illustrate any Regional tendencies.

CCG SAR Database

All aeronautical, maritime, and humanitarian SAR cases are captured in DND�s Search and Rescue Mission Management System (SMMS) database. The maritime SAR incidents are extracted from SMMS and migrated over to SISAR which is the CCG SAR proprietary database. As a result of technical challenges and human errors, incidents nationwide are under reported in the SISAR database. The following represents the total missing cases, by Region from 2006-2010; Quebec 0.9%; Newfoundland and Labrador 2.9%; Central and Arctic 2.7%; Maritimes 4.6%; and Pacific 30.9%. In order to mitigate against the possibility of bias towards favourable cases of reported incidents in SISAR, the evaluation team examined a sample of the missing data and upon consulting with an expert, concluded that the reported cases are not biased.

4. Major Findings

4.1 Relevance

Evaluation Question #1.1: Is there a continued need for the program?

Findings

The Maritime CCG SAR program is largely needed due to the sheer size of Canada�s maritime territory which boasts the world�s longest coastline; longest inland waterway; largest archipelago; greatest tidal range8; and a 5.3 million km2 coastal search and rescue area to which Canada has committed itself to monitor and coordinate responses to incidents9.

An incident is an event where persons are threatened, either immediately or potentially, by some danger and may require some assistance. Incidents of a maritime nature far outnumber all other terrestrial and aeronautical type incidents. In fact, over 75% of all federal SAR10 responses are related to maritime incidents. The graph below11 describes the total number of maritime incidents responded to over the past five years. It illustrates that the potential need to coordinate, investigate and respond to maritime incidents throughout Canada is quite elevated and may be trending slightly upwards over time.

Number of reported maritime incidents per year
Source: SISAR database

The demand for SAR services stems from several sources12. Approximately one in four Canadians own a recreational boat with increasing preference towards ownership of smaller, more powerful water craft. This is largely influenced by changing demographic and economic trends. Recreational boaters are involved in over 60% of maritime incidents. Also, the commercial fishing industry on both coasts is comprised of approximately 50,000 licensed fishing vessels, and have accounted for 23%13 of all maritime incidents.

There are increasing pressures in the Arctic to ensure Canada can respond to SAR incidents. For example, cruise line excursions along the East and West Coast and into the Arctic have increased significantly over the last five years. Furthermore, the Arctic is expected to yield an increase in maritime traffic from various industries, including those exploiting untapped oil and gas resources14. In recognition of this need, Canada ratified the Arctic Search and Rescue Agreement15 which commits Canada to international cooperation on search and rescue incidents that occur in the Arctic. As such, all CCG SAR program key delivery partners interviewed agreed that the CCG SAR program is required to meet the needs of Canadians and/or to meet the requirements of international agreements.

The CCG SAR program relies heavily on its partnerships in order to make it one of the best maritime SAR programs in the world16. There is a need for the CCGA as it is a critical partner which makes valuable contributions to CCG SAR missions. The CCGA participates in over 22.9%17 of all SAR missions, and is in some cases the sole responder. Their presence ensures that there is suitable coverage in some remote areas, or in areas where the CCG SAR program is challenged to provide its own resources. Their services are so valued that the 2007 CCG SAR Needs Analysis18 put forward eight recommendations, requiring the CCG SAR program to recruit new CCGA members in areas where there was a lack of primary SAR response units, notably in the Great Lakes, the St-Lawrence River, and various areas along the coasts of Nova Scotia, PEI, the Lower North Shore of Quebec and Newfoundland. All of CCG SAR interviewees agreed that the CCGA was needed as an essential support service to the maritime component of the SAR system.

Evaluation Questions #1.2: To what extent are the objectives of the program aligned with departmental and government wide priorities?

Findings

The CCG SAR / CCGA Program expected result of �assisting people in maritime distress� aligns with the DFO Strategic Objective of Safe and Accessible Waterways. For example, maritime accidents do occur, necessitating strong capacities for search and rescue from the CCG. Having this capacity ensures that Canadians benefit from safe and accessible waterways19.

The CCG SAR program aligns directly with the safety component of the Government of Canada outcome area of �A safe and secure Canada� where government department program activities are expected to maintain the safety and security of Canada and its citizens. CCG SAR / CCGA also indirectly contributes towards the outcome area of �Strong Economic Growth� in which program activities are expected to create an environment conducive to economic growth and to promote the development of all sectors of the economy. The CCG SAR program supports the maritime economy in general. To illustrate, the fishing sector is crucial for the Canadian economy, as the value of the landed catch is estimated at $1.8 billion annually over the last ten years. CCG SAR supports the fishing industry, which accounts for 23% of all maritime SAR calls.

Evaluation Questions #1.3: Is the current role of the federal government appropriate in delivering the program?

Findings

The CCG SAR mandate is derived from several Acts and international conventions. The Constitution Act, 1867 gives the Parliament of Canada exclusive legislative jurisdiction over matters of navigation and shipping. With respect to coast guard services, the Oceans Act provides (S.C. 1996, c. 31) states that the powers, duties and functions of the Minister of the Department of Fisheries and Oceans extend to the maritime component of the federal search and rescue program. Under the Canada Shipping Act, 2001 (S.C. 2001, c. 26), the Minister may designate persons as Rescue Coordinators to organize search and rescue operations. A Rescue Coordinator may direct any vessel to take part in a search or to otherwise render assistance on being informed that a person, a vessel or an aircraft is in distress or is missing in Canadian waters or on the high seas off any of the coasts of Canada under circumstances that indicate that they may be in distress. Finally, Canada has committed itself to complying with an international SAR plan20, which stipulates that no matter where an accident occurs, the rescue of persons in distress at sea will be coordinated by a SAR organization and, when necessary, by co-operation between neighbouring SAR countries .

The federal government is the legally mandated and internationally recognized authority to provide maritime SAR services. SAR responsibilities lie within a federal jurisdiction. International agreements as well as the various domestic demands for SAR services depend on various public administration apparatuses to oversee the maritime SAR program in Canada21. This supports the view of almost all program personnel and program partners that there is an essential role for the federal government in overseeing and coordinating maritime SAR services in Canada. The CCGA may become involved in a maritime SAR operation, by virtue of a Rescue Coordinator�s ability to involve any vessel to take part in a search, but it is clearly the federal government that is best aligned to oversee maritime SAR operations.

8Source: 2006-07 DFO Departmental Performance Report.

9Source: International Maritime Organization: International Convention on Maritime Search and Rescue

10Source: National Search and Rescue Secretariat � National Strategic Directions 2005-2009.

11Note that data inputted into database is incomplete. Actual number of incidents for some years would be greater.

12Source: 2005 Evaluation of Program Integrity I SAR Component.

13Source: National Search and Rescue Secretariat � National Strategic Directions 2005-2009.

14Source: Senate SCOFO 40-2 Issue 2.

15The Agreement on Cooperation on Aeronautical and Maritime Search and Rescue (SAR) in the Arctic, or in short the Arctic Search and Rescue Agreement, was signed on May 12, 2011 by member states of the Arctic Council, including Canada, Denmark, Finland, Iceland, Norway, Russia, Sweden and the United States.

16Source: CCG Search and Rescue Needs Analysis, 2007.

17Based on CCG SAR program (25491) and CCGA (5838) M1-M4 incidents over the past five years.

18The SAR Needs Analysis studied Canada�s maritime search and rescue (SAR) system over the five-year period 2000�2004 (inclusive). The SAR system was assessed system to identify opportunities for improvement.

19Source: DFO Report on Plans and Priorities, 2010-2011

20Source: International Maritime Organization: International Convention on Maritime Search and Rescue

21Source: Senate SCOFO 40-2 Issue 10.

4.2 Effectiveness

Evaluation Question 2.1: To what extent are the CCG SAR system / CCGA members prepared and available to respond?

Findings

To determine the extent to which the CCG SAR system and CCGA members are prepared and available to respond, the evaluation examined whether planning documents are appropriate and implemented to deliver SAR services; the existence and appropriateness of training and exercises; the availability of equipment and resources to respond to SAR taskings; and specifically for CCGA, the recruitment and retention of members.

Planning

The CCG SAR program does not produce an annual national search and rescue strategic planning document however, the program has several documents that support CCG SAR from a planning perspective. For example the 2009-2012 CCG Business Plan includes a high-level summary of the program outlining accomplishments, future efforts as well as key initiatives. The National Search and Rescue Manual is developed under the joint authority of National Defence and the CCG. The purpose of the manual is to assist all federal departments and agencies involved in SAR to meet the SAR objectives of the Canadian Government. CCG SAR program also works in collaboration with Fleet to draft an annual Fleet Operational Plan for each fiscal year to ensure adequate resource coverage. Each JRCC has a Major Marine Disaster Contingency Plan. The purpose of these plans is to supplement the National Search and Rescue Manual with detailed information specific to a SAR region to provide guidance and information to all agencies that could be involved in a major maritime disaster. The Levels of Service document outlines six main services provided by the program and four main associated service standards. Several studies22 have also been conducted to support the improvement of program management and planning processes. Key delivery partners, such as Fleet, DND and CCGA, however, noted a lack of involvement in the CCG SAR planning process23.

All six CCGA Corporations submitted an annual Business Plan during the time period of the evaluation that met the minimum requirements set out in the Contribution Agreement. The purpose of the Business Plans is to identify all CCGA activities and related expenses for approval for funding under the Contribution Agreement. The intended use of these plans is for CCG and CCGA officials to determine and allocate funding to the six CCGA Corporations. There was no evidence to suggest that CCG and CCGA officials are using the annual Business Plans as intended, considering that the allocation of funding has generally remained the same for all six CCGA Corporations over the past five years. In addition, the content of the business plans did not vary significantly from year to year, as the activities and spending generally remained the same over the period of the Contribution Agreement. With respect to the format of the Business Plans, there was no common look and feel to the plans and the content of the plans varied significantly from Region to Region as did the depth and breadth of the information and performance data included. The plans did not demonstrate sufficient detail for decision making and lacked in-depth information regarding the intended use of the funds by category and timeframe24.

Key delivery partners, such as CCG SAR program personnel stated the official role of CCG SAR in the CCGA planning process is unclear. There appears to be no formal approach to the CCGA planning process and as a result, the Business Plans varied from Region to Region and lacked a national approach consistent with the CCG SAR program.

While documents and studies exist to support CCG SAR planning, there is no official strategic approach to national and regional planning. The CCGA produces an annual Business Plan however the plans are not intended to be strategic planning documents. A strategic approach to planning is a key activity with respect to preparedness. Identifying activities and areas of need, from a strategic, more longer-term perspective, is critical to the success of the CCG SAR program. Since CCGA is an integral component of the CCG SAR program and makes an important contribution to its overall success, identifying activities and areas of need that align with those of CCG SAR will only further contribute to the overall success of maritime SAR. A strategic approach to planning that aligns two key players, such as CCG SAR and CCGA in the delivery of SAR services would enhance the preparedness of the CCG and the success of maritime SAR.

Recommendation #1: We recommend that the Director, Maritime Safety Systems develop a strategic approach to regional and national planning of the Canadian Coast Guard Search and Rescue program and ensure the Canadian Coast Guard Auxiliary Corporations planning aligns with the Canadian Coast Guard�s Search and Rescue program planning.

Training and Exercises

The CCG SAR program has various national training requirements and standards. A JRCC/MRSC Coordinator is required, according to standard course curriculum, to pass four specialized courses at the CCGC25 prior to working at a rescue centre. Required training courses at the College were consistently attended by CCG personnel over the past five years26. Once a JRCC/MRSC Coordinator has completed the required specialized SAR training at the College, they are required to participate in the JRCC/MRSC on-the-job Training Program, as well as continuous training throughout the course of their career. SAR training standards also exist for the Inshore Rescue Boat (IRB) Program, Rescue Specialist Program and various medic programs.

The CCGA developed an overall National Training Standard in 2001 in cooperation with the CCG. The standards set out the minimum training requirements and certification standards for CCGA members. Each Auxiliary Corporation at a minimum adheres to the National Training Standard but each region has developed a unique, regional specific training standard, guide and manual in addition to the National Training Standard.

For each Auxiliary Corporation, adherence to the National Training Standard is set against specific circumstances, conditions, aspirations, funding realities, style of membership and recruitment/succession issues. Accordingly, regional training programs vary in focus, scope and delivery methods. Auxiliary Corporations use their own benchmark criteria and strategies to validate trainer and member competency, program content and training outcomes27.

The majority of CCG SAR and CCGA program personnel are satisfied with overall training, citing it as useful, appropriate, available and accessible. Training sufficiently prepares CCG SAR and CCGA program personnel to respond to a maritime incident28.

Survey results indicate that large scale planning exercises29 are regarded as necessary and useful in preparing for a maritime incident however, respondents are of the opinion that an insufficient number of these exercises occur in a given year. Survey respondents identified that increasing the frequency of large scale planning exercises and improving the follow up after such exercises could contribute to the improvement of preparedness. There was no evidence of a national or regional large scale planning exercise standard or approach for the CCG SAR program. Given that there is no official CCG SAR strategic planning document, no baseline has been established with regard to what is considered a sufficient number of large scale planning exercises to be conducted in a given year.

After Action Reports are intended to be produced following a large-scale planning exercise to identify and share lessons learned and best practices with participating partners. After Action Reports are inconsistently produced, not well communicated and infrequently used to update planning or policy and procedural documents30. Taking all regions into consideration, only 14% of After Action Reports were produced over the past five years.

Large Scale Planning Exercices compared to After Action Reports produced Source: 2006-2010 CCG SAR Administrative data

The total number of CCG SAR led training exercises has increased over the past five years with the exception of last year when it significantly dropped31.

The total number of CCGA led training exercises has increased over the past five years while the total cost to conduct these exercises has not increased. The CCGA continues to conduct more training exercises each year for less. Results from the CCG SAR and CCGA surveys suggest that there are a sufficient number of CCGA led exercises32.

Table 4.0: Number and cost of CCGA led training exercises
  2006/07 2007/08 2008/09 2009/10 2010/11
Training exercises 1353 1567 1982 2224 2566
Cost

$966,213

$1,002,248

$887,163

$880,341

$930,633

Source: CCGA Business Plans and Administrative Data

CCG SAR program has various national training requirements and standards but does not have a national standard or approach for conducting large scale planning exercises or the development and dissemination of After Action Reports. Large scale planning exercises were found to be necessary and useful in preparing for a maritime incident. However, the number of exercises, how they should be conducted and with whom has not been identified or standardized. After Action Reports are intended to identify and share lessons learned and best practices for the purpose of contributing to improving preparedness but a national approach to preparing and disseminating these reports has not been developed. The program has not identified a national standard or approach for conducting large scale planning exercises or the development of a process for the production and dissemination of After Action Reports. Developing a national standard in conjunction with a strategic approach to planning would enhance preparedness.

Recommendation #2: We recommend that the Director, Maritime Safety Systems develop a national standard or approach for conducting large scale exercises including the development of a process for the production and dissemination of After Action Reports.

CCGA Recruitment and Retention

Overall CCGA membership is declining and recruitment of new members remains a challenge. Although new membership has been increasing over the past five years, a larger proportion of members have left the CCGA. The number of new members joining compared to those leaving the CCGA was fairly consistent over the past five years with the exception of 2010/11 in which a significant number of members left the CCGA in comparison to the number of members who joined.

Table 5.0: CCGA Membership
  2006/07 2007/08 2008/09 2009/10 2010/11
Membership 4266 4261 4194 4007 3979
New 220 235 335 360 383
Leaving 220 166 419 328 662
Source: CCGA Administrative Data

CCGA recruitment, awards and recognition activities are listed as authorized administrative activities within the Contribution Agreement but only awards, gifts and recognition program expenses are listed as eligible expenditures. Recruitment costs were not listed as eligible expenditures. In other words, there is no specific funding allocated for recruitment of new members in the Contribution Agreement33.

Results from the interviews suggest that recruitment of new CCGA members and retention is a challenge in many Regions. There is a lack of funding for recruitment in Regions even though there have been identified short fall areas requiring response capacity. Seven specific recommendations emerged from the 2007 SAR Needs Analysis regarding the need for the CCG to work with the CCGA to enhance response capacity by recruiting new owner-operated vessels in specific areas. CCGA has not received any funding to act on these recommendations.

Availability of Equipment and Resources

The necessary CCG SAR resources and equipment are not always available when needed to successfully resolve a maritime SAR incident. With regard to availability of resources, the 2007 SAR Needs Analysis found that the CCG is concerned with the level, or quality of SAR services provided during SAR missions as there is currently a lack of qualified personnel who may provide advanced level first aid and/or offshore survival and rescue techniques. With regards to equipment availability, the 2007 SAR Needs Analysis also found that not all small vessels in the CCG Fleet are currently meeting the CCG Fleet Order 207 (CGFO 207) which requires specific SAR equipment to be carried at all times on Fleet units and that equipment be maintained in good operational order.

The Level of Service document outlines four service standards with regard to availability of resources; however it does not include any service standards with regards to availability of equipment34, as equipment requirements are specified in the CGFO 207. Equipment (vessels) is provided by Fleet and there is currently no service level agreement between Fleet and CCG SAR.

For the most part, the CCGA and their equipment are available when needed35, however there are no performance measures or response times for the CCGA with regard to availability. On average, over the past five years, the CCGA has participated in 22.9% of the SAR incidents36.

Effectiveness Evaluation Question 2.2
To what extent are maritime SAR cases resolved
To what extent do the CCGA members contribute to the resolution of SAR responses

Findings

To determine the extent to which maritime SAR cases are resolved, the evaluation attempted to examine the appropriateness of a response to an incident. However CCG SAR program does not currently have a method of measuring appropriateness, therefore timeliness in terms of levels of service and reaction times were examined.

CCG SAR Level of Service document contains several service standards. For the purpose of this evaluation, evaluators assessed the following service standard to determine timeliness: Search and Rescue Units will depart on a SAR tasking within 30 minutes or less 99% from the time of receiving a SAR tasking (reaction time). The time it takes a tasked vessel to reach an indentified incident (response time) varies significantly and is not measured by the program. The CCG SAR program service standard was met for 96% of the incidents between 2006 and 2010. During this period when the service standard was met, the average reaction time across categories (M1-M4)37 when the service standard time was being met was 7.7 minutes. Many of the key partners interviewed stated that the CCG SAR program task the appropriate resources and equipment to resolve a maritime SAR incident.

Reaction Time across Categories Source: SISAR database

The service standard of 30 minutes or less 99% of the time is currently the only measure being used to determine the timeliness of a response to an incident by a CCG SAR resource. There are currently no service standards which measure the length of time it takes a tasked vessel to reach an identified incident. The CCG SAR program does not currently have a method of measuring appropriateness of a response other than measuring timeliness via the service standard of 30 minutes or less 99% of the time.

The 2007 SAR Needs Analysis found that level of service and service standards as they are currently defined are not conducive to SAR system performance being precisely measured. It was recommended that the existing maritime SAR level of service and service standards be reviewed and revised to ensure measureable and realistic service standards. The level of service and service standards were reviewed by the CCG SAR program in 2010 and it was determined that changes were not required at that time with the exception of the Sea Island dive team.

The evaluation was unable to measure the CCGA�s appropriateness of a response to a maritime SAR incident. There is no Level of Service document or service standard for the CCGA with regard to appropriateness of a response to a maritime SAR incident and as such, data is not available with respect to timeliness of a response (e.g. standby postures or reaction times).

The CCG SAR program currently has only one service standard to measure timeliness of a response and no measures to determine the appropriateness of a response to an incident. CCGA is an integral component of CCG SAR however they are not able to measure appropriateness as they do not currently have service standards. There is currently no means of measuring or communicating CCGA appropriateness of a response or their contribution to overall maritime SAR38. Furthermore, the CCG SAR program would benefit from a review of all of its standards to ensure they are measurable and accurately reflect maritime SAR performance.

Recommendation #3: We recommend the Director, Maritime Safety Systems review the current service standards for the Canadian Coast Guard Search and Rescue program for relevance and measurability and determine whether additional service standards are required to appropriately measure the success of the program. In addition, we recommend the development of service standards for the Canadian Coast Guard Auxiliary that align with the Canadian Coast Guard Search Rescue program service standards.

Effectiveness Evaluation Question 2.3
To what extent are people in maritime environment assisted

Findings

To determine the extent to which people in maritime environment are assisted, the evaluation examined the distribution of maritime incidents across categories; delivery partners� perceptions of people in maritime distress assisted; and lives saved from those at risk.

Maritime incidents are categorized into four categories (M1 to M4). M1 is a distress incident; M2 is a potential distress; M3 is a situation resolved in the uncertainty phase; and M4 is a known maritime related false alarm or hoax39.

The following chart demonstrates the distribution of reported maritime incidents across categories for the CCG SAR program and CCGA (M1-M4) over the past five years. The CCG SAR program assisted in more maritime incidents (25,491) over a five year period compared to CCGA (5838). Of all reported maritime incidents across categories over the past five years involving the CCG SAR program, 5% were distress incidents (M1) and 10% were potential distress incidents (M2).

Martitime Incidents Across Categories Source: SISAR database

Delivery partners� perceptions of the CCG SAR program and CCGA with respect to assisting people in maritime distress is that personnel are very dedicated, professional and make a significant contribution. On average, CCG SAR program contributed to saving 97% of lives at risk and CCGA contributed to saving 96% of lives at risk over the past five years40.

22Source: 2009 Employee Workload Study; 2007 CCG SAR Needs Analysis

23Source: CCGA and CCG SAR interviews

24Source: 2007 Canadian Coast Guard Auxiliary Audit

25The four specialized training courses offered at the College are; Maritime Search and Planning; SAR Mobile Facilities / On-Scene Coordinators; SAR Mission Coordinators; and Aeronautical Search Masters.

26Source: 2006-2010 CCG SAR Administrative Data

27Source: 2008 CCGA Training Overview Report

28Source: CCGA and CCG SAR interviews and survey results

29On the water training exercises with Fleet crews and various external partners

30Source: CCGA and CCG SAR survey results and CCG SAR interviews

31Source: 2006-2010 CCG SAR Administrative Data

32See Appendix B figure 1 for a detailed breakdown of CCGA (Pacific, Central, East) and CCG SAR responses

33In 2010 a memo was provided to each CCGA President from the CCG SAR Manager indicating that recruitment costs were an eligible expenditure but no official change has been made to the Contribution Agreement.

34CCG Fleet order 207 defines the SAR equipment that is required on board a vessel

35Source: CCGA and CCG SAR Interview and survey results

36Source: SISAR database

37M1: distress / M2: Potential Distress / M3: Situation Resolved in the Uncertainty Phase / M4: Known Maritime Related False Alarms or Hoax

38Given that the CCGA is a volunteer organization that responds when it has the capacity and capability to do so, service standards have not been developed in the past.

39A distress incident is where people(s) are threatened by grave and imminent danger and require immediate assistance. A potential distress is whereby the potential exists for a distress incident if timely action is not taken. A situation resolved in the uncertainty phase, is when a disabled vessel in no immediate danger; a disoriented or lost vessel in no immediate danger or; other related incidents involving a vessel with no person on board. Finally, a known maritime related false alarm or hoax is when a located overdue vessel that is in no difficulty but that had failed to advise of a change of plans or had failed to report, the investigation of a maritime related sighting that proved false or an emergency position indicating radio beacon false alarm

40Source: SISAR database

4.3 Efficiency

Evaluation Question 3.1 To what extent is the design and delivery of the program appropriate to produce outputs

Findings

To determine the extent to which the design and delivery of the CCG SAR program and CCGA are appropriate to produce outputs, the evaluation examined roles and responsibilities between CCG SAR and key delivery partners and CCG SAR and CCGA performance data in supporting decision-making.

Roles and Responsibilities

The National Search and Rescue Manual is issued under joint authority between the DND and CCG. The manual presents the federal SAR policy and organization, and the interdepartmental structure established to provide effective SAR. The manual also outlines roles and responsibilities with various partners. The manual was recently updated by the CCG SAR program, in cooperation with DND; however it has yet to be formally approved. Until the manual has been formally approved by the CCG SAR program and DND, the International Aeronautical and Maritime Search and Rescue Manual can be used.

There are several key internal delivery partners that are not included in the National Search and Rescue Manual. For those key internal delivery partners, there was no evidence that any official documents or Memorandum of Understanding (MOU) describing key roles and responsibilities between themselves and the CCG SAR program exist. For example, roles and responsibilities between Fleet and the CCG SAR program have not been formally documented. However, a Service Level Agreement has been drafted between Fleet and Maritime Services with a proposed SAR program Annex. This Agreement has not been approved by all parties and therefore has yet to be implemented. There is no official document or MOU to describe roles and responsibilities at a national level between MCTS and the CCG SAR program. Finally, the Contribution Agreement between the CCG SAR program and CCGA does not outline specific roles and responsibilities of the CCGA or how the CCGA is to work with CCG SAR program41. The document does, however, outline categories of authorized CCGA activities.

According to Fleet partners, roles and responsibilities between themselves and the CCG SAR program are not clearly defined and understood42. DND partners identified roles and responsibilities between themselves and the CCG SAR program are clearly defined but not well understood43. According to CCG SAR program interviews and survey results, the program�s perception of roles and responsibilities with key delivery partners are clearly defined and understood with the exception of Fleet. Roles and responsibilities between CCG SAR and key delivery partners are not always clearly defined and well understood, particularly with Fleet. Fleet is the primary CCG on-water responder for maritime distress situations44. There is no clear definition of CCG SAR�s expectations of Fleet and no clear understanding between the two programs of the implications of CCG SAR requirements of Fleet. Currently there is no formal document describing roles and responsibilities and no service level agreement between the two programs.

Recommendation #4: We recommend that the Commissioner of the Canadian Coast Guard formally define Search and Rescue program roles and responsibilities between Fleet and Maritime Services.

Performance Data and Decision Making

The CCG SAR program has limited performance data collection and reporting. The program formerly produced a Maritime Search and Rescue Incidents Annual Report but as of 2001 no longer does so and nothing similar has been implemented in its place. The CCG produces mid and year-end reviews that focus on overall client service, efficiency and effectiveness, people and key initiatives, and ongoing services. Aspects of CCG SAR have been discussed in the reviews such as the SAR Needs Analysis, SAR Rescue Capability, SAR Technical Training and the renewal of the CCGA Contribution Agreement, however, the actual performance of the CCG SAR program is not adequately discussed in these reviews.

There are two data systems that can be used to generate search and rescue statistics in the rescue centres. First, SMMS, under the responsibility of the DND, records stores and allows quick retrieval of information that enables the appropriate resources to be tasked during an incident. Second, SISAR, a CCG program, is a web-based database that integrates all regional response data into one national system. These two systems are automatically interconnected as SMMS response data is migrated into SISAR. SMMS data is reviewed for completeness, accuracy, and conformation to SAR business standards before migrating to SISAR. Cases and associated fields that have failed to meet these standards are not successfully converted into SISAR. On average, over the past five years, approximately 8.4% of incidents are missing from the SISAR database45. SISAR data is not being used to generate performance reports and as such, not consistently relied upon to inform decision-making.

Results from the CCG SAR interviews indicate that all program personnel stated CCG SAR performance data is not sufficient in supporting decision-making. The main issues raised were the transfer of data between the SMMS and SISAR and the need for better performance indicators. The 2007 SAR Needs Analysis found that the CCG does not have a nationally consistent means of tracking or measuring the annual delivery of SAR services. It was recommended that the program take the initiative to develop a nationally consistent tool to track and measure SAR service delivery as provided by Fleet, as a means of measuring SAR performance. Such a tool has not been developed.

CCGA data collection is inconsistent across regions and there is no official performance reporting to the CCG SAR program. According to the Contribution Agreement, the CCGA is required to provide an Annual Contribution Report to the Minister of Fisheries and Oceans on all authorized activities; 11 indicators were outlined in the Agreement. CCGA is not producing the Annual Contribution Report. Reporting on financial aspects of CCGA activities is done annually via the Business Plans as well as other various reporting through the CCGA insurance reporting documents. The 2007 CCGA Results-based Management Accountability Framework (RMAF) outlines five performance indicators but these indicators are not reported on in a formal document or report. Overall CCGA performance data is not accurate as a result of either data not being collected or the varying quality of unique data collection systems implemented in each Region46. Performance data seems to be more accurate in Regions using the SAR Management System (SMS).

CCGA performance data does not always support decision-making. There is no common reporting requirement for the annual Business Plans nor is there a requirement for an official performance report. As a result, performance data is not used to support overall planning or the allocation of funding from year to year. For example, Regions receive the same funding each year regardless of performance or need from the previous year.

The CCG SAR program and CCGA performance data do not always support decision-making as a result of limited and inconsistent performance data collection and reporting. Since CCGA is an integral component of the CCG SAR program and makes an important contribution to its overall success, it is imperative that CCGA performance data and reporting efforts are aligned with CCG SAR. Given the programs have very similar outcomes, the type of information collected should also be very similar. Aligning performance data collection and reporting will only strengthen the overall performance story of the maritime SAR system.

Recommendation #5: We recommend the Director, Maritime Safety Systems revise the Canadian Coast Guard Search and Rescue program Performance Measurement Strategy to accurately measure program performance. In addition we recommend the Canadian Coast Guard Auxiliary performance data collection and reporting efforts align with the Canadian Coast Guard Search and Rescue program.

41Roles and responsibilities are outlined in the CCGA National Guidelines but are not included or attached to the Contribution Agreement.

42Source: CCG SAR partner interviews and survey results

43Source: CCG SAR partner interviews and survey results

44Fleet maintains an operationally ready fleet of 116 vessels, which includes 41 dedicated search and rescue lifeboats stationed strategically throughout the country.

45Source: 2006-2010 SISAR administrative data

46Source: CCGA interviews

4.4 Economy

Evaluation Question #10: Are CCG SAR / CCGA operating in a way that minimizes the use of resources to achieve its intended outcomes?

Findings

The CCG SAR program leverages volunteers and equipment to deliver maritime SAR services generating substantial savings to the federal government. Historically and as the current evaluation confirms, the CCGA makes an important contribution through the provision of thousands of volunteer hours and vessels funded through the contribution agreements at a cost to the federal government of approximately $5 million dollars per year. The 2004 evaluation of the CCGA conducted a cost-effectiveness exercise47 that focused on the volunteer contribution of CCGA-Newfoundland and compared this to what it would have potentially cost the CCG SAR program to hire the equivalent in paid staff. This exercise found that the CCG SAR program would have had to hire at least 32 new staff members to address the 52,045 hours that the volunteers carried out in 2001. In short, there was a $43.55 return on public good investment, or in other words, every dollar expended, resulted in a cost�avoidance48 of approximately $43. Today, the federal government continues to enjoy a similar type of leveraging as demonstrated in the table below which describes the CCGA�s more recent contribution of resources, equipment and assignments to maritime incidents.

Table 6.0: CCGA Resource Contribution
  Number of Volunteers Vessels Taskings
2006-07 4,266 1,209 1,804
2007-08 4,261 1,198 1,829
2008-09 4,194 1,165 1,738
2009-10 4,007 1,166 1,749
2010-11 3,979 1,133 1,942
Source: CCGA program documents

The CCG estimates that without the support of the CCGA, the CCG would need to spend some $337 million to purchase a similar fleet of vessels and an additional $200 - $300 million annually in salaries and benefits49. The continued contribution of CCGA resources and equipment demonstrates that the federal government continues to benefit from substantial savings for the overall delivery of maritime SAR services.

Alternative Delivery Approaches

As for alternative delivery approaches and achieving CCG SAR program results more economically, the evaluation conducted a comparative analysis of the maritime CCG SAR program delivery model to those of the United States of America (USA), the United Kingdom (UK), Australia (AU) and New Zealand (NZ). The evaluation found that while all of these countries employ variations of the government funded / volunteer based model, some countries achieve greater degrees of leveraging from the volunteer sector than the CCG SAR program model.

The Canadian and the American models both have a primary SAR coordination and response function that is led by the federal government (Canadian Coast Guard / United States Coast Guard) and supported by a government funded civilian volunteer organization (Canadian Coast Guard Auxiliary / United States Coast Guard Auxiliary). The Government bodies are responsible for coordinating the response to maritime SAR incidents and the volunteer body provides support during SAR missions.

In the UK, the SAR coordination function is led by Her Majesty�s Coastguard (HMCG). The HMCG�s volunteer component, the Coastguard Rescue Service, provides the UK�s coastal search and rescue capability. There is also the Royal National Lifeboat Institution (RNLI), an independent volunteer organization whose role is to provide a 24-hour lifeboat search and rescue service to 100 nautical miles out from the coast of the United Kingdom and Republic of Ireland. It is a large organization where for example in 2008 it cost approximately �138.5M to operate the RNLI. The RNLI is independent from Government and relies on voluntary contributions and legacies for its income.

In Australia, the maritime SAR coordination function is led by the Australian Maritime Safety Authority. There are several private volunteer coast guard organizations that provide the SAR capability. The two largest organizations are the Royal Volunteer Coastal Patrol and the Australian Volunteer Coast Guard. Both are run entirely by volunteers, and both are financially self-sufficient relying on financial donations from corporate sponsors and other fundraising activities.

Finally in New Zealand, The Royal New Zealand Coastguard is a civilian volunteer charitable organization and the primary SAR service provider. Larger incidents are managed by the Maritime New Zealand, a Crown entity, with support from the New Zealand Defense Force. Maritime New Zealand assumes 15% of the financial burden for its nation�s maritime SAR activities, while the remaining 85% is assumed by the Royal New Zealand Coast Guard through fundraising activities.

This comparative analysis illustrates that in the UK, AU and NZ the volunteer sectors to a greater extent are more self-sufficient than in the Canadian context, benefiting from greater economies for the federal government towards the achievement of SAR outcomes.

In recent years, the Canadian Coast Guard has experienced decreased funding and increased operational costs. It may be beneficial to further examine potential alternative approaches and best practices in other countries, and adapting these to further leverage resources from the volunteer sector. This may assist the CCG SAR program to achieve SAR outcomes more economically.

47The data derived and the subsequent analysis were meant to provide the reader with a reasonable sense as to the cost-effectiveness of volunteer work within the CCGA-Newfoundland and was not meant to be quoted as the definitive source for exact and precise figures

48Cost Avoidance is defined as the reduction or elimination of a new cost that would have otherwise occurred

49Source: 2007 Integrated Results-Based Management Accountability Framework

5. Conclusions

In 2011 the Evaluation Directorate undertook an evaluation of the CCG SAR program and the CCGA. The evaluations were planned and conducted separately but presented in a combined report. Recommendations stemming from the main findings were formulated to allow for improvements to the program where necessary and to inform future decision-making. The evaluations focused on the core issues in assessing value for money: relevance and performance, including effectiveness, efficiency and economy. It assessed the extent to which the CCG SAR program and CCGA have achieved their outcomes stemming from activities outlined in their respective logic models. The timeframe for this evaluation covers the period from 2006-07 to 2010-11. It is inclusive of the National Capital Region as well as five Regional offices. 

Relevance

Overall the evaluation found the CCG SAR program and CCGA to be relevant. The objectives are aligned with departmental and government wide priorities and the current role of the federal government is appropriate in delivering both programs.

Effectiveness

For the most part, the evaluation found the CCG SAR program and CCGA to be effective. Several areas were identified where improvements could increase the effectiveness of the programs. There is no official national strategic approach to planning for the CCG SAR program and the CCGA annual Business Plans are inconsistent across Regions. A strategic approach to planning that aligns two key players, such as CCG SAR and CCGA in the delivery of SAR services would enhance the preparedness of the CCG and the success of maritime SAR. The CCG SAR program has various national training requirements and standards but does not have a national standard or approach for conducting large scale planning exercises or the development and dissemination of After Action Reports. Developing a national standard or approach would contribute to improving overall preparedness. There is currently no means of measuring or communicating CCGA appropriateness of a response or their contribution to overall maritime SAR as a result of no established service standards. The CCG SAR program currently measures timeliness of a response using one of its service standards, however does not have a method of measuring appropriateness of a response to an incident. The CCG SAR program would benefit from a review of all its service standards to ensure they are measurable and accurately reflect maritime SAR performance.

Efficiency

The evaluation findings suggest that the CCG SAR program and CCGA demonstrate several efficiencies but some improvements were noted. Roles and responsibilities between the CCG SAR program and key internal delivery partners are not always clearly defined and well understood, particularly with Fleet. Currently there is no formal document describing roles and responsibilities and no service level agreement between the two programs. The CCG SAR program and CCGA performance data do not always support decision-making as a result of limited and inconsistent performance data collection and reporting. Since CCGA is an integral component of the CCG SAR program and makes an important contribution to its overall success, it is imperative that CCG SAR has a Performance Measurement Strategy that accurately measures program performance and CCGA performance data and reporting efforts are aligned with those of CCG SAR. Aligning performance data collection and reporting will strengthen the overall performance story of the maritime SAR system.

Economy

Overall the evaluation found that the CCG SAR program leverages volunteers and equipment to deliver maritime SAR services, generating substantial savings to the federal government. The CCGA makes an important contribution to maritime SAR services through the provision of volunteers and vessels funded through contribution agreements. In recent years, the CCG has experienced decreased funding and increased operational costs.   Examining potential alternative approaches and best practices in other countries, and adapting these to further leverage resources from the volunteer sector may assist the CCG SAR program to achieve SAR outcomes more economically. 

The evaluation made several recommendations to improve the performance of the CCG SAR program and CCGA.

Annex A: Evaluation Matrix

Annex A: Evaluation Matrix
Issue Indicators Data Source
1.0 Relevance
1.1 Is there a continued need for CCG SAR / CCGA? 1.1.1 Evidence of a need for program Document Review
1.1.2 Delivery partners attest to the importance/need for the program Interview, Survey
1.1.3 Trend of annual number of maritime SAR incidents relative to total maritime traffic (CCG SAR) and Trend of annual number of CCGA responses relative to total number of maritime SAR incidents (CCGA) Document Review
1.2 To what extent are the objectives of CCG SAR / CCGA aligned with departmental and governmental wide priorities? 1.2.1 Degree of program alignment with:
  • Government of Canada objectives and priorities
  • DFO objectives, priorities and strategic outcome
  • Program objectives, vision and mission
Document Review
1.3 Is the current role of the Federal Government appropriate in delivering CCG SAR / CCGA? 1.3.1 Mandate is consistent with federal jurisdiction Document Review
1.3.2 Perceived importance / appropriateness of federal government role in assisting people in maritime distress (CCG SAR) and Perceived importance / appropriateness of federal government role in administering the program (contribution agreement) in assisting people in maritime distress (CCGA) Interviews, Survey
2.0 Effectiveness (Program Outcomes)
2.1 To what extent is the CCG SAR system / CCGA members prepared and available to respond 2.1.1 SAR plans are implemented and appropriate to deliver SAR services (CCG SAR) and National and Regional plans are appropriate and implemented to deliver SAR services (CCGA) Document Review, Interviews, Survey
2.1.2 Existence and appropriateness of training and exercises Document Review, Interview, Survey
2.1.3 Availability of equipment and resources Document Review, Interview, Survey
2.1.4 Recruitment and retention of CCGA members (CCGA) Document Review, Interviews
2.2 To what extent are maritime SAR cases resolved (CCG SAR) and To what extent do the CCGA members contribute to the resolution of SAR responses 2.2.1 Appropriateness of response to an incident Document Review, Interviews, Survey
2.2.2 External factors / elements affecting the resolution of a SAR incident Document Review, Interviews
2.3 To what extent are people in maritime distress assisted 2.3.1 Distribution of maritime incidents across categories Document Review
2.3.2 Delivery partners perceptions of people in maritime distress assisted Interviews
2.3.3 Distribution of people assisted Document Review
3.0 Efficiency
3.1 To what extent is the design and delivery of CCG SAR / CCGA appropriate to produce outputs 3.1.1 Extent to which roles and responsibilities of partners are clearly defined and understood (CCG SAR) Document Review, Interviews, Survey
3.1.2 Extent to which performance data supports decision-making and departmental accountability requirements Document Review, Interviews
3.1.3 Factors that support or hinder efficiency Document Review, Interviews
4.0 Economy
4.1 Is CCG SAR / CCGA operating in a way that minimizes the use of resources to achieve its intended outcomes 4.1.1 Comparison of Canada’s maritime SAR / CCGA with similar initiatives in the U.S. and other countries Document Review
4.1.2 Outputs are produced are of a quality and quantity acceptable to support the achievement of intended outcomes Document Review, Interviews

Annex B: Survey Results

Figure 1.0
Answers to the survey question : There are sufficient CCGA led training opportunities in a given year.

Annex C: Management Action Plan

Management Action Plan in Response to the Evaluation of Canadian Coast Guard (CCG) Search and Rescue and Canadian Coast Guard Auxiliary (CCGA)

Annex C: Management Action Plan
Recommendations

Recommendation 1:
We recommend that the Director, Maritime Safety Systems develop a strategic approach to regional and national planning of the Canadian Coast Guard Search and Rescue program and ensure the Canadian Coast Guard Auxiliary Corporations planning aligns with Coast Guard’s Search and Rescue program planning.

Strategy

The CCG SAR program will develop and seek approval for a new risk-based methodology for the Search and Rescue Needs Analysis (“Risk-based Analysis of Maritime SAR Delivery” or “RAMSARD”) which will identify the operational risks in each SAR area, including any needs for additional capacity or capabilities from partners. The implementation of the methodology will involve engaging stakeholders, partners and clients in discussions on SAR preparedness and response. As a result, it will influence the planning process by providing sound information for use in operational planning within Regions and with partner organizations (e.g. Regional SAR Plans) and identify activities and areas of need from a strategic perspective. The RAMSARD process will also provide a tool for credible risk-based decisions regarding the planning and allocation of maritime SAR resources.

As an initial step of the renewal of the Contribution Agreement with the CCGA, the CCG SAR program will engage CCGA in a discussion on the “Strategic Directions” of the SAR program for the next 5 years. The “Strategic Directions” will be based on the principles, priorities, expectations and expected results of the CCG SAR program and will be aligned with CCG priorities and resources. This will influence the longer-term strategic planning of the CCGA as well as the shorter-term year to year planning done through the CCGA Business Planning Process.

Management Actions Actions Completed Actions Outstanding Target Date Supporting Evidence
CCG & Risk-based Analysis of Maritime Search and Rescue Delivery
1) Develop a new methodology for SAR Needs Analysis (“Risk-based Analysis of Maritime SAR Delivery - RAMSARD) a) Presented to SAR Superintendents for discussion in Summer 2011     Draft methodology document
  b) Draft methodology presented to Maritime Services Executive Board (MEB) in September 2011    
  • Draft methodology document
  • Deck presentation
  c) Illustrative example (SAR Area 301 – Victoria) used to test new methodology in October 2011 presented to MEB     Updated deck presentation for MEB
Adoption of final RAMSARD methodology   d) Presentation of RAMSARD for approval to Management Board June 2012  
Implementation of RAMSARD methodology   e) Products of year one of “Risk-based Analysis of Maritime SAR Delivery” completed for review by SAR National Manager and Regional SAR Superintendents Products presented to MEB for discussion and recommendations to Management Board December 2013  
    g) Recommendations to Management Board based on results of RAMSARD year 1 March 2013  
2) Strategic Directions for CCGA Contribution Program
2) CCG SAR program will develop a 5 year “Strategic Directions” document to outline the vision, priorities and CCG expectations of the CCGA every 5 years prior to negotiating the renewal of the Contribution Agreement   a) Discuss and develop the outline for the “Strategic Directions” at a SAR Superintendents Meeting b) MEB to review Draft “Strategic Directions” c) Coast Guard to discuss “Strategic Directions” with Auxiliary September 2012  
    d) Coast Guard Management Board approval of a final “Strategic Directions” e) Document
will be signed by DG, Maritime Services and used to guide discussions with CCGA during the renewal of the Contribution Agreement
October 2012  
Recommendations
Recommendation 2:
We recommend that the Director, Maritime Safety Systems develop a national standard or approach for conducting large scale exercises including the development of a process for the production and dissemination of After Action Reports.
Strategy

CCG SAR program will examine its current approach to planning large scale exercises in the context of a review of its overall national exercise program. This will include both tabletop and on-the-water exercises. The initial focus will be developing an approach to large scale on-the-water exercises, as per the recommendation in the report, including the dissemination and of After Action Reports which are used to document lessons learned from these exercises.

Following review and recommendations on a national approach to large scale on-the-water exercises, the program will also discuss and make a decision on the appropriate number of exercises per year (including large scale on-the-water exercises), an approach to documenting the exercises and how to effectively share lessons learned internally and with participating partners, and the adoption of any national tools (e.g., exercise planning tool kit, template).

Management Actions Actions Completed Actions Outstanding Target Date Supporting Evidence
1) The CCG SAR program will review its Exercise Program, with particular focus on large scale exercises, to facilitate a more national approach to exercise planning and reporting on lessons learned Course and venue booked for Montreal (mid-February) a) Regional SAR Preparedness Officers will receive training on how to design and plan methodologies for conducting effective large-scale exercises February 2012  
    b) Regional SAR Preparedness Officers will meet following the training to discuss current regional approaches and potential national tools to support exercise planning process (e.g., national template for exercises) and processes/methods that could be adopted across the organization. February 2012  
    Proposed approach discussed is presented to National CCG SAR Manager and SAR Superintendents for discussion and approval. (Note: The approach would encompass both exercise planning and the dissemination of After Action Reports as per recommendation) April 2012  
2) Determine the number of annual exercises (both large on-the water exercises and tabletop) to be executed Nationally   a) SAR Superintendents will meet to discuss the appropriate number of SAR exercises, both tabletop and large scale on-the-water exercises that should be planned and executed annually in each region, given current and projected fiscal pressures. September 2012  
    b) CCG SAR program to make recommendation to Maritime Services Executive Board on number of exercises (both large scale exercises and other) to be budgeted and planned for nationally December 2012  
3) Exercise Working Group to Review Implementation of new tools, plans and processes   a) Preparedness Officers will provide a report to National SAR Manager and Superintendents on lessons learned and effectiveness of new approaches/tools adopted during exercise season March 2013  
Recommendations
Recommendation 3:

We recommend that the Director, Maritime Safety Systems review the current service standards for the Canadian Coast Guard Search and Rescue program for relevance and measurability and determine whether additional service standards are required to appropriately measure the success of the program. In addition, we recommend the development of service standards for the Canadian Coast Guard Auxiliary that align with the Canadian Coast Guard Search and Rescue program service standard.

Strategy

The CCG SAR program will review its levels of service and service standards to assess whether there are relevant and measurable standards that could be put in place to ensure clients have clear expectations in terms of the services the CCG SAR program is capable of delivering. The program will also monitor and report its success in meeting its service standards to demonstrate program performance to the public.

The CCG SAR program will also determine with the CCGA any feasible service standards for the CCGA moving forward.

Management Actions Actions Completed Actions Outstanding Target Date Supporting Evidence
1) Review of Coast Guard Search and Rescue Levels of Service and Service Standards
The CCG SAR program will review its levels of service and service standards and assess feasibility for change  

a) SAR Superintendents to begin discussion on levels of service and service standards

b) Document to be drafted that identifies possible changes to Levels of Services & Service Standards and process to change

c) In consultation with Fleet Executive Board (FEB), “Levels of Service and Service Standard: Proposed Changes” Document to be presented to MEB for approval

December 2012  
2) Coast Guard Auxiliary Development of Service Standards
Engage CCGA in discussions on service standards and determine what is feasible for next 5 year Contribution Agreement   a) CCG SAR program to discuss service standards with CCGA and propose working group be established to examine options for CCGA October 2012  
    b) Integrate any new service standards into renewed CCGA Contribution Agreement documentation April 2013  
Recommendations

Recommendation 4:
We recommend that the Commissioner of the Canadian Coast Guard formally define Search and Rescue roles and responsibilities between Fleet and Maritimes Services.

Strategy

In November 2011, a Service Level Agreement was signed between Director General, Fleet and Director General, Maritime Services.

Program Annexes, which are expected to include roles and responsibilities, service expectations, and performance measures are still being negotiated.  CCG SAR Maritime Services will continue into discussions with CCG Fleet to formally define their respective roles and responsibilities with respect to delivery of Coast Guard SAR services and performance measures that will be monitored and reported on moving forward.

Management Actions Actions Completed Actions Outstanding Target Date Supporting Evidence
1)  Service Level Agreement (Program Annex) between SAR and Fleet
1)  The CCG SAR program will meet with Fleet to finalize the Program Annex of the Service Level Agreement (SLA) (Note:  Program Annex will defines the roles and responsibilities between Fleet and Maritime Services vis-à-vis delivery of the Coast Guard SAR program) a) In 2010/11, SAR drafted a Program Annex with proposed roles and responsibilities between Fleet and CCG SAR program   Completed Draft Program Annex completed by CCG SAR program for discussion with Fleet
    b) CCG SAR program will work with CCG Fleet to finalize Program Annex September 2012  
    d)  Secure sign-off from DG, Fleet and DG, MS on the CCG SAR program Annex March 2013  
Recommendations

Recommendation 5:
We recommend that the Director, Maritime Safety Systems revise the Canadian Coast Guard Search and Rescue program Performance Measurement Strategy to accurately measure program performance. In addition, we recommend the Canadian Coast Guard Auxiliary performance data collection and reporting efforts align with the Canadian Coast Guard Search and Rescue Program.

Strategy

CCG SAR program will revise their Performance Measurement Strategy with advice and guidance from the Evaluation Directorate.

CCG SAR program will incorporate CCGA performance measures within the CCG SAR Performance Measurement Strategy to demonstrate the alignment of activities, outputs, and outcomes.  

During the 2012/13 fiscal year, in the negotiation process for the 2013 renewal of the Contribution Agreement, CCG SAR program will formalize with CCGA their data collection and reporting requirements and update the Contribution Agreement accordingly. 

Management Actions Actions Completed Actions Outstanding Target Date Supporting Evidence
1)  Alignment of Performance Measures & Logic Models

The CCG SAR program will review and update its Performance Measurement Strategy.
(Note: CCGA activities, outputs, outcomes and performance measures will be integrated into CCG SAR program Performance Measurement Strategy)

 

a)  Review and revise CCG SAR program and CCGA performance measures with advice and guidance from DFO Evaluation Directorate and Regions

b)  Integrated CCGA Performance Measures into CCG SAR Performance Measurement Strategy 

May 2012  
    c)  Implement updated Performance Measurement Strategy  March 2013  
2)  CCGA Data Collection & Reporting Requirements
CCG SAR program  will enter into discussions with CCGA to formalize the collection and reporting requirements for performance data collection and reporting with the CCGA  

a)  Finalize data collection and reporting requirements and performance measures at the appropriate Joint CCG SAR program/CCGA National Board Meeting

b)  CCG SAR program to finalize with CCGA an annual performance report format, for next 5 years

March 2013