EVALUATION OF FLEET OPERATIONAL
READINESS PROGRAM:
FLEET OPERATIONAL CAPABILITY
SUB-PROGRAM

6B160
FINAL REPORT
JULY 9, 2015

EVALUATION DIRECTORATE


Table of Contents

ACKNOWLEDGEMENTS AND ACRONYMS


ACKNOWLEDGEMENTS

The Evaluation Directorate acknowledges and thanks all individuals who gave of their time and input for this evaluation of the Canadian Coast Guard�s Fleet Operational Capability sub-program. In particular, the Evaluation Directorate acknowledges and thanks all Coast Guard managers, staff and its clients who took the time to share their thoughts through interviews and an online survey. The Evaluation Directorate also acknowledges the time and effort given by senior program management in assisting the evaluation team from the planning phase through reporting.

ACRONYMS


List of acronyms
ACV Air cushion vehicles
CCGS Canadian Coast Guard Ship
COE Centres of Expertise
DFO Fisheries and Oceans Canada
DND Department of National Defence
FAIS Fleet Activity Information System
MSET Marine Security Enforcement Teams
NHQ National Headquarters
O&M Operations and Maintenance (budget)
OGDs Other Government Departments
OAG Office of the Auditor General
PAA Program Alignment Architecture
PM Performance Measurement
RCMP Royal Canadian Mounted Police
SLAs Service Level Agreements

EXECUTIVE SUMMARY


INTRODUCTION

This report presents the results of an evaluation of the Fleet Operational Capability sub-program, one of three sub-programs contained within the Canadian Coast Guard Fleet Operational Readiness Program. The evaluation focused on the extent to which the sub-program demonstrated value for money by assessing its relevance and performance, the latter including effectiveness, efficiency and economy, in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation. The other two sub-programs, Fleet Maintenance and Fleet Procurement, were evaluated separately following the same criteria, with reporting taken place in March 2014.

The evaluation, conducted by the Department of Fisheries and Oceans’ Evaluation Directorate, covers a six-year period from 2007-08 through 2012-13. The sub-program operates nationally within the National Capital Region and across the Coast Guard’s three regions: Western, Central and Arctic, and Atlantic. A 2009 evaluation of the Fleet Operational Readiness program contributed to informing this evaluation.

PROGRAM PROFILE

The Fleet Operational Readiness program, which is administered by the Canadian Coast Guard, is aligned with Fisheries and Ocean Canada’s Safe and Secure Waters strategic outcome, as reflected in the department’s Program Alignment Architecture (PAA). This program is comprised of three components, the first being the focus of this evaluation report (i.e. Fleet Operational Capability sub-program). This sub-program ensures that certificated professionals safely, effectively, and efficiently operate vessels, Air Cushioned Vehicles (ACVs), helicopters, and small craft that are ready to respond to the Government of Canada’s on-water and marine related needs. The sub-program includes Operational Personnel, Operational Support, Operational Business and Maritime Security units. Activities associated with the sub-program are guided by a number of international conventions, domestic legislation and Treasury Board policies.

The Fleet Operational Readiness program supports Coast Guard programs, science, fisheries and aquaculture management activities of the Department of Fisheries and Oceans (DFO), and the activities of a number of other government departments (OGDs) needing on-water delivery in support of their mandates. As of October 2014 the Coast Guard fleet consisted of 119 vessels (including 5 Air Cushioned Vehicles), over 1000 small craft and a fleet of 21 helicopters. In 2012-13 expenditures for the Fleet Operational Capability sub-program totaled $248.6 million.

EVALUATION METHODOLOGY

The evaluation of the Fleet Operational Capability sub-program covered a six-year period from 2007-08 through to 2012-13. The work was undertaken by the DFO’s Evaluation Directorate between November 2012 and October 2014. The evaluation covered the Coast Guard’s National Headquarters and its three regions: Western, Central and Arctic, and Atlantic. The logic model used for this evaluation is from the April 2012 performance measurement strategy developed for the Fleet Operational Readiness Program. The evaluation team developed the questions based on Treasury Board’s Policy on Evaluation, a review of program documents and evaluation planning discussions with key program personnel. The data collected during the evaluation drew on the following lines of evidence: document and website review; administrative data; literature review; key informant interviews with 19 staff and 33 clients; an online survey distributed to 249 program staff (excluding senior and junior seagoing as well as administrative staff); and, two case studies. Findings were then determined by triangulating the multiple lines of evidence with conclusions being drawn from the findings. This approach was taken in order to demonstrate reliability and validity of the findings and to ensure that conclusions and recommendations are based on objective and documented evidence.

EVALUATION FINDINGS AND RECOMMENDATIONS

Relevance

The evaluation found evidence that the Fleet Operational Capability sub-program is a fundamental component of the Fleet Operational Readiness Program and plays a central role in supporting the Canadian Coast Guard in responding to the Government of Canada�s on-water and marine related needs. Moreover, the sub-program aligns directly with federal and departmental roles and priorities, and it also contributes to supporting the priorities of other federal organizations. Lastly, the evaluation noted that economic growth in Canada depends heavily on trade and maritime commerce, which relies in turn on the Coast Guard and the support it provides to its clients in order to have the civilian fleet presence that supports a safe and secure marine sector. Consequently, in addition to the sub-program aligning with DFO's strategic outcome of Safe and Secure Waters, the evaluation provided evidence that the Fleet Operational Capability sub-program is also integral to the delivery of Coast Guard and DFO programs that align with the department�s two other strategic outcomes, namely Economically Prosperous Maritime Sectors and Fisheries, and Sustainable Aquatic Ecosystems.

Effectiveness

Overall, the evaluation found that the Fleet Operational Capability sub-program is delivering the planned operational days for a majority of its clients. This is supported by evidence that vessel availability was stable during the timeframe of the evaluation and that the number of days delivered were within reasonable tolerance targets of the operational days outlined in the Integrated Fleet Operational Plan. However, for the Environmental Response Services Program, the Fleet Operational Capability sub-program was unable to deliver the requested number of planned on-water exercise days for five of the six years covered by the evaluation. Although some improvement was noted in the latter years of the evaluation, the number of days actually delivered were outside the tolerance level established by the sub-program. Nonetheless, it is equally important to note that when Fleet Operational Capability vessels became unavailable the Coast Guard continued to have the capacity to respond to the on-water needs of the Environment Response Services Program through the use of the program�s fleet of pollution response vessels or through the reprioritization of other multi-tasked fleet vessels.

The Coast Guard fully supports Canada�s Marine Oil Spill Preparedness and Response Regime at all times. Within the framework of the regime, Transport Canada sets the guidelines and regulatory structure for the preparedness and response to marine oil spills. With respect to response, Canada's Marine Oil Spill Preparedness and Response Regime is based on the polluter-pay principle. The polluter is typically called upon to manage the response to a spill when it occurs and appoints an On-scene Commander. The response organizations provide the response required to manage and clean-up the spill and the Coast Guard monitors the overall response to ensure that it is effective, timely and appropriate to the incident. The Coast Guard takes the lead as on-scene commander during an incident if the polluter is unable to respond, is unwilling to take action or is unknown. Moreover, the evaluation did not find any evidence suggesting that the sub-program was unable to meet the needs of the Environmental Response Services Program when it came to responding to an actual environment response incident.

Efficiency and Economy

This evaluation can confirm that approximately 98% of the operational days delivered by Coast Guard were guided by Service Level Agreements (SLA) and Memorandum of Understanding (MOU). This was not the situation noted in the 2009 Evaluation of Fleet Operational Readiness. At that time the evaluation recommended that the Coast Guard ensure that Service Level Agreements are entered into between the Fleet Operational Capability sub-program (known as Fleet prior to 2009) and its clients.  The existence of these agreements are crucial as they are designed to set out the expectations of each party involved and cover activities such as accountabilities for program delivery including, planning and changes to plans as well as any information requirements for the clients. Most importantly, these agreements assist Coast Guard in further establishing the concept of Fleet Operational Readiness.

During the period covered by the evaluation the Canadian Coast Guard has undergone significant changes. This included its shore-based organizational structure being reduced from five to three regions as of October 2012. While the motivation for some of these changes can be traced back to a 2007 report from the Office of the Auditor General (OAG) recommending that the Coast Guard standardize its regional organizational structures in order to reduce inconsistencies and increase national guidance, they also represented the Coast Guard’s response to Strategic Review decisions originating from Budget 2011 as well as those of the Deficit Reduction Action Plan in anticipation of Budget 2012. Considering that this reorganization took place near the end of this evaluation, the outcomes are still too early to assess. In addition, Fleet Operational Readiness’ structure is relatively new, with the three sub-programs only becoming part of the department’s PAA mid-way through the evaluation.

The performance measurement strategy for the Fleet Operational Capability sub-program provided the evaluation team with the performance data necessary to support the evaluation. The only aspect requiring further development are the actual performance indicators for the long term outcome outlined in the Fleet Operational Readiness Program performance measurement strategy. In the absence of more concrete indicators for the long term outcome, the evaluation team identified “proxy” indicators in consultation with the sub-program.

The Integrated Fleet Operational Plan is an example of a best practice noted by the evaluation. The process ensures that the individual program requirements are aligned with proposed budget allocations and is affordable and deliverable. This process encouraged a sharing of information based on experiences and best practices across all regions. Lastly the plan was seen as being adaptable in order to reflect changes which might result from vessel breakdowns, weather delays and changes in client priorities.

Recommendations

With the above findings in mind, the evaluation has made the following two recommendations.

Recommendation 1: The Commissioner should ensure that the Fleet Operational Readiness program develops a strategy to meet the on-water exercise needs of the Environmental Response Services Program to fulfill Coast Guard program requirements.

Recommendation 2: The Commissioner should ensure that the Fleet Operational Readiness program builds on lessons learned during the early stages in implementing its performance measurement strategy pertaining to all three sub-programs.  As it relates to the Fleet Operational Capability sub-program its focus should be on the performance indicators for its long-term outcome.  In order to ensure interdependencies between all three sub-programs are explicit, Fleet Operational Capability should coordinate its efforts with the Fleet Maintenance and Fleet Procurement sub-programs to update the existing performance strategy of the Fleet Operational Readiness Program.

1. INTRODUCTION


1.1 PURPOSE OF THE EVALUATION

This report presents the results of the Fleet Operational Capability sub-program evaluation, one of three sub-programs of the Canadian Coast Guard’s Fleet Operational Readiness program. The evaluation is in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation, which requires all direct program spending to be evaluated every five years. The evaluation commenced in 2012, in accordance with the multi-year Departmental Evaluation Plan, and was completed in October 2014. Recommendations stemming from the main findings were developed to support improvements to the program and to inform future decision-making.

1.2 PARAMETERS

The main objective of this evaluation was to determine the extent to which the sub-program was managed effectively and efficiently and had achieved its intended outcomes. The evaluation covers the six-year period 2007-08 through 2012-13 and was undertaken between November 2012 and October 2014 by the Department of Fisheries and Oceans’ Evaluation Directorate. The evaluation is inclusive of the National Capital Region and the Coast Guard’s three regions: Western, Central and Arctic, and Atlantic. A previous evaluation of the Fleet, completed in 2009, covered the period 2002-03 through 2006-07. This evaluation builds on its findings and recommendations.

The two other sub-programs of the Canadian Coast Guard’s Fleet Operational Readiness program, Fleet Maintenance and Fleet Procurement, were evaluated separately and were reported on in March 2014. The extent to which all three sub-programs contributed to meeting operational, program and Government of Canada requirements is being assessed as part of the Fleet Operational Capability sub-program evaluation.

1.3 REPORT STRUCTURE

The executive summary provides a brief summary of the evaluation findings, conclusions and recommendations. Section 2 presents an overview of the Fleet Operational Readiness program and its sub-programs. Section 3 describes the evaluation methodology, followed by a discussion of the main findings in section 4, and conclusions and recommendations in section 5. Annex A presents the Management Action Plan. Annex B presents the key issues, questions, indicators and data sources used for the evaluation.

2. PROGRAM PROFILE


2.1 PROGRAM MANDATE

The Fleet Operational Readiness program, which is administered by the Canadian Coast Guard, is aligned with Fisheries and Ocean Canada’s Safe and Secure Waters strategic outcome, as reflected in the department’s PAA. This program is comprised of three components, the first being the focus of this evaluation report (i.e. Fleet Operational Capability sub-program). A separate evaluation report was issued in March 2014 and addressed the latter two components (i.e. Fleet Maintenance and Fleet Procurement sub-programs). The program contributes to maritime safety and security through the provision of maritime infrastructure which in turn contributes to safe navigation and the protection of life and property. This includes, for example, providing staffed vessels, ACVs, helicopters and small craft ready to respond to on-water and maritime-related requirements.

The Fleet Operational Readiness program supports Coast Guard programs, science, fisheries and aquaculture management activities of DFO, and the activities of a number of OGDs needing on-water delivery in support of their mandates. Such clients include the Department of National Defence (DND), Environment Canada, the Royal Canadian Mounted Police (RCMP), Canada Border Services Agency and Transport Canada.

Across the Federal Government, Budget 2010 announced a number of cost containment measures to reduce operating expenditures of government programs. The Coast Guard participated in both the Strategic Review decisions originating from Budget 2011 as well as the Deficit Reduction Action Plan in anticipation of Budget 2012. While its high-level strategic priorities have not changed, the Coast Guard subsequently restructured its shore-based organization to achieve greater efficiencies and to ensure that a nationally consistent approach aligned with Departmental and Government of Canada priorities.  These actions resulted in a reduction of management and overhead by shifting from the previous five (5) region to a three (3) region model. Certain directorates were refocused in headquarters and changes were made in the way that certain administrative functions were delivered both at headquarters and in regions.

As an operational Agency, the Canadian Coast Guard relies heavily on its vessels and shore-based assets to deliver maritime programs and services critical to Canadians. The document review noted that approximately 80% of the Coast Guard’s annual budget supports the concept of operational readiness.

2.2 SUB-PROGRAM ACTIVITIES

Fleet Operational Capability

The Fleet Operational Capability sub-program includes Operational Personnel, Operational Support, Operational Business and Maritime Security1 units. This sub-program ensures that certificated professionals safely, effectively, and efficiently operate vessels, ACVs, helicopters, and small craft that are ready to respond to the Government of Canada’s on-water and marine related needs.

  • Operational Personnel - responsible for all shore-based and seagoing operational personnel issues, including professional development and certification and operational directives and standards.
  • Operational Support - provides support to Senior Management through the National Co-ordination Centre, provides operational requirements for assets that shore and seagoing personnel use, and operational requirements analysis.
  • Operational Business - provides national analysis, planning, monitoring, and reporting for shore and fleet operations. The Operational Business Branch manages Coast Guard’s two capital Centres of Expertise (COE): Fleet COE and Equipment and Moveable Assets COE.
  • Maritime Security- provides departmental and interdepartmental policy coordination of national and maritime security issues, manages and leads the Coast Guards contribution to Marine Security Operations Centres, and is implementing the Incident Command System across the Canadian Coast Guard.

Activities associated with the Fleet Operational Capability sub-program are guided by a number of international conventions, domestic legislation and Treasury Board policies. All vessels and aircraft operate under codes of rigorous safety, security, personnel and maintenance management practices. The Canadian Coast Guard College is an important contributor to the delivery of this sub-program.


1 Maritime Security joined Operations in October 2012 as part of the restructuring of Coast Guard

2.3 EXPECTED RESULTS

Fleet Operational Capability

The expected result is an operationally capable fleet having the capacity to respond to the current operational needs and requirements of the Government of Canada. This is measured by determining the number of operational days delivered vs. planned. The target is 90%.

2.4 PROGRAM LOGIC MODEL AND PERFORMANCE MEASUREMENT

The Fleet Operational Readiness program has a performance measurement strategy in place. The strategy details the program and the three sub-programs (i.e. Fleet Operational Capability, Fleet Maintenance and Fleet Procurement) and covers the resources, physical structure and reporting relationships of the program. The program’s logic model, shown in Figure 1, informed this evaluation. The top row represents the Fleet Operational Capability sub-program.

The performance measurement strategy includes a matrix presenting the measures, targets, indicators and data collection process for assessing the effectiveness of each sub-program. The Evaluation Directorate’s December 2012 assessment of the strategy noted that further detail was required to better understand the program vision and objectives, how the program overall and sub-programs work, and risk mitigation strategies. Moreover, it was noted that the strategy would benefit from a logic model narrative explaining the underlying theory of change, the relationship between the boxes and flow of arrows.

Although the strategy was formally set in place at the latter part of the period being evaluated, it was decided in consultation with the Coast Guard that the Fleet Operational Readiness matrix would be used for the current evaluation of the Fleet Operational Capability sub-program as there was evidence that performance indicators were accessible and no other framework had been used previously. In essence, this evaluation afforded the opportunity to test the strategy to determine its usefulness going forward and assess whether any changes might be required.

Figure 1. Logic Model for Fleet Operational Readiness Program

Logic Model for Fleet Operational Readiness Program

2.5 PROGRAM STAKEHOLDERS

The Fleet Operational Capability sub-program ensures that certificated professionals safely, effectively, and efficiently operate vessels, ACVs, helicopters, and small craft that are ready to respond to the Government of Canada’s on-water and marine related needs. Figure 2 shows the distribution of Fleet Operational Capability Clients for fiscal year 2012-13. Its main clients are the combination of all six Coast Guard programs. DFO clients include, for example, programs within the Ecosystems and Fisheries Management sector, specifically Fisheries Management and Compliance and Enforcement, as well as programs within the Ecosystems and Oceans Science sector.  External stakeholders that rely on Coast Guard vessels and helicopters to support their programs include other federal departments and agencies, specifically Environment Canada, Natural Resources Canada, the Natural Sciences and Engineering Research Council of Canada, the RCMP and the DND.

Figure 2. Distribution of Fleet Operational Capability Clients

Distribution of Fleet Operational Capability Clients

Source: Fleet Operational Capability Data

2.6 GOVERNANCE

The Director General, Operations is responsible for the governance and delivery of the Fleet Operational Capability sub-program. This position reports directly to the Coast Guard Commissioner and is part of the Coast Guard Management Board – the Coast Guard’s senior decision-making body. This position and others, such as the Deputy Commissioners Operations and Vessel Procurement, Assistant Commissioners and Directors General, Integrated Technical Services, Major Projects, National Strategies and Integrated Business Management Services, contribute to maritime safety and security through the provision of maritime infrastructure which in turn contributes to safe navigation and the protection of life and property.

In May 2012, the Coast Guard announced the restructuring of its shore-based organization, to take place during 2012-13, with the aim of achieving greater efficiencies and national consistency. The Fleet Directorate merged its national operational oversight functions with similar functions carried out by Maritime Services to form a new Operations Directorate. Three other directorates are part of the organizational structure: Integrated Technical Services, National Strategies and Integrated Business Management Services. The composition of the Management Board remains much like before, with the four directorates represented by their Directors General.

The Operations Directorate in Ottawa works with regional Fleet Operations and Coast Guard Programs, who are then responsible to operationalize and carry out programs and services.

2.7 PROGRAM RESOURCES

Fleet Operational Capability operates annually with operating (vote 1) and capital       (vote 5) funds. Prior to 2012-13, the expenditures for Fleet Maintenance and Fleet Procurement were also included with Fleet Operational Capability.  However, starting in 2012-13 the expenditures for all three sub-programs are reported separately. The following table presents the data available for 2012-13 for the Fleet Operational Readiness program.

Table 1. Fleet Operational Readiness Resources

Fleet Operational Readiness Resources
2012-13 Financial Resources
$ millions Planned Actual Difference
Fleet Maintenance 61.4 83.9 22.5
Fleet Procurement 125.7 99.1 -26.6
Fleet Operational Capability 230.7 248.6 17.9
Total 417.8 431.6 13.8
Human Resources (FTEs) Planned Actual Difference
Fleet Operational Readiness 2,784 2,743 -41

Source: Departmental Performance Report 2012-13

2.8 RISK PROFILE

The following risks are aligned with the Coast Guard’s Corporate Risk Profile and reflect risks relevant to Fleet Operational Readiness as a whole.

  • Investment in Asset Base – There is a risk that the Coast Guard will be unable to procure and maintain the needed asset base, in a timely fashion, in order to deliver mandated services. The age of the existing assets is an inherent source of risk which needs to be addressed by increased maintenance and timely replacement.
  • Human Resources – There is a risk that the Coast Guard will be unable to sustain a sufficient and representative workforce with the appropriate competencies in order to adequately support, deliver and manage services.
  • Service Delivery – There is a risk that the Coast Guard will not meet the expectations and evolving needs of Canadians and its stakeholders with available resources. The ability to provide operational platforms is dependent on having the appropriate human resources, strengthened management and adequate fleet and land based assets.

3. METHODOLOGY


3.1 SCOPE

The evaluation of the Fleet Operational Capability sub-program covered a six-year period from 2007-08 through to 2012-13. The work was undertaken by the DFO’s Evaluation Directorate between November 2012 and October 2014. The evaluation covered the Coast Guard’s National Headquarters and its three regions: Western, Central and Arctic, and Atlantic. Senior program management assisted throughout the evaluation. In addition to participating in interviews, they provided documents and statistical data for the evaluation team to review, identified other interviewees and clients to contact, and provided feedback to the evaluation team on such items as the evaluation plan, survey questions and the evaluation report.

3.2 EVALUATION APPROACH AND DESIGN

A non-experimental design was chosen. The evaluation used a logic model combined with multiple lines of evidence drawing on both qualitative (e.g., interviews, program documents) and quantitative data (e.g., administrative data, survey). The various sources of input provided the basis for corroborating the evidence (triangulation) to arrive at valid findings and conclusions. The logic model used for this evaluation is from the April 2012 performance measurement strategy developed for the Fleet Operational Readiness Program.

Neither an experimental nor quasi-experimental design would be appropriate, as they are    not advisable for this sub-program. First, the activities delivered by the Fleet Operational Capability sub-program are not ones that can be randomly assigned into two groups, one which benefits from program initiatives and one which does not. Second, experimental-type designs are very time-consuming and resource intensive beyond what is appropriate and necessary to produce a timely evaluation of the program at reasonable cost. Rigour in a non-experimental design is achieved by virtue of using the combination of elements described above, i.e., the logic model and analysis based on multiple lines of evidence and both qualitative and quantitative data.

3.3 KEY ISSUES AND EVALUATION QUESTIONS

The evaluation matrix in Annex B presents the key issues investigated (relevance, effectiveness, efficiency and economy), the specific evaluation questions addressed and the corresponding lines of evidence used for each question. The evaluation team developed the questions based on Treasury Board’s Policy on Evaluation, a review of program documents and evaluation planning discussions with key program personnel.

3.4 DATA COLLECTION

The evaluation drew on the following lines of evidence to gather qualitative and quantitative data:

  • A document and website review examined various materials from the Fleet Operational Capability sub-program, the Canadian Coast Guard as a whole, Fisheries and Oceans Canada, including previous evaluation and audits, and other federal government and external sources.
  • Administrative data was sourced from Coast Guard and other departmental publications, as well as spreadsheet information provided by the program.
  • A literature review of fleet operations and management, based on 30 sources of information, contributed to addressing relevance and economy questions.
  • Key informant interviews were held with 52 respondents: 19 staff and 33 clients within the Coast Guard and DFO as well as other federal departments. These included interviews with the Directors General, Directors, and chiefs or senior managers in each region including National Headquarters. The majority of headquarter interviews were conducted in person, with regional personnel interviewed by phone.
  • An online survey of all program staff (excluding senior and junior seagoing as well as administrative staff) was also conducted. The survey had a response rate of 35%, with 88 respondents out of 249 invited.
  • Two case studies were undertaken. The first focused on activities in the North. This case study explored the impact of increased activities in the North, the implications for the sub-program and its response, and the contribution of the sub-program to meeting Canada’s evolving needs and interests in the North, particularly as part of the Northern Strategy. The second case study examined the Marine Security Enforcement Team program (MSET). Introduced in 2005-06, MSET is a joint program of the RCMP and the Canadian Guard Guard to provide an armed, on-water, fast-response enforcement capability on the Great Lakes-St. Lawrence Seaway. The MSET case study was chosen to build on a previous RCMP evaluation of the program to better understand relevance and economy of this program from the perspective of Coast Guard’s mandate.

The terms in Table 2 have been used in some instances to report the proportion or frequency of responses in place of numerical data.

Table 2: Proportion Terms for Percentage of Responses

Table 2
Percentage of Respondents Proportion Terms
All 100%
Almost all 90-99%
Most 80-89%
Over/more (e.g. half, one-quarter etc…)  equals 4 to 7% above
Less than/not quite (e.g. half, one-quarter etc…) equals 4 to 7% below
Around (e.g. half, one-quarter etc…) equals ± 4%
Close to/about (e.g. half, one-quarter etc…)  equals ± 2-3%
None 0%

3.5 ANALYTICAL METHODS

The data collected from each evaluation method detailed under Section 3.4 was summarized in separate lines of evidence to address each of the evaluation issues and questions contained in the evaluation matrix. The results from each line of inquiry were then integrated for each question and analyzed together to determine the overall findings and arrive at valid conclusions. In other words, findings were determined by triangulating the multiple lines of evidence. Conclusions were then drawn from the findings.

3.6 LIMITATIONS AND MITIGATION STRATEGIES

Although the evaluation encountered some challenges and limitations that are outlined below, these limitations were mitigated, as much as possible, through the use of multiple lines of evidence and triangulation of data. This approach was taken in order to demonstrate reliability and validity of the findings and to ensure that conclusions and recommendations are based on objective and documented evidence.

Survey of Program Staff

After consultation with the sub-program the survey was distributed to Coast Guard staff with responsibility for delivering the Fleet Operational Capability sub-program; namely, sea-going personnel (i.e. Commanding Officers) as well as individuals providing shore-based operational support and management services. As noted previously, Senior and Junior seagoing and shore-based administrative staff were not included in the staff survey. The response rate of 35% provided survey results which had a margin of error of +/- 8.5% (19 times out of 20)2. A total of 88 surveys were completed out of the possible 249 staff members invited to participate in the survey.

The survey remained open for 13 weeks to ensure that program staff were given sufficient time to respond. In addition to the initial email announcing the survey launch, three reminder emails were sent at approximately one month intervals. The timing of the reminder emails was chosen to best reach as many sea-going personnel as possible as well as those who had previously been out of the office. With each of the latter two reminders Senior Management were also asked to strongly encourage their respective staff to participate in the survey.

Surveying all sea-going personnel would have been a challenge as many of these staff do not have an individual DFO email address. However, as a mitigation strategy, the evaluation utilized the vessel based email assigned to the Commanding Officer so that all Commanding Officers of each vessel had an opportunity to respond to the survey. The opinions of Commanding Officers were thought to be critical as they have broad perspective of the operations of their vessel.

Key Informant Interviews

One possible limitation for the evaluation is that staff interviewees may overly emphasize the positive aspects of their program, thus limiting the credibility of information provided. The evaluation mitigated the potential for self-interest bias by structuring interview questions so that interviewees were not always commenting on areas where they have a vested self-interest. In addition, in order to mitigate the potential for bias, comments and findings were balanced with data from the other lines of evidence obtained during the evaluation.


2  http://fluidsurveys.com/survey-sample-size-calculator/

4. MAJOR FINDINGS


This section combines information from all the lines of evidence used and presents the findings according to the broad evaluation issues of relevance and performance which covers effectiveness, efficiency and economy.

4.1 RELEVANCE

Overall Findings

The evaluation examined whether the Fleet Operational Capability sub-program continues to be relevant. Specifically, it looked at whether it addresses an essential need, whether it aligns with the federal government’s role and priorities, and whether it aligns with departmental priorities. The evidence indicates that on all three accounts the sub-program continues to be relevant and essential.

Continuing Need for the Fleet Operational Capability Sub-Program


Key Findings:There is a need for Fleet Operational Capability sub-program in order to meet the Coast Guard’s civilian fleet obligations.


Canada’s Civilian Fleet Obligations

The Canadian Coast Guard owns, manages and operates a variety of marine assets in order to fulfill its mandate and obligations as the sole operator of the Government of Canada’s civilian fleet. As of October 2014 the Coast Guard fleet consisted of 119 vessels (including 5 ACVs), over 1000 small craft and a fleet of 21 helicopters.

The Coast Guard has formalized its obligations to use its vessels in support of various programs in service level agreements, memoranda of understanding and other agreements with clients and partners. The Coast Guard delivers its mandate through a wide range of activities. These include conducting search and rescue operations, placing and maintaining navigational buoys, providing marine communications and vessel traffic services, icebreaking, waterways management channel maintenance in navigable inland waters such as the Great Lakes-St. Lawrence Seaway system, ensuring an effective pollution response to all ship-source and mystery-source pollution spills into the marine environment and supporting other government departments, boards and agencies of the federal government.

The Coast Guard also supports DFO to carry out work in the marine and fresh water environment. For example, Conservation and Protection officers use Coast Guard vessels to patrol fishing areas and conduct inspections at sea to ensure compliance with all of the regulations designed to ensure orderly fisheries. The Coast Guard also supports marine science research, done by the Department, related to the conservation, protection and sustainable development of fisheries resources, fish habitat management, and navigational charts, as well as studying the influences of climate variations and aquatic ecosystems.

With the support of Coast Guard ships, the Canadian Hydrographic Service is able to produce marine charts. Natural Resources Canada conducts marine geology from Coast Guard ships and Environment Canada acquires much of its meteorological information from weather buoys launched from Coast Guard vessels.

The Coast Guard’s involvement in maritime security is based on its mandate under the Oceans Act to support federal departments and agencies mandated to provide security and enforcement within Canada. These organizations include DND, RCMP, the Canada Border Services Agency, Transport Canada and Public Safety Canada. Coast Guard’s primary role is to provide vessels and Coast Guard officers and crew help in developing an awareness of possible maritime threats to Canada’s security, provide on-water enforcement and responsiveness and to safeguard the safety and security of Canadians. The Coast Guard participates in the Coastal and Great Lakes-St. Lawrence Seaway Marine Security Operations Centres, led by DND and the RCMP respectively, to help detect, assess and support the response to any threat to marine security that could affect the safety, security, environment or economy of Canada. In addition, the Coast Guard is involved with the RCMP in the Marine Security Enforcement Team program. Together, they deliver a dedicated security enforcement capacity on the Great Lakes-St. Lawrence Seaway system.

The Fleet Operational Capability sub-program is vital to ensuring that marine assets staffed by certificated professionals are available to support Coast Guard’s obligations and delivery of federal government programs. Coast Guard assets must be able to operate for extended periods of time safely, consistently and effectively and often in harsh and hazardous conditions. In addition, the Coast Guard fleet supports on-water program delivery components not included in the mandates of other federal government departments.

Economic Growth

Economic growth in Canada depends heavily on trade and maritime commerce, which relies in turn on secure, sustainable harbours, safe waters, aids to navigation, marine communications and services, traffic management, icebreaking, channel maintenance, and reliable and modern hydrographic products and services. The programs and services supported by Coast Guard provide the maritime presence necessary in delivering direct and indirect services to the country’s marine sector. The document review noted that the marine sector transports, on average, 97% of Canada’s exports and 76% of its imports. Coast Guard supports the maritime economy and facilitates maritime commerce by ensuring the safe and efficient navigation of Canadian waterways. More specifically, Coast Guard provides essential search and rescue services, enables the on-water fisheries enforcement and science activities of DFO and supports various other departments and agencies that have a front-line role in maritime security. The case study noted that northern prosperity and development, in particular, are fostered by the secure access to Canada’s northern waters provided by Coast Guard icebreakers. Climate change, notably in the Arctic, is already extending the duration of the commercial shipping season, and may further intensify the demand for marine science and other activities. In addition, Northern communities benefit from supplies delivered by Coast Guard vessels.

Alignment with Federal Role and Priorities


Key Findings:The Coast Guard derives its mandate for a civilian fleet from federal legislation. Fleet Operational Capability aligns with the Government of Canada�s outcome area of a safe and secure Canada, Canada�s Northern Strategy and the interests and priorities of other federal departments.


Governing Legislation

Federal legislation gives the Minister of Fisheries and Oceans Canada and the Canadian Coast Guard responsibilities relevant to owning, operating and maintaining a civilian fleet. The Constitution Act, 1867, the Oceans Act, the Canada Shipping Act, 2001 and the Arctic Waters Pollution Prevention Act also have a bearing on the Coast Guard with respect to owning and operating a civilian fleet.

The Constitution Act, 1867, s. 91 (9, 10) confers on the federal parliament exclusive legislative authority over all matters related to navigation, navigational aids (such as beacons, buoys and lighthouses) and shipping on the country’s oceans and inland waterways. The Oceans Act (SC 1996, c. 31, s. 41-43) gives the Minister of Fisheries and Oceans responsibility for providing: aids to navigation; marine communications and traffic management services; icebreaking and ice-management services; channel maintenance;  marine search and rescue;  marine pollution response; and  support of other federal departments, boards and agencies by providing ships, aircraft and other marine services. The Canada Shipping Act, 2001 gives the Minister powers and responsibilities concerning: aids to navigation; search and rescue; pollution response; and vessel traffic services. The Arctic Waters Pollution Prevention Act (RSC, 1985, c. A-12) has a bearing on the extent of Canada’s jurisdiction in Arctic waters and hence the area to be covered by the Coast Guard with respect to its Environmental Response capabilities.

Federal Role

Interviewees and survey respondents consider that it is appropriate for the federal government to own and manage a civilian fleet. Such a fleet contributes to maritime safety and security while providing a platform in support of other government programs such as marine science research. Staff and clients interviewed cited the need for stable and consistent access to the diverse knowledge and skillsets necessary to provide services such as icebreaking, search and rescue, aids to navigation and scientific research, and Coast Guard vessels as a visible support to sovereignty and Government of Canada on-water presence.

Alignment with Federal Priorities

The document review confirmed that Fleet Operational Capability contributes to federal government priorities. It ultimately aligns with the Government of Canada’s outcome area for a safe and secure Canada, one of 16 outcomes identified in the federal government’s whole-of-government framework. It also aligns with Canada’s Northern Strategy priorities, presented in the federal government’s 2007 and 2013 Speech from the Throne. The Fleet Operational Capability sub-program’s contribution includes activities such as ensuring icebreakers are available in support of Canada’s sovereignty presence and through such missions as escorting other vessels, transporting dry cargo and fuel to isolated northern communities that are not connected by road or rail, conducting science research, and supporting search and rescue missions.

Coast Guard vessels support the interests and priorities of other federal departments and agencies such as Environment Canada, Natural Resources Canada, National Research Council Canada and Aboriginal Affairs and Northern Development Canada. Coast Guard vessels are used to assist the DND and the RCMP in sovereignty and security exercises and operations. For example, the Coast Guard participated with other federal departments in a Canadian Forces-led northern Arctic sovereignty and emergency preparedness exercise. During the 2010 Olympic and Paralympic Games, Coast Guard vessels were deployed to assist with marine security.

Further, the case study noted that since the introduction of the National Security Policy in 2004, Coast Guard’s support role has changed from mainly one of responding to periodic and contingency needs to one of also providing ongoing support for security initiatives — a prime example being the joint operations with the RCMP in the Great Lakes-St. Lawrence Seaway area. Use of Coast Guard vessels provides the MSET program with an operational capability it otherwise would not have.

The same 2004 National Security Policy lead to the creation of three Marine Security Operations Centres.  The Centres are located on the East and West coasts as well as in the Great Lakes-St. Lawrence Seaway area and are unique examples of inter-agency integration.  DND is the lead for the two coastal Centres and the RCMP is the lead for the Great Lakes-St. Lawrence Seaway. Participating departments include DND, RCMP, the Canada Border Services Agency, Transport Canada, DFO and the Coast Guard.

Alignment with Departmental Priorities


Key Finding: Fleet Operational Capability aligns directly with DFO’s strategic outcome of Safe and Secure Waters. It also contributes to the department’s two other strategic outcomes.


As a Special Operating Agency of the Department of Fisheries and Oceans, the Coast Guard helps DFO meet its strategic outcome to ensure safe and secure waterways for Canadians. The Fleet Operational Capability sub-program is one of three under the Fleet Operational Readiness Program. 

The document review, staff interviewees and staff survey all confirmed that the Fleet Operational Capability sub-program aligns with DFO’s strategic outcome of Safe and Secure Waters. The Coast Guard’s Fleet Operational Readiness program, through the Fleet Operational Capability sub-program, as well as Fleet Maintenance and Fleet Procurement sub-programs, is responsible for providing safe, reliable, available and operationally capable vessels, ACVs, helicopters and small craft in support of the department’s mandate and other federal government on-water requirements. It serves as a mechanism for the delivery of Coast Guard and Government of Canada’s on-water mandate and its operations are client focused. The Coast Guard has a responsibility to support other federal departments, such as the RCMP, as specified in Section 41(e) of the Oceans Act. Joint programs such as MSET enable its participation in marine security activities and are directly aligned with the Department’s strategic outcome of Safe and Secure Waters.

In addition to the sub-program aligning with DFO’s strategic outcome of Safe and Secure Waters, the document review and interviews provided evidence that the Fleet Operational Capability is also integral to the delivery of DFO programs that align with the department’s two other strategic outcomes, namely Economically Prosperous Maritime Sectors and Fisheries, and Sustainable Aquatic Ecosystems. For example, Coast Guard crew and vessels are used to deploy and retrieve buoys in support of marine navigation and by DFO’s Compliance and Enforcement program in support of fisheries. As interviewees mentioned, DFO could not fulfil its marine science mandate, such as fisheries research or hydrographic surveying, without a fleet. In essence, Fleet Operational Capability supports all three departmental strategic outcomes.

4.2 EFFECTIVENESS

The evaluation examined whether vessels, ACVs, helicopters and small craft were operational with the required certificated professionals on board  (immediate outcome) and whether an operationally capable fleet has the capacity to respond to the current operational needs and requirements of the Government of Canada (intermediate outcome) in support of the federal government’s on-water needs and priorities (long-term outcome). It also investigated whether external factors or challenges likely had an impact on the results and whether the Fleet Operational Capability sub-program resulted in any unintended outcomes.

IMMEDIATE OUTCOMES


Key Finding: Vessels, ACVs, Helicopters and small craft were operational with the required certificated professionals on board in order to meet program requirements.


Utilization of Vessels by Operational State

The indicator for the achievement of this outcome is the number of operational days vessels were available versus unavailable. As a result of the Coast Guard’s transition from the FAIS database to the iFleet database, the evaluation team analyzed the data from the beginning, middle and final years of evaluation when reviewing of this indicator. The data in Table 3 shows that this indicator has remained relatively stable and in the final year of the evaluation reached 72% vessel availability.

Table 3. Utilization of Vessels by Operational State

Utilization of Vessels by Operational State
Available Unavailable Total
Assigned Un-assigned Sub-total Lay-up Maintenance Other Sub-total
Planned Unplanned
2007-08
67% 1% 68% 19% 9% 3% 2% 32% 100%
2010-11
65% 1% 66% 20% 10% 2% 2% 34% 100%
2012-13
72% 0% 72% 15% 10% 1% 2% 28% 100%

Sources: Fleet Annual Reports 2007-08 and 2010-11;iFleet

This same table shows that vessel unavailability also remained relatively stable during the time period of the evaluation. These two pieces of evidence, taken as a whole, contribute to the explanation of availability for programming. This initial assessment of vessel availability is crucial in the planning process and is necessary in order to support the program requirements of clients. Subsequent to this, clients are invited to identify their programming requirements which are expressed in the number of operational days or helicopters hours required and the time of the year they are required. Many of the requirements are based on historical trends and are often based on set timelines for the client to deliver their programs. For example, there are only certain times of the year that DFO Science can do scientific surveys.

Vessel Availability and Operational Planning

Coast Guard prepares an Integrated Fleet Operational Plan every year based on both the availability of vessels to deliver the service and the requirements that have been identified by its clients. What is not clear is what might constitute an appropriate percentage for vessel availability. Considering that vessel unavailability is largely comprised of planned and unplanned maintenance as well as vessel layup both of which are necessary activities regardless of the age of a fleet, 72% availability is conceivably a reasonable percentage to demonstrate achievement of this immediate outcome. However, the evaluation team was unable to make this determination as the Coast Guard had not identified a performance target for this indicator.

Evidence from staff interviews and survey corroborate this finding with most of those interviewed and a little less than half of the staff survey respondents commenting that the program’s immediate outcome of operational vessels, ACVs, helicopters and small craft with the required certificated professionals on board was being met to “a considerable or great extent of the time”.

Several individuals from both the staff and client interviews noted that the Coast Guard is subject to policies that mean having the required certificated personnel on board is necessary for the vessel to sail. Although it has not yet adversely impacted the Coast Guard’s ability to ensure that sufficient certificated professionals are available, staffing shortages and competition with the private sector for qualified personnel were noted as issues by around one-third of the staff interviewees.  Moreover, approximately one-quarter of those surveyed answered that they “disagree or strongly disagree” with the statement that the program has sufficient certificated on-board officers and crew to undertake scheduled missions.

During the period covered by this evaluation, Coast Guard continued to focus on its human resource challenges. In the case of Fleet Operational Capability, the document review noted that in addition to the historical trend showing that attrition is approximately 4.5% per year, it is projected that an estimated 20% of all indeterminate, seasonal and term over three months Ships’ Officers, Ship’s Crew and Seagoing Personal will be eligible to retire by 2016. Other pressures, such as the requirement for more marine engineers to meet the needs of implementing the Vessel Maintenance Management strategy, will further increase the demand for competent marine personnel. The document review noted that some initiatives are well underway to help address these challenges. These include increased intake of Officer-Cadets at the Coast Guard College, launching the implementation of the Ship’s Crew Certification Program, enhanced arrangements with provincial nautical schools, and significant progress on recruitment and retention led by the Coast Guard National Labour Force Renewal Group.

IMMEDIATE OUTCOMES


Key Finding Fleet Operational Capability: The evaluation found evidence of an operationally capable fleet having the capacity to respond to the needs and requirements of the government of Canada.


Days Planned vs. Days Delivered

The measure of the sub-program’s success is based primarily on the number of days delivered versus days planned. Figure 3 indicates that the sub-program consistently responded to the needs of a majority of its clients. For example, in 2012-13 this resulted in 44,779 days delivered versus 45,587 days planned representing the delivery of approximately 98% of the operational days planned.

Figure 3. Days Planned vs. Days Delivered3

Days Planned vs. Days Delivered Graph

Source: Fleet Operational Capability data


3 Coast Guard noted that the decrease in the number of actual days planned versus days delivered between 2009-10 and 2010-11 are due to changes in data definitions and transition from FAIS to iFleet and did not represent an actual reduction.

Approximately half of staff interviewed as well as those surveyed responded that the intermediate outcome of an operationally capable fleet which has the capacity to respond to the operational needs and requirements of the Government of Canada is being met to “a considerable or great extent of the time”. However, a little more than one-quarter of the clients stated that this capacity is becoming “very limited” due to the age of the fleet and, while thus far operational needs and requirements are mostly being met, there is little ability for the Coast Guard to meet unplanned demands without impact to another program.

Strategy to meet the operational needs

Evidence from the document review and key informant interviews suggest that in order to have the capacity to meet the operational needs and requirements of the Government of Canada, Coast Guard fully deploys and assigns all its available assets to programming activities. As a consequence there are no backup vessels or little flexibility in the event of vessel breakdown or increased demand. Although program data shows that vessel breakdowns and delays have decreased between 2007/2008 and 2012/2013, when it does occur it has an adverse impact on programming. In some instances clients interviewed responded that they were not able to complete the planned programming. The evaluation did not assess if certain programs were more affected than others by vessel breakdowns (i.e. because of the length of their missions) and/or by changing priorities when a vessel is multitasking (i.e. when a vessel performs two or more tasks simultaneously).

Slightly less than one-half of all client respondents answered that the sub-program was able to provide the right vessel for their program requirements “a considerable extent of the time”. Roughly one-third of clients responded that the sub-program was able to provide the right vessel “a moderate extent of the time”, noting that with multitasking you do not always get the optimum vessel to support their program requirements.  Respondents recognized that the optimal vessel is not always available and most of the clients interviewed noted that the crew provided were optimal in supporting their program requirements “a considerable extent of the time”.

Existence of a Functioning Fleet Safety and Security Manual

Part of the evidence demonstrating that Coast Guard is operationally capable in meeting the needs of the Government of Canada includes the commitment to maintain a work environment that complies with, and at times exceeds, regulatory health, safety and security requirements. The Coast Guard Fleet Safety Management Manual describes the management system for the safe operation of ships and for the prevention of pollution by ships. Evidence from staff interviews and survey emphasized that roles, responsibilities and accountabilities communicated through the Safety and Security Manual are well documented. The results indicate that Coast Guard operates in a safe and secure manner as demonstrated by Figure 4 depicting the number of hazardous occurrences4. More specifically, the trend shows the number of occurrences were consistently below target for the period covered by the evaluation.

Figure 4. Number of hazardous occurrences, as per the safety management system

Figure 4 Graph

Figure Source: National Coast Guard Safety Management System Annual Review for 2012-2013


4 A Hazardous Occurrence is an accident/illness or a near-miss rising out of, linked with, or occurring during the course of employment that results in or has the potential to result in personal injury or damage to property, equipment or pollution to the marine environment.

This evidence appears to indicate an improvement in the operational readiness. Lastly, almost all client interviewees responded that they are confident that the Coast Guard takes all possible measures to ensure that service delivery is done in a safe and secure environment.

Existence of Service Level Agreements /Memorandum of Understanding

The purpose of the current evaluation is not to assess these agreements themselves. Rather, the responsibility of measuring the service performed against the agreed upon expectations outlined in the agreements belong to the Coast Guard in collaboration with its respective clients. This also includes regular reporting to the client programs on the service provided.  However, this evaluation can confirm that approximately 98% of the operational days delivered by Coast Guard were guided by SLAs and MOUs. Which had not been the situation noted in the 2009 Evaluation of Fleet Operational Readiness.  At that time the 2009 evaluation recommended that the Coast Guard ensure that agreements are entered into between the Fleet and its clients.

The existence of these agreements are crucial as they are designed to set out the expectations of each party involved and cover activities such as accountabilities for program delivery including, planning and changes to plans as well as any information requirements for the clients. Most importantly, these agreements assist Coast Guard in further establishing the concept of Fleet Operational Readiness. A concept that can be traced back many years but was more formally adopted in fiscal year 2007-2008 when it replaced Fleet Services as a sub-activity in the Program Alignment Architecture. The concept is centered on the theme of ensuring that the Coast Guard has the means and the ability and the appropriate management functions to respond to the on-water and marine related needs of its clients in a safe, timely and effective manner.

These agreements also play a vital role in executing the Integrated Fleet Operational Plan which is funded by clients based on their approved planned operational days. In the case of Coast Guard programs, the funds required to support them are included directly in the Fleet Operations budget. Whereas for DFO programs, these funds are transferred from DFO programs to the Fleet Operations for their approved planned operational days.

Almost all respondents noted that the agreements contributed to the efficient delivery of service. The most common responses were that they have allowed both parties to clarify issues and were good tools for communications. The document review noted that many of the agreements were originally implemented as pilot projects and have since been, or are scheduled to be re-negotiated.

LONG-TERM OUTCOME


Key Findings: The evaluation found evidence to confirm that an operationally capable fleet was available to respond to a majority of the needs and requirements of the Government of Canada. However, interviewees and survey respondents noted their concern due to little or no back up with limited flexibility in the event of a vessel breakdown or increased demand. This combined with the fact that over 70% of the Coast Guard fleet is beyond the half-way point of their remaining operational life may impact operational readiness in the future should there be any significant increase in demand or unexpected delays in the delivery of new vessels and/or systems.


Evidence of access to an operationally ready fleet

Evidence from the 2014 Evaluation of the Fleet Maintenance and Fleet Procurement sub-programs demonstrated that the Coast Guard is making progress toward renewing the fleet and ensuring that Canada has access to an operationally ready civilian fleet. More specifically, since 2005, the federal government has invested $6.8 billion in the Canadian Coast Guard’s fleet:

  • A total of $1.4 billion in budgets 2005, 2007, 2008 and 2010 ― to acquire nine mid-shore patrol vessels, four scientific research vessels (three for fishery science, one for oceanographic science), a hovercraft and Canada’s first polar icebreaker;
  • $175 million in Canada’s 2009 Economic Action Plan ― for major repair work on 40 large vessels and to acquire 98 new small vessels and craft; and
  • $5.2 billion as part of Canada’s 2012 Economic Action Plan ― to procure up to 10 large vessels, 18 to 21 small vessels and up to 24 helicopters; to provide additional funding for the polar icebreaker project; to extend the lives of 16 Coast Guard vessels; and to complete mid-life modernizations on two hovercraft. 

Running parallel to the procurement of new assets were strategic investments in vessel maintenance and vessel life extension; this measure was required in order to ensure that older vessels could be maintained in operation until new vessels could be put into service in support of federal programs. This strategy seems to be working as Figure 5 indicates that Coast Guard experienced a reduction in the number of operational days lost due to delays during the timeframe of this evaluation.

Figure 5. Service Time Lost Due to Delays

Figure 5 Graph

Source: FAIS & iFleet

The document review noted that the primary reasons for under delivery were vessel breakdown, weather delays and changes in client priorities or because of changes to government priorities (e.g. the reassignment of the planned programming to another program).  Figure 5 shows that Fleet Operational Capability has reduced its service time lost due to delays from 3% in 2007-08 to 1% in 2012-13. Most notably, in 2012-13, delays were at the lowest point of the past five years at only 1 percent of overall availability.

Days planned versus days delivered

It should be noted that there have been variances between individual client programs with some clients receiving more than originally planned. However, over the timeframe of this evaluation a majority were well within the Coast Guard normal tolerance range5. For example, services delivered to Search and Rescue, Maritime Security and Oceans Science programs were all within the tolerance range. Whereas the service delivered to Icebreaking and Ecosystems and Fisheries Management fall within the reasonable tolerance level.

In cases where actual services delivered exceeded the planned number of days, higher than expected or unforeseen events required more days to be delivered.  Such is the case for the Search and Rescue Program. Lastly, the document review noted that the time spent delivering other government department programs was significantly higher than originally planned due to the federal government’s increased emphasis on its Northern Strategy.

At an overall level, Table 4 provides evidence that the Coast Guard is able to consistently deliver the number of planned service days to its clients.  At this global level the Coast Guard has a target of meeting 90% of planned service days.

Table 4. Percentage of service delivered versus service planned for each program

Table 4
Programs 2007-2008 2008-2009 2009-2010 2010-2011 2012-2013 Individual Client 5-year average
Search and Rescue 107% 106% 103% 101% 109% 105%
Aids to Navigation 82% 86% 88% 82% 104% 88%
Icebreaking 101% 100% 90% 86% 93% 94%
Maritime Security 120% 80% 100% 91% 97% 98%
Environmental Response* 44% 120% 51% 67% 67% 70%
Marine Communications & Traffic Services 89% 154% 192% 248% 180% 173%
DFO Ecosystem and Ocean Sciences 97% 99% 102% 94% 98% 98%
DFO Ecosystems and Fisheries Management 85% 92% 98% 86% 95% 91%
Other Government Departments 100% 127% 150% 122% 109% 122%
Overall 5-year average for all Clients 92% 107% 108% 109% 106%  

Source: FAIS and iFleet data

* For Environmental Response, the planned service days are to conduct on-water exercises.


5 The sub-program aims to deliver 100% of service demands. The normal tolerance range is plus or minus 10%, given operational, environmental and program fluidity.

Fleet Operational Capability Clients

As noted in Table 4, service delivery remains close to targets established in the Integrated Fleet Operational Readiness Plan for the majority of Fleet Operational Capability Sub-Program clients. The only exceptions are with the Aids to Navigation Program, DFO Ecosystems and Fisheries Management and the Environmental Response Services Program. Although below the requested levels of service for both Aids to Navigation Program and the Ecosystems and Fisheries Management Program the percentage of service delivered were reasonably close to the tolerance levels identified by the sub-program. However, on-water exercise days for the Environment Response Services Program continue to be under delivered. For example evidence from the document review noted that in 2010�11, 100 days were planned for Environmental Response to conduct on-water exercises, while 67 days were actually delivered. Although this represents an improvement over the 2009�10 level and could be considered as evidence that the sub-program is making greater efforts to meet the on-water exercise needs of the Environmental Response Services Program, it still represents a value of approximately 30% below the number of on-water-exercise days requested and approved in the Integrated Fleet Operational Plan.

It is equally important to note that when Fleet Operational Capability vessels became unavailable the Coast Guard continued to have the capacity to respond to the on-water needs of the Environment Response Services Program through the use of the program�s fleet of pollution response vessels or through the reprioritization of other multi-tasked fleet vessels.

The Coast Guard fully supports Canada�s Marine Oil Spill Preparedness and Response Regime at all times. Within the framework of the regime, Transport Canada sets the guidelines and regulatory structure for the preparedness and response to marine oil spills. With respect to response, Canada's Marine Oil Spill Preparedness and Response Regime is based on the polluter-pay principle. The polluter is typically called upon to manage the response to a spill when it occurs and appoints an On-scene Commander. The response organizations provide the response required to manage and clean-up the spill and the Coast Guard monitors the overall response to ensure that it is effective, timely and appropriate to the incident. The CCG takes the lead as on-scene commander during an incident if the polluter is unable to respond, is unwilling to take action or is unknown. Furthermore, the sub-program and specialized Environmental Response personnel are on standby 24 hours a day, seven days a week, to investigate and respond to pollution incident reports received regionally, nationally and internationally, and to work with commercial partners to monitor and manage cleanup efforts. Moreover, the evaluation did not find any evidence suggesting that sub-program was unable to meet the needs of the Environmental Response Services Program when it came to responding to an actual environment response incident.

The Environmental Response Services Program is responsible for ensuring the cleanup of ship-sourced spills of oil and other pollutants into Canadian waters. This includes:

  • monitoring cleanup efforts by polluters; and
  • managing cleanup efforts when polluters are unknown, or unwilling or unable to respond to a marine pollution incident.

Other aspects of the program are conducting on-water exercises as well as developing regional and area response plans for program personnel when responding to a spill in order to improve Canada�s response capabilities.

Very Little Redundant Capacity

Overall, both staff and client interviewees responded that the Government of Canada currently has access to an operationally ready fleet. This combined with evidence from Coast Guard that service time lost due to delays is shrinking suggests the Government of Canada continues to have access to an operational ready fleet.  However, most interviews with staff and clients noted that the fleet is currently operating with very little redundant capacity so when breakdowns or other unplanned events occur and vessels are reassigned; there is an impact on the programs.

Figure 6 demonstrates that Coast Guard still faces significant challenges to ensure continued access to an operationally ready fleet stemming from the advanced age of many of its currently-operating vessels. The remaining operational life for large and small vessels is 24% and 28% respectively. Any significant delays in the delivery of new vessels and/or systems or the unexpected breakdown may impact operational readiness and consequently Government of Canada access to an operational ready fleet. The remaining operational life for helicopters is 3%. The Coast Guard is currently renewing the entire fleet of helicopters, with first expected delivery in 2014-15.

Figure 6. Percentage of Operational Life Remaining

Figure 7 Graph

Source: Fleet Operational Capability Data

EXTERNAL FACTORS, CHALLENGES


Key Findings: The evaluation found evidence that, although external and internal factors and/or challenges may have influenced results of the program, there are established systems in place to monitor and mitigate these risks.


Evidence of established systems in place to monitor and mitigate internal/external risks

Through the document review the evaluation found evidence that Coast Guard Corporate Risk Profiles were prepared in 2006, 2009 and 2014. These risk profiles provided risks at the Agency-level but did not provide specific risks for the Fleet Operational Readiness program nor the Fleet Operational Capacity sub-program. The Coast Guard Strategic Business Management Branch of the National Strategies Directorate is currently developing a risk profile specifically for the Program.

External Risks

Human Resources

The first external risk concerns Human Resources. The evaluation team recognizes that this risk could also be categorized as an internal risk, however, for the purpose of this evaluation the issue will be presented as an external risk. This risk was rated as very high in the 2006 Coast Guard Risk Profile and by 2014 had decreased to a moderate/high. A little more than one-quarter of the staff interviewees noted that the Coast Guard is facing an increasingly competitive labour market particularly with respect to other marine sector industries. This impacts the Coast Guard’s ability for recruitment and retention of skilled employees. The staff survey and the document review also pointed to challenges with respect to the demographic shifts notably the rising attrition rates in key Coast Guard positions (Executives, Ship’s Crew, Ship’s Officer, Engineering and Scientific Support, General Technician) and the impact of the size and availability of the skilled workforce. The threats related to this risk will be difficulties related to managing the workload (a loss of corporate memory, lack of sufficiently-skilled employees and therefore an increased need for corporate oversight). This threat might also accentuate gaps with regards to employment equity and compliance with official languages policies.

Key controls to avoid these threats include an array of Human Resource planning efforts and the use of formal career progression programs. On the other hand, opportunities created by this challenge is to use existing (e.g. Blueprint 2020) and new human resources tools (e.g. Coast Guard Performance Management Directive) in combination with the Coast Guard’s good reputation to attract and retain skilled employees.

Fuel Prices

The document review and staff survey found fuel prices as a second external risk for Fleet Operational Capability. Although this operational risk is not specifically identified in the Coast Guard Corporate Risk Profiles, the 2007/08 Fleet Annual Report notes that a fluctuation of 1 cent in fuel prices has an impact of $630,000 per year for the fleet. This risk has been mitigated by the program with management tools such as the introduction in 2009 of Coast Guard Fleet Fuel Financial Management Policy. As well, in 2012, there was a review of the current fuel management and purchasing strategies.

Internal Risks

Age of fleet

The age of vessels was identified as a prominent internal risk in staff and client interviews as well as the document review.  Drivers for this internal risk include; impacts from funding shortfalls and deficit action plans, assets exceeding their operational life, difficulty in divesting assets in a timely manner and project management constraints (e.g. bargaining powers with suppliers, increased central agency requirements). This internal challenge was ranked as a high risk in 2006 in the Coast Guard Corporate risk profile and there was minimal change in 2009 and 2014. Threats from this risk include negative impacts on program delivery, exacerbating conditions of assets in a precarious state, increased exposure to liability and litigation as well as the Coast Guard’s reputation.

Coast Guard has control measures in place for these threats notably by modifying operations management (e.g. ensuring regular vessel condition assessments, adjusting service level agreements in collaboration with clients) and through project management (e.g. establishing good relations with marine industries and suppliers, contributing to the implementation of the National Shipbuilding Procurement Strategy). Additional mitigation strategies include the start of several vessel life extensions, the delivery of new fleet assets and helicopters and the development of an Asset Management Plan.

Governance

Governance is the second internal risk. As such, a moderate risk rating was given to this issue. In 2007, the Office of the Auditor General (OAG) identified the need for the Coast Guard to standardize its regional organizational structures in order to reduce inconsistencies in regional procedures and practices. The evaluation found that both of these OAG recommendations have been addressed but may still require fine-tuning. First, in 2009 the Coast Guard launched the implementation of a standard organization that would require three years to complete. Secondly, in 2012, the Coast Guard adopted a new concept of operations which had five key objectives one of which was the reduction of management and overhead by merging several regional organizations. Furthermore in the 2014 Coast Guard Risk Profile, the Coast Guard’s transformation risk was identified as medium-high due to breadth and depth of the organizational change to roles and responsibilities in the new governance structure. Indeed, the staff interviews and staff survey rated the governance and management within regions as functioning “to a moderate extent or more” and between regions it was rated as being “moderate” compared to governance and management at Headquarters and between Headquarters and regions which were generally rated as functioning “to a moderate extent or less”. It should be noted that the Coast Guard reorganization took place near the end of this evaluation therefore the outcomes of this initiative are still too early to assess.

4.3 EFFICIENCY AND ECONOMY

Overall Findings

The evaluation examined whether the Fleet Operational Capability sub-program has the appropriate governance and management processes and systems in place to support program delivery, whether roles and responsibilities are clear, and whether the program is using its resources efficiently. Over the period of the evaluation the program has addressed issues related to governance, management structures and performance monitoring. The program also adopted practices which favour operational and allocative efficiencies and the minimization of resources.

Governance and Management Structure


Key Findings:The Coast Guard’s new organizational structure and the concept of operations approved in 2013 were put in place to address recommendations from the 2007 report of the OAG which called for more regional standardization and national guidance. As well, Fleet Operational Capability monitors program progress using a performance measurement strategy approved in 2012 and has signed Service Level Agreements, which include performance measurement frameworks, between itself and its clients.


Governance and Committee Structure

Between 2007/2008 and 2011/2012, the Fleet Program, which became the Fleet Operational Capability sub-program in the 2008-2009 DFO PAA, was supported by the Fleet Executive Board, its subcommittees as well as corporate oversight committees (workforce, investment, safety and security and College). A 2007 OAG report noted that the Coast Guard required a more coherent national institution notably with respect to standardizing its regional organizations as well as providing national guidance.

In 2012 the Coast Guard started implementing its Shore-based Organizational Structure whose aim was to both standardize regional structures and to streamline the agency by reducing the number of regions from five to three.

As well, in 2013, the Coast Guard approved a new concept of operations. This model strengthens national guidance by integrating the on-water portions of its programs6 in an Operations Directorate, which includes Fleet Operational Capability. Moreover, within the current concept of operations, although the committee structure within the Coast Guard stayed the same, the composition and mandate of committees have changed. Figure 7 presents the current Coast Guard’s committee structure and affiliated committees.


6 Search and Rescue, Aids to Navigation, Environmental Response, Icebreaking, Marine Communications and Traffic Services.

Figure 7. Governance Structure

Figure 8 Governance Structure

This new concept of operations ensures that all governance bodies are aligned with the strategic direction provided by the Management Board and that the organization improves communication flow between management committees and management and employees. For example, the Operations Directorate reports to a single functional management committee, the Operations Executive Board, strengthening alignment between the Fleet Operational Capability sub-program and Coast Guard’s program delivery.

Corollary, the staff interviews and staff survey revealed that governance and management at National Headquarters (NHQ) and between NHQ and regions were perceived as being weaker than within regions and between regions. However, this perception might be related to the extensive organizational changes that have occurred within NHQ since 2012/2013 making it difficult for regional staff to identify their NHQ counterpart(s).

Performance Monitoring

Performance Measurement Strategy

The performance measurement (PM) strategy is a results-based management tool that is used to guide the selection, development and ongoing use of performance measures. Its purpose is to continuously monitor and assess the results of programs as well as the economy and efficiency of their management; to make informed decisions and take appropriate, timely action with respect to programs; to provide effective and relevant departmental reporting on programs; and to ensure that credible and reliable performance data are being collected to effectively support evaluation.

A review of program documents noted that as of April 2012, the Fleet Operational Readiness Program, which includes the Fleet Operational Capability sub-program, had a PM strategy in place as required by the Treasury Board’s Policy on Evaluation. This is a considerable step forward since this provides the program with its first senior management approved program profile, program objectives, logic model and PM indicators. Indeed, certain PM indicators from this strategy were used in this evaluation report.

Performance Frameworks in Fleet Operational Capability Service Agreements

Performance monitoring of Service Level Agreements (SLA) with Coast Guard clients are also in place covering nearly all operational days. However, performance measurement frameworks included within these service agreements may offer opportunities for improvement. In one case, the SLA PM Framework was still under development even though the SLA was officially signed. In other cases, the PM Framework was a basic outline rather than a complete PM matrix. As well, in some SLA annual performance reports several indicators identified in the PM framework were missing. Lastly, not quite half of the clients interviewed noted that the annual performance reports were provided to their program “to a moderate extent or less”. Perhaps further attention to PM frameworks in SLAs would strengthen discussions between Fleet Operational Capability and their clients regarding performance and performance monitoring. Considering that the various SLAs expire on different timelines, this finding will not result in a formal recommendation in the evaluation report per se but the Fleet Operational Capability sub-program should consider it as they renew existing or negotiate new SLAs.   

Complement, duplicate or overlap?


Key Findings: The Fleet Operational Capability sub-program does not duplicate or overlap any other federal, provincial or territorial programs. Indeed, Fleet Operational Capability vessels are assets which dovetail the needs of DFO programs and other federal, provincial and territorial departments. As well, use of alternative service delivery provides a necessary complement to the Fleet capacity.


Overlap or duplication

The document review noted that the Oceans Act and the Canada Shipping Act, 2001 provide a specific mandate to the Coast Guard which is not duplicated and does not overlap other federal, provincial or territorial legislation. As well, almost all staff and client interviews noted that Fleet Operational Capability complemented other public and private fleets in Canada. For example, the case study noted that the Rush-Baggot Treaty of 1817 precludes the use of armed vessels by Canada and the United States in the Great Lakes. Therefore since the Canadian Navy vessels may not be used in this location, a civilian fleet, such as the Coast Guard vessels, is complementary with respect to the Navy. In terms of the Marine Security Enforcement Teams on the Great Lakes and St-Lawrence, RCMP vessels and those of the Fleet Operational Capability have distinct designs underlining those assets’ specific roles when undertaking security missions. Furthermore, interviewees and survey respondents did not identify any operational activities undertaken by Fleet Operational Capability which may lie outside of their mandate.

Alternative Service Delivery

Of particular interest to this question was the practice of using alternative service delivery (e.g. chartering vessels and helicopters, the Coast Guard Auxiliary, contracting certain Aids to Navigation work) to ensure operational capacity. Almost all staff and client interviews as well as the staff surveyed thought that the use of alternative service providers complements Coast Guard capacity in an economical manner.

Opportunities for Improving Efficiency


Key Findings: Overall, Fleet Operational Capability provides an economically efficient service to its clients. The program strives for both operational and allocative efficiencies in order to optimize its use of resources.


Economically Efficient Services to Clients

Almost all the staff interviewed and just above three-quarters of staff survey respondents thought that Coast Guard clients were being provided at least to “a considerable extent” an economically efficient service. The scores for the DFO clients and OGD clients were similar albeit the number of survey respondents that were of the opinion that an economically efficient service was being provided to these clients was slightly lower at just below three-quarters of respondents.

Operational Efficiency

Operational efficiency generally focuses on the relationship between resources and the direct products or services generated from the activities of the program. Fleet Operational Capability achieves operational efficiency using both tools and procedures.

Tools

Almost all staff and client interviews found that new technologies contribute to efficient operations to a moderate extent. They provided examples such as the use of a differential global positioning system to replace the “tide staff”7, and multi-beam sonar and electronic navigation with global positioning systems. As well, about half of staff surveyed agreed that new technologies assisted in achieving operational efficiencies. However, Fleet Operational Capability staff underlined the need to optimize and integrate existing technologies in order to fully benefit from these tools.

The contribution of vessel multitasking to operational efficiency was ambiguous. Staff interviewed (mainly composed of shore-based staff), were, on the whole, very supportive of multitasking as an approach to optimize operations. Clients interviewed understood the need for multitasking; however, certain respondents underlined the need to find a better balance in terms of allocation of operational days when Search and Rescue is part of a multitasked mission. Alternatively, the staff survey, whose sample population also includes sea-going personnel, draws a more dubitative conclusion, noting difficulties with multitasking when vessels were not originally designed to multitask.

The challenge includes ensuring that staff have the training and experience to proficiently undertake all activities related to multitasking. Finally, survey respondents mentioned the lack of stability in pricing for clients incurring from multitasking. Survey respondents noted that the cancelation of a mission by a client sometimes has repercussions in terms of cost on the other clients left in the multitasked mission. The document review also found that for the first 5 years covered by the evaluation, multitasking constituted more than approximately 11% of all operational days (see Table 5). While in the final year of the evaluation multitasking represented approximately 19% of all operational days, a significant increase in the use of this approach.

The document review and staff interviewees noted that the multitasking has been utilized by the Coast Guard almost right from its inception in the 1960s.  More recently the Coast Guard has employed multitasking as a strategy to support its clients as they experience decreased purchasing power due to inflation. Multitasking provides some efficiencies and additional flexibility to re-direct programs to other vessels when the need arises.


7 A tide staff can be used as part of a tide station to measure water levels at fixed intervals of time.

Table 5. Percentage of Delivered for Multitasking

Table 5
Year % Delivered (Operational Days)
2007-2008 12.3%
2008-2009 11.7%
2009-2010 10.7%
2010-2011 8.9%
2011-2012 10.7%
2012-2013 18.8%

Source: FAIS & iFleet

Procedures

Fleet Operational Capability has a suite of SLAs and MOUs which cover, in terms of clients, 98% of its operational days. Almost all staff and client interviews noted that agreements foster good communications between the parties and helped to provide stability in resource allocation planning. Over half of these agreements stipulated a business relationship principle that services must be delivered in a cost effective manner.

Another aspect of Fleet Operational Capability where strong communications facilitated operational efficiency was with the Regional Operations Centres. Regional Operations Centres allow Coast Guard to dispatch and coordinate assets that are at sea. Almost all clients interviewed noted that Regional Operations Centres assisted to “a considerable extent” in supporting the Program and its clients.

Allocative Efficiency

Allocative efficiency generally focuses on the relationship between resources and the difference(s) that result from the direct products or services generated from the activities of the program. Planning was seen by staff as a cornerstone of Fleet Operational Capability efficiency. Clients interviewed also were of the opinion that planning of Fleet Operational Capability was exemplary. Planning activities include operational, financial and administrative aspects of the program.

The Integrated Fleet Operational Plan is a crucial document in the operational planning process. It allows both forecasting the allocation of operational days to clients as well as providing the basis for contingency planning. Almost all clients interviewed noted that the sub-program informs them of any changes in its annual plan “a considerable extent of the time”. However, only about half of client interviewees felt that the Fleet Operational Capability was efficient at handling additional requests or services that happened outside the normal planning process. This might be a consequence of Coast Guard’s limited redundancy in its current fleet.

The document review found evidence of financial planning such as preparation of financial frameworks, budget analysis and investment planning. In 2007 there was a commitment to clarify planning functions and strengthen financial frameworks. This same year, the Coast Guard completed the National Costing Model. After 2008, fiscal restraints gave the Fleet Operational Capability sub-program an additional incentive to refocus allocation of funds. As well in 2009 Fleet Operational Capability released the Fleet Fuel Financial Management Policy and in 2010 it introduced the Integrated Investment Plan.

A limitation which hampered allocative efficiency which was noted by over one-quarter of staff interviewed was the current administrative and staffing processes. However, the document review provides evidence of a reduction in overhead due to standardization of regional organizations. As well, in terms of human resources planning, the document review found that the Coast Guard has produced and implemented a yearly Strategic Human Resources Plan since 2008/09. It has also emphasized the development and use of pools of qualified applicants to help streamline staffing processes.

Use of Best practices and Lessons learned


Key Findings: Evidence suggests that Fleet Operational Capability’s planning practices are commendable.


The document review found that the current planning process is an example of a best practice. Fleet Operational Capability has evolved from a short-term planning approach where investments are driven by assets to a more holistic and integrated planning methodology where investments are driven by program and client requirements. The literature review supported this holistic and integrated approach which takes into account both operational parameters as well as longer term considerations when planning for optimal fleet composition and scheduling.

In order to ensure client program requirements are aligned with proposed budget allocations and to ensure that the plan is affordable and doable, the Integrated Fleet Operational Plan process starts with client consultations in early summer. A draft operational plan is then presented to the Fleet Executive Board and later submitted to the Coast Guard Management Board for final approval.

The challenge facing the Coast Guard is to develop a plan that clients can afford and that the organization can deliver. The document review noted that the Integrated Fleet Operational Plan development process has evolved to being more of a bottom-up than a top-down process, with all regional requirements exceeding the cost estimates calculated in the costing model being addressed through business cases presented during workshops typically held in mid-fall. The aim is to achieve a more collaborative decision-making process that encourages a sharing of information based on experiences and best practices across all regions.

Planned, Adjusted and Delivered Days

Another aspect of fleet operational planning which has been consistently strong is in-year planning. Figure 8 compares planned, adjusted and delivered days. This figure clearly shows that once the planned operational days are adjusted within the fiscal year, they are very close to number of delivered days. In other words the planning process is accurate. This is attested by the performance indicator showing that the program has consistently delivered 95% of the total number of operational days planned.

Figure 8. Comparison of Operational Days

Figure 9 Governance Structure

Source: Integrated Fleet Operational Plan (2008-09, 2010-11, 2012-13), Coast Guard Program Data

Minimizing Use of Resources


Key Findings: Fleet Operational Capability has taken measures to optimize inputs in both of its main budget headings namely: salaries and operations & management.


This section assesses the extent to which resource use has been minimized in the implementation and delivery of the sub-program. The minimum does not refer to an absolute minimum but rather a contextualized and optimized minimum. This contextualized and optimized minimum is determined by analyzing the degree to which input costs were minimized based on the program’s  context, input characteristics (such as quality, quantity, timeliness and appropriateness), extent of impact on outcome achievement and the alternatives available. In the context of the Fleet Operational Capability sub-program, this analysis first looked at the relation between planned and actual expenditures for each unit for the time period of the evaluation.

Fleet Operational Readiness Context:

  • Between 2007-08 and 2010-11 the Fleet Operational Readiness Program was not part of the approved DFO PAA structure; hence the financial information for these fiscal years is unavailable at the sub-program level.
  • Starting in 2011-12, the Fleet Operational Readiness Program became part of the DFO PAA structure. However, due to the complexity of the redesign, the expenditures for the Fleet Maintenance sub-program and the Fleet Procurement sub-program were grouped under Fleet Operational Capability sub-program for the remaining two years of the evaluation, further complicating the analysis. For example, Salary & Operating and Maintenance activities for all 3 sub-programs are grouped under a single sub-program (Fleet Operational Capability) and not broken down by sub-program.  

The implications as it pertains to the current evaluation of the Fleet Operational Capability sub-program is that it creates a limitation of not being able to compare planned with actual expenditures within an individual sub-program.

Going forward, as of fiscal year 2014-15 the appropriate coding is in place for each of the individual sub-programs to directly report on their respective expenditures separately.

As a consequence, the evaluation re-grouped the financial data from the sub-programs and compared what percentage of the overall budget each sub-program represented.

Table 6 notes that in 2012-13 Fleet Operational Capability had $248.6 million in actual spending and a staff of 2,588 full-time equivalents. Within the total Fleet Operational Readiness budget, the Fleet Operational Capability sub-program represents approximately 58% of the total spending. Salaries account for about 70% of its budget with the remaining 30% being allocated to operations and management.

Table 6. Fleet Operational Readiness Approved Budget for 2012-13

Table 6
($ millions)
Sub-Program Actual Spending
(authorities used)
% of Total
Fleet Maintenance 83.9 19%
Fleet Procurement 99.1 23%
Fleet Operational Capability 248.6 58%
Total 431.7 100%

Source: Departmental Performance Report 2012-13

Minimization of Cost Related to Personnel

The document review revealed that Fleet Operational Capability payroll costs are analyzed regularly and when necessary action is taken to control the costs. In addition, prioritizing the allocation of training monies contributes to optimizing the budget dedicated to personnel. Staff and client interviews singled out additional strategies aimed at minimizing the cost related to personnel. For example, reducing crew sizes when possible, doing crew changes at the most economically efficient time of a vessel mission, limiting crewing levels during vessel non-operational periods and creating pools of qualified personnel to streamline staffing processes. In terms of training, the staff survey and the document review provided evidence that training resources were used as an economical approach to address human resources risks such as attrition and the competitiveness of the labour market. For example in 2008, a performance review system for personnel was introduced followed by the roll-out of the continuous learning and development framework in 2009-10.

Minimization of Cost Related to Operations and Management

Fleet Operational Capability has initiated actions to ensure that inputs related to both operations and management are minimized.

Operations

The use of a mixed delivery model was noted in the literature review as a cost-effective approach to provide fleet services. Over the period covered by the evaluation, the Fleet Operational Capability sub-program has emphasised the implementation of vessel multitasking as a manner to optimize their current and future assets. Furthermore, the Coast Guard uses alternative service delivery for several of its clients to ensure cost-effectiveness of operations. These include:

  • contracting to third parties part of the buoy tendering;
  • using the Canadian Coast Guard Auxiliary to support Search and Rescue activities;
  • engaging the private sector as the primary responders in environmental response situations; and
  • chartering vessels and helicopters to complement fleet operational readiness.

Fleet Operational Capability also ensures that impacts from variations in fuel cost are minimized. Keeping fuel budgets under control is crucial as a change of 1-cent results in a difference of $630,000 per year for the Program. The document review revealed that systematic efforts were made to control fuel costs such as developing the Fleet Fuel Financial Management policy in 2009, introducing the controlled financial allotment for fuel expenditures in 2011 and reviewing the current fuel management and purchasing strategies in 2012. Another mitigation measure cited by clients is the adjustment of the steaming speed in order to reduce fuel usage.

Management

The staff survey and the document review noted that budget reviews (i.e. Strategic and Operational Review, Targeted Review, Fleet review, reductions in O&M) have given the program opportunities to refocus the allocation of program resources. Another budgetary practice revealed by the staff survey and the document review which optimizes inputs is the delegation of responsibilities for vessel budgets to both the regional and vessel authorities.

Program inputs are also minimized with the use of strong program coordination and the sharing of resources. An example of coordination is the optimization of vessel allocation in order to reduce the number of patrols. Fleet Operational Capability also shares resources which allow it to optimize its fleet. The sub-program has often allowed for “opportunity missions” on its vessels when they have unused capacity before they sail. It has also shared resources with the Royal Canadian Navy for the Inshore Rescue Boat Program. A third example is the redeployment of heavy icebreakers from the former Maritimes region and the Newfoundland/Labrador region into a single region in order to reduce infrastructure costs.

The staff survey and the document review found that Fleet Operation Capability makes extensive use of guidelines, manuals and agreements to assist in defining appropriate resources and ensuring that they are timely. The development of a costing model in 2007-08, which the program uses for cost-recovery purposes, is noteworthy. Other examples include:

  • updating and reviewing the lifeboat station guidelines;
  • operationalizing the Fleet Safety and Security Manual; and,
  •  using a business principle related to cost-effectiveness in service level agreements with its clients.

5. CONCLUSIONS AND RECOMMENDATIONS


5.1 RELEVANCE

This evaluation concluded that the Fleet Operational Capability sub-program is a fundamental component of the Fleet Operational Readiness Program and plays a central role in responding to the Government of Canada’s on-water and marine related needs.  The sub-program ensures that certificated professionals safely, effectively, and efficiently operate vessels, Air cushions Vehicles, helicopters, and small craft.  There is evidence of a legislated mandate for the Coast Guard to play such a role and have such a fleet. The sub-program aligns directly with federal and departmental roles and priorities, and it contributes to supporting the priorities of other federal organizations (e.g. participating in the coastal and Great Lakes-St. Lawrence Seaway Marine Security Operations Centres, led by Department of National Defence and the Royal Canadian Mounted Police respectively). In addition, the Coast Guard is involved with the RCMP in the Marine Security Enforcement Team program on the Great Lakes-St. Lawrence Seaway system.  Lastly, economic growth in Canada depends heavily on trade and maritime commerce, which relies in turn on Coast Guard programs and the support it provides to its clients in order to have the civilian fleet presence that supports a safe and secure marine sector.

5.2 EFFECTIVENESS

Operational fleet with the required certificated professionals on board

A conclusion to be drawn from the evaluation is that the vessels were operational with the required certificated professionals on board in order to meet most of the Government of Canada’s on-water and marine related programming needs. This is supported by evidence that vessel availability was stable during the timeframe of the evaluation.  In addition, for a majority of the programming, the days delivered were consistently in line with the planned programing days requested by the clients or were within reasonable tolerance levels.

Operationally capable fleet having the capacity to respond

Evidence that Coast Guard is able to deliver on the number of days identified in the Integrated Fleet Operation Plan demonstrates that Coast Guard had an operationally capable fleet having the capacity to meet the needs and requirements for most of its clients.  However, interviewees and survey respondents noted their concern of there being little or no back up and limited flexibility in the event of a vessel breakdown or increased demand.

Access to an operationally ready fleet

The Government of Canada had access to an operationally capable fleet. However, over 70% of the Coast Guard fleet is beyond its half way mark for its operational life which may impact access to an operationally ready fleet in the future should there be any significant increase in demand or unexpected delays in the delivery of new vessels and/or systems. The recent Fleet Maintenance and Fleet Procurement evaluation (March 2014) noted that there was evidence of some progress in acquiring new capital assets. The same evaluation noted that vessel reliability had declined; leaving vessels more susceptible to breakdowns. Conversely, program data reviewed in the current evaluation showed that although vessel reliability had declined, the amount of service time lost due to delays had decreased between 2007-08 and 2012-13. Although the reasons for these delays were sometimes driven by the physical condition of a vessel, the vast majority of delays were due to weather, difficult ice conditions, or the need to wait for a favourable tide. 

Delivering the approved number of operational days

The information reviewed by the evaluation team indicates that overall, the Coast Guard is delivering the approved planned operational days for a majority of its clients. However, there are some exceptions. More specifically, the service delivered versus service planned is below the requested amounts for Aids to Navigation Program and the DFO Ecosystems and Fisheries Management but close to reasonable tolerance targets set by Coast Guard.

Lastly there is one program where sub-program was unable to consistently deliver the number of planned on-water exercise days approved in the Integrated Fleet Operational Plan; namely, the Environmental Response Services Program. Although some improvement was noted in the latter years of the evaluation, the number of days delivered were outside the 90% tolerance level and demonstrates that the Coast Guard continues to face challenges to support the on-water exercise needs of this client.

The challenges are likely a reflection of there being little or no back up and limited flexibility in the event of a vessel breakdown or increased demand from other Coast Guard clients.

Recommendation 1: The Commissioner should ensure that the Fleet Operational Readiness program develops a strategy to meet the on-water exercise needs of the Environmental Response Services Program to fulfill Coast Guard program requirements.

5.3 EFFICIENCY AND ECONOMY

Recent changes to the Coast Guard shore-based organization are intended to achieve greater consistency throughout Coast Guard operations and were also a response to the Government’s Economic Action Plan 2012. Considering that this reorganization took place near the end of this evaluation, the outcomes are still too early to assess.

It is expected that Fleet Operational Capability will continue to realize efficiencies in support of the needs identified by its clients through its use of Service Level Agreements (SLA) and Memorandum of Understanding (MOU). 98% of its operational days are administered by a suite of SLAs and MOUs with staff and clients noting that these agreements foster good communications and has helped to provide stability in the planning process. The evaluation noted that the performance measurement frameworks for these agreements were at various stages of development and may offer opportunities for improvement. However, considering that the agreements with clients renew on different timelines, there will be no formal recommendation in the evaluation report but the Fleet Operational Capability sub-program should consider it as they renew existing or negotiate new agreements. Over half of these agreements stipulated in a business relationship principle that services must be delivered in a cost effective manner.

The Integrated Fleet Operational Plan is an example of a best practice as it addresses the programming needs of its clients. The planning process is based on consultations with clients in order to develop a draft plan to ensure that the individual program requirements are aligned with proposed budget allocations and to ensure that the plan is affordable and deliverable. Recent changes to the consultation process have introduced a bottom-up rather than a top-down approach. Also, all regional requirements that exceeded the cost estimates calculated in the costing model are addressed through business cases presented during the planning process. This resulted in a more collaborative decision-making process that encouraged a sharing of information based on experiences and best practices across all regions. Lastly the plan is adaptable and can be adjusted in order to reflect changes which might result from vessel breakdowns, weather delays and changes in client priorities.

The performance measurement strategy for the Fleet Operational Capability sub-program provided the evaluation team with the performance data necessary to support the evaluation. More specifically, the evaluation utilized available program data to report on all outcomes. The only aspect requiring further development are the actual performance indicators for the long term outcome outlined in the Fleet Operational Readiness Program performance measurement strategy. In the absence of more concrete indicators for the long term outcome, the evaluation team identified “proxy” indicators in consultation with the sub-program.

A comparable conclusion was identified in the Fleet Maintenance and Fleet Procurement Evaluation (March 2014). Consequently, with the completion of both evaluations, the evaluation team is able to conclude that although the Coast Guard is demonstrating progress with regards to the implementation of the performance measurement strategy for the Fleet Operational Readiness Program, continued efforts are necessary in order for it to reach its full potential of being a tool to inform ongoing program management, program decisions and evaluations, and to facilitate an understanding of the program’s overall impact.

Recommendation 2: The Commissioner should ensure that the Fleet Operational Readiness program builds on lessons learned during the early stages in implementing its performance measurement strategy pertaining to all three sub-programs.  As it relates to the Fleet Operational Capability sub-program its focus should be on the performance indicators for its long-term outcome.  In order to ensure interdependencies between all three sub-programs are explicit, Fleet Operational Capability should coordinate its efforts with the Fleet Maintenance and Fleet Procurement sub-programs to update the existing performance strategy of the Fleet Operational Readiness Program.

ANNEX A: MANAGEMENT ACTION PLAN


Management Action Plan
Recommendation

Rationale: Evidence reviewed during the evaluation indicated that the Fleet Operational Capability sub-program was not able to meet the on-water exercise targets outlined in the Integrated Fleet Operational Readiness Plan for some of its clients.

Recommendation 1: The Commissioner should ensure that the Fleet Operational Readiness program develops a strategy to meet the on-water exercise needs of the Environmental Response Services Program to fulfill Coast Guard program requirements.

Strategy
Operational Support has noted that the program delivery shortfall is caused by the non-delivery of planned on-water exercise days. The development of a national exercise program and the compliance monitoring thereof will give early and correctable indicators of performance levels.
Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Reason for Change in Due Date Output
Drill down data analysis to inform/refine way forward (Operational Business) March 2015   Analysis findings summary/report
Develop and implement a national exercising program and monitor compliance with program. December 2015   National exercising program monitoring and implementation strategy
Management Action Plan
Recommendation

Rationale: A similar recommendation was identified in the Fleet Maintenance and Fleet Procurement Evaluation (March 2014). The intent of current recommendation is for the Fleet Operational Capability sub-program to coordinate its efforts with the Fleet Maintenance and Fleet Procurement sub-programs. As it relates to the Fleet Operational Capability sub-program its focus should be on the performance indicators for its long-term outcome.

Recommendation 1: The Commissioner should ensure that the Fleet Operational Readiness Program builds on lessons learned during the early stages in implementing its performance measurement strategy pertaining to all three sub-programs. As it relates to the Fleet Operational Capability sub-program its focus should be on the performance indicators for its long-term outcome. In order to ensure interdependencies between all three sub-programs are explicit, Fleet Operational Capability should coordinate its efforts with the Fleet Maintenance and Fleet Procurement sub-programs to update the existing performance strategy of the Fleet Operational Readiness Program.

Strategy

Fleet Operational Readiness

Fleet Operational Capability (FOC) sub-program

As part of DFO's Performance Measurement Strategy Implementation Plan, Coast Guard developed a Performance Measurement Strategy (PM Strategy) for Fleet Operational Readiness program in 2012-13. In 2015-16, Operations plans to develop a draft PM Strategy for Fleet Operational Capability sub-program in line with the existing PM Strategy for the Fleet Operational Readiness Program. This will include a revised logic model and updated performance measures for the Program, including the FOC sub-program. Operations will review its draft PM Strategy in the context of this evaluation and lessons learned to finalize its PM Strategy for inclusion into the Fleet Operational Readiness PM Strategy.

While a full PM Strategy for the Fleet Operational Capability sub-program is not yet in place, Operations reports continuously on the sub-program by providing percentage of operational days delivered versus planned days delivered reports and on the average daily unit cost per operationally ready seagoing personnel reports.

The implementation of the PM Strategy indicates the start of the formal and ongoing management activity of program performance assessment within Operations. This ongoing management activity will produce annual performance reports to Operations Extended Board to support senior management in making informed decisions on areas for improvement in program services and delivery.

Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Reason for Change in Due Date Output
(Operational Business) Create and submit Fleet Operational Capability Risk Profile in support of developing the PM Strategy March 2015   Fleet Operational Capability sub-program Risk Profile
(Operational Business) Submit the PM Strategy for the Fleet Operational Readiness program to Evaluation for comment April 2015   Draft PM Strategy
(Operational Business) Revise the Fleet Operational Capability based on Evaluation comments June 2015   Revised PM Strategy
(Operational Business) Complete Draft PM Strategy for the Fleet Operational Capacity sub-program in line with the PM Strategy for the Fleet Operational Readiness Program September 2015   Draft Fleet Operational Capability sub-program PM Strategy
(Operational Business) Create PM Strategy Fleet Operational Capability Implementation Plan to reside within the PM Strategy Fleet Operational Readiness implementation plan March 2016   Fleet Operational Capability sub-program PM Strategy Implementation Plan
(Operations) Commence the implementation of the PM Strategy as per the action plan (including data collection, data analysis and use of data for decision making). NOTE: reporting on the status of implementation will be done via the DFO's bi-annual reporting of the PM Strategy action plan to the CFO. December 2016   Bi-annual CFO performance measurement strategy reports