EVALUATION OF FLEET OPERATIONAL READINESS PROGRAM:
FLEET MAINTENANCE AND
FLEET PROCUREMENT
SUB-PROGRAMS

FINAL REPORT MARCH 2014

EVALUATION DIRECTORATE


Table of Contents

ACKNOWLEDGEMENTS AND ACRONYMS


ACKNOWLEDGEMENTS

The Evaluation Directorate acknowledges and thanks all individuals who gave of their time and input for this evaluation of the Canadian Coast Guard's Fleet Maintenance and Fleet Procurement sub-programs. In particular, the Evaluation Directorate acknowledges and thanks all Coast Guard managers and staff who took the time to share their thoughts through interviews and an online survey. The Evaluation Directorate also acknowledges the time and effort given by senior program management in assisting the evaluation team from the planning phase through reporting.

ACRONYMS


List of acronyms
ACV Air cushion vehicles
CCGS Canadian Coast Guard Ship
DFO Fisheries and Oceans Canada
ITS Integrated Technical Services
LCAMS Life Cycle Asset Management
OECD Organisation for Economic Co-operation and Development
O&M Operations and maintenance (budget)
PAA Program Alignment Architecture
TSB Transportation Safety Board

EXECUTIVE SUMMARY


INTRODUCTION

This report presents the results of an evaluation of two Canadian Coast Guard Fleet Operational Readiness sub-programs, namely Fleet Maintenance and Fleet Procurement. The evaluation focused on the extent to which the two sub-programs demonstrate value for money by assessing their relevance and performance, the latter including effectiveness, efficiency and economy, in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation. The third sub-program, Fleet Operational Capability, is being evaluated separately, with reporting scheduled for fall 2014.

The evaluation, conducted by the Department of Fisheries and Oceans’ Evaluation Directorate, covers a five-year period from 2007-08 through 2011-12. The sub-programs operate nationally within the National Capital Region and across the Coast Guard’s three regions: Western, Central and Arctic, and Atlantic. A 2009 evaluation of the Fleet Operational Readiness program contributed to informing this evaluation.

PROGRAM PROFILE

The Coast Guard owns, manages and operates Canada’s civilian fleet through its Fleet Operational Readiness program. This program is part of the department’s Program Alignment Architecture under the strategic outcome of Safe and Secure Waters. Fleet Maintenance provides asset advisory, technical solution, maintenance and disposal services. Fleet Procurement facilitates the development and acquisition of new fleet assets through renewal planning and project management services. These two programs contribute to ensuring the Coast Guard has an operationally ready fleet to support delivery of its own programs as well as other Fisheries and Oceans Canada programs and those of other stakeholders within and external to the federal government. For the period evaluated, the Coast Guard’s fleet included over 115 vessels and 22 helicopters. Planned expenditures for Fleet Maintenance in 2011-12 totaled $47.1 million and for Fleet Procurement they totaled $129.6 million.

EVALUATION METHODOLOGY

For the evaluation, we used a logic model, multiple lines of evidence, qualitative and quantitative data sources and triangulation of the findings from the different lines of evidence to arrive at valid findings and conclusions. The findings are based on data drawn from the following lines of evidence: document and website review, both program and other sources; administrative data; a literature review; interviews; a case study of a refit project; a media scan; a survey; and a comparative analysis of large vessel procurement programs. As the results of the survey and comparative analysis were limited in their scope, they served as secondary supporting lines of evidence that informed a few selected questions.

Although both vessels and helicopters are part of the Coast Guard’s fleet, the evaluation focused on vessel maintenance only, as these activities are directly overseen by the Coast Guard. The Coast Guard contracts with Transport Canada to manage, operate and maintain its helicopters. As well, no helicopter procurement projects were under way during the period covered by the evaluation.

EVALUATION FINDINGS AND RECOMMENDATIONS

Relevance

Fleet Maintenance and Fleet Procurement are integral to the Canadian Coast Guard fulfilling its legislated mandate and its obligations as the owner, manager and operator of Canada’s civilian fleet. Fleet assets are essential to the delivery of a wide range of federal programs and services, such as icebreaking services, waterways management, marine research, search and rescue, environmental response, marine security and more. The two sub-programs align with departmental and federal priorities by contributing to marine safety and security, Canada’s Northern Strategy and the National Shipbuilding Procurement Strategy.

Both sub-programs contribute directly to the availability of an operationally ready fleet. Vessels and helicopters must be maintained in compliance with Transport Canada regulations in order to be operationally available. With the majority of fleet assets in the latter half of their lifecycle, corrective maintenance in the form of major repairs, refurbishments and refits is often required. As with any fleet, aging capital assets must be replaced. Replacing major assets is a long-term endeavour, and the Coast Guard needs to ensure that new fleet assets are procured in accordance with service needs and to maximize efficiency in terms of use (multi-tasking), design and life cycle maintenance and costs.

In addition to the array of programs and services the fleet supports, marine activity in Canada’s Arctic waters continues to increase. This and a current lack of sufficient Arctic marine infrastructure necessitate greater fleet capacity in the North. According to the international Arctic Council, Arctic nations, including Canada, do not have the necessary capacity in place in the form of icebreakers, search and rescue capability, ship salvage and so forth to ensure safe and efficient waterways. Bulk and cruise shipping activity in particular are expected to continue increasing in Canada’s Arctic waters over the ensuing years. The potential for serious accidents increases as more vessels venture into Arctic waters with crews unfamiliar with the territory and its dangerous ice conditions and underwater hazards. Recent incidents, in both Canadian and Russian Arctic waters, underscore the reliance of private vessels on icebreakers when accidents occur. 

Effectiveness

The evaluation found that vessels are being maintained to meet regulatory requirements. Overall, maintenance work tends to be more corrective in nature, with limited ability for preventive maintenance due to financial constraints. The advanced age of many vessels is reflected in the fact that vessel reliability has declined and the remaining operational life of both large and small vessels is less than 30%. Some new assets were procured, with projects underway to replace large vessels in the years ahead. Fleet Procurement’s long-term renewal plan outlines replacement plans well beyond projects currently under way, to ensure other vessels are procured as others reach the end of their useful life. Fleet Procurement is still a very new sub-program, having been introduced in 2009-10, with projects and outcomes typically requiring several years for fruition.

Additional funding has been a significant component in enabling Fleet Maintenance and Fleet Procurement to proceed with necessary work. For the past decade, Fleet Maintenance has received about $47 million annually in addition to its capital budget to help cover maintenance costs. However, this in itself has not been sufficient, as Fleet Maintenance had to rely on funding from the 2009 Economic Action Plan to undertake some major repairs. Fleet Procurement relies on funding approval beyond its base budget for the majority of its projects. Between 2005 and 2012, the federal government approved an additional $6.8 billion mainly for procurement projects, but also including some monies for major repairs to large vessels.

Shipyard capacity, national shipbuilding priorities and a competitive labour market are other key factors that will have a strong bearing on Fleet Procurement project timelines and outcomes in the immediate years ahead and beyond. This in turn has implications for Fleet Maintenance and the need to keep old vessels operational longer than anticipated, some well beyond their initial design life.

Efficiency and Economy

Fleet Operational Readiness’ structure is relatively new, with the three sub-programs part of the department’s Program Alignment Architecture as of 2009-10. In addition, the Coast Guard has undergone significant changes, with its regional structure reduced from five to three as of October 2012. While it is too soon to determine the extent to which these changes will result in greater consistency and efficiency, early indications are that these and other recent initiatives, particularly the Vessel Maintenance Management Manual and the newly introduced Centres of Expertise, are positive efforts in the right direction. However, greater consistency in practice needs to be clearly demonstrated going forward.

Procurement of new vessels and helicopters should result in greater economies and efficiencies. Older vessels require considerably more maintenance, and the majority of fleet vessels, especially the large ones, are advanced in age. Coast Guard’s Fleet renewal plan calls for further economies and efficiencies to be achieved by employing fewer vessel designs, standardized systems and fewer vessels. Life cycle costing and maintenance requirements are also to be taken into account by Fleet Procurement.

Two areas continue to present opportunities for improvement. First, the sub-programs would benefit from a fully integrated asset management and financial information system. This system should also be useful for maintenance planning and scheduling. A departmental 2011 audit noted this lack of integrated information systems for maintenance and materiel management. As the Coast Guard has already committed to integrating its systems in response to the audit, no subsequent recommendation is warranted at this time. Second, the performance measurement strategy needs to be modified to ensure performance data on outcomes is available.

Recommendation 1: The Commissioner should ensure that the Fleet Operational Readiness program builds on lessons learned during the early stages in implementing its performance measurement strategy pertaining to Fleet Maintenance and Fleet Procurement. The strategy, including the logic model, should be reviewed and revised to ensure it incorporates appropriate performance indicators, outputs and outcomes to inform ongoing program management, program decisions and evaluations, and to facilitate an understanding of the program’s overall impact.

1. INTRODUCTION


1.1 PURPOSE OF THE EVALUATION

This report presents the results of the evaluation of Fleet Maintenance and Fleet Procurement, two of three sub-programs of the Canadian Coast Guard’s Fleet Operational Readiness program. The evaluation is in accordance with the Treasury Board of Canada’s 2009 Policy on Evaluation, which requires all direct program spending to be evaluated every five years. The evaluation commenced in 2012, in accordance with the multi-year Departmental Evaluation Plan, and was completed in January 2014. Recommendations stemming from the main findings were developed to support improvements to the program and to inform future decision-making.

1.2 PARAMETERS

The main objective of this evaluation was to determine the extent to which the two sub-programs are managed effectively and efficiently and have achieved the intended outcomes. The evaluation covers the five-year period 2007-08 through 2011-12 and was undertaken between November 2012 and January 2014 by the Department of Fisheries and Oceans’ Evaluation Directorate. The evaluation is inclusive of the National Capital Region and the Coast Guard’s three regions: Western, Central and Arctic, and Atlantic. A previous evaluation of the Fleet, completed in 2009, covered the period 2002-03 through 2006-07. This evaluation builds on its findings and recommendations.

Another sub-program, Fleet Operational Capability, is being evaluated separately and will be reported on in 2014-15. The extent to which all three sub-programs contributed to meeting operational, program and Government of Canada requirements can best be assessed as part of the Fleet Operational Capability evaluation.

1.3 REPORT STRUCTURE

The executive summary provides a brief summary of the evaluation findings, conclusions and recommendations. Section 2 presents an overview of the program. Section 3 describes the evaluation methodology, followed by a discussion of the main findings in section 4, and conclusions and recommendations in section 5. Annex A presents the key issues, questions, indicators and data sources used for the evaluation.

2. PROGRAM PROFILE


2.1 PROGRAM MANDATE

The Fleet Operational Readiness program, which is administered by the Canadian Coast Guard, is aligned with Fisheries and Ocean Canada’s Safe and Secure Waters strategic outcome, as reflected in the department’s Program Alignment Architecture (PAA). The program contributes to maritime safety and security through the provision of maritime infrastructure which in turn contributes to safe navigation and the protection of life and property. This includes, for example, providing staffed vessels, air cushion vehicles (ACVs), helicopters and small craft ready to respond to on-water and maritime-related requirements. This program is comprised of three components, the latter two of which are the focus of this evaluation report: Fleet Operational Capability, Fleet Maintenance and Fleet Procurement. A separate evaluation report, to be issued later in 2014, will address Fleet Operational Capability.

The Fleet Operational Readiness program supports Coast Guard programs, science, fisheries and aquaculture management activities of the Department of Fisheries and Oceans (DFO), and the activities of a number of other government departments needing on-water delivery in support of their mandates. Such clients include the Department of National Defence, Environment Canada, the Royal Canadian Mounted Police, Canada Border Services Agency and Transport Canada.

2.2 SUB-PROGRAM ACTIVITIES

Fleet Maintenance

This sub-program, which is delivered by the Coast Guard’s Integrated Technical Services Directorate, involves the delivery of asset management services for Coast Guard vessels, ACVs, helicopters and small craft.1 Following a life cycle asset management approach, Fleet Maintenance ensures the availability and reliability of fleet assets through the provision of life-cycle investment planning, engineering, maintenance and disposal services. Services are provided through one of four categories:

  • Advisory Services ‒ Expert technical and life-cycle asset management counsel to ensure cost-efficient maintenance of Coast Guard capital assets.
  • Clients’ Project Services ‒ Analysis, design and development of technical solutions to meet new or modified operational requirements, and application of technical solutions from construction or installation through commissioning of assets.
  • Maintenance Services Preventive, corrective and predictive maintenance to support new and existing vessels in acceptable condition and at acceptable operating levels.
  • Disposal Services‒ Disposition of surplus assets including integrated logistic support elements, environmental considerations and reclamation of useable components, such as systems and equipment, as well as recycling of waste. 

Fleet Procurement

Fleet Procurement plans, manages, verifies and accepts the design and construction of new large and small vessels, ACVs, helicopters and small craft consistent with Coast Guard operational requirements, as identified in the Fleet Renewal Plan and the Integrated Investment Plan.  The sub-program provides project management and internal controls (e.g., challenge function to project scope, expenditure management) and ensures that new vessels are delivered according to an approved schedule.

Fleet Procurement is delivered by the Coast Guard’s Vessel Procurement Sector with support from Operations (previously Fleet Directorate) and Integrated Technical Services, and in collaboration with Public Works and Government Services Canada and Transport Canada, according to established legislation and Treasury Board policies and directives.

The sub-program also includes fleet renewal implementation. These responsibilities encompass engineering and project management activities associated with the procurement, design, development, construction, installation, verification and acceptance of fleet assets, while ensuring that the delivery of new assets is in accordance with a project’s scope, cost and schedule. There are two primary functions:

  • Fleet Renewal Planning ‒ Developing strategic documents such as the Fleet Renewal Plan, acquisition options, costs, benefits and risk assessments, and consultation and coordination with clients, partners and other stakeholders.
  • Fleet Renewal Implementation ‒ Provision of project management support services using a defined Project Management Framework to ensure projects remain within scope, cost and schedule.

1 The Canadian Coast Guard owns a fleet of helicopters, but contracts with Transport Canada by way of a Memorandum of Understanding. The Coast Guard is financially responsible to Transport Canada for pilot salaries, equipment repairs, and new parts, whereby Transport Canada is responsible for managing, operating, and maintaining the fleet of helicopters.

2.3 EXPECTED RESULTS

Fleet Maintenance

The expected result is a reliable fleet that responds to the operational needs and requirements of the Government of Canada. During the period covered by this evaluation, this was measured using a condition rating2 for vessels, with a target of 64.4 for large vessels and a target of 65.8 for small vessels in terms of acceptable risk tolerance for reliability, availability and maintainability. The condition level should be maintained or improve over time.

Fleet Procurement

The expected result is a modern fleet that responds to the operational needs and requirements of the Government of Canada. During the period covered by this evaluation, this was measured by assessing the remaining operational life of large and small vessels and helicopters against an overall target of 50%. The measure was subsequently changed in 2012-13.


2 The Coast Guard determines vessel condition through a physical examination of each vessel, resulting in a vessel condition score or rating. The target rating represents the acceptable risk tolerances against which the aggregate of all results are measured.

2.4 PROGRAM LOGIC MODEL AND PERFORMANCE MEASUREMENT

Fleet Operational Readiness has a performance measurement strategy in place. The strategy details the program and the three sub-programs (Fleet Operational Capability, Fleet Maintenance and Fleet Procurement) and covers the resources, physical structure and reporting relationships of the program. The program’s logic model, shown in figure 1, informed this evaluation. The bottom two rows represent Fleet Maintenance and Fleet Procurement respectively.

The strategy includes a matrix presenting the measures, targets, indicators and data collection process for assessing the effectiveness of each sub-program. It also includes a matrix for assessing the efficiency of Fleet Maintenance and Fleet Procurement. The Evaluation Directorate’s December 2012 assessment of the strategy noted that further detail is required to better understand the program vision and objectives, how the program overall and sub-programs work, and risk mitigation strategies. The strategy would also benefit from a logic model narrative explaining the underlying theory of change, the relationship between the boxes and flow of arrows.

Figure 1. Logic Model for Fleet Operational Readiness Program

Logic Model for Fleet Operational Readiness Program

Although the strategy was formally set in place after the period being evaluated, it was decided in consultation with the Coast Guard that the Fleet Operational Readiness matrix would be used for the two evaluations as no other framework had been used previously. In essence, this evaluation afforded the opportunity to test the strategy to determine its usefulness going forward and assess whether any changes might be required.

2.5 PROGRAM STAKEHOLDERS

Fleet Maintenance and Fleet Procurement contribute to ensuring the Coast Guard has operational vessels, ACVs, helicopters and small craft available to deliver a range of programs, including its own and other federal programs.

The two sub-programs’ main clients are Coast Guard staff involved in operations, plus DFO science and fisheries staff. More specifically, Coast Guard vessels, ACVs and helicopters are essential platforms for the delivery of the following Coast Guard programs: Marine Navigation, Search and Rescue Services, Maritime Security, Environmental Response Services, and Marine Communications and Traffic Services. DFO clients include, for example, programs within the Ecosystems and Fisheries Management sector, specifically Fisheries Management, and Compliance and Enforcement, and within the Ecosystems and Oceans Science sector. 

External stakeholders that rely on Coast Guard vessels and helicopters to support their programs include other federal departments and agencies, specifically Environment Canada, Natural Resources Canada, the Natural Sciences and Engineering Research Council of Canada, the Royal Canadian Mounted Police and the Department of National Defence. As well, Public Works and Government Services Canada is a key stakeholder in the procurement of services and capital assets. In particular, it has a key role in the National Shipbuilding Procurement Strategy announced in June 2010, which involves Fleet Procurement. The Strategy’s Secretariat announced the selection of two shipyards in October 2011.

2.6 GOVERNANCE

The Deputy Commissioner, Vessel Procurement and the Directors General, Integrated Technical Services and Major Projects are responsible for the governance and delivery of Fleet Procurement and Fleet Maintenance. These positions report directly to the Coast Guard Commissioner and are part of the Coast Guard Management Board, the Coast Guard’s senior decision-making body. All of these positions and others, such as the Assistant Commissioners and Directors General, Operations and Integrated Business Management Services, contribute in one way or another to ongoing maintenance and renewal of fleet assets. The senior human resources advisor, the senior legal advisor, the senior communications advisor and the Executive Advisor to the Commissioner are ex officio members of the Management Board.

In May 2012, the Coast Guard announced the restructuring of its shore-based organization, to take place during 2012-13, with the aim of achieving greater efficiencies and national consistency. The Fleet Directorate merged its national operational oversight functions with similar functions carried out by Maritime Services to form a new Operations Directorate. Three other directorates are part of the organizational structure: Integrated Technical Services, National Strategies and Integrated Business Management Services. The composition of the Management Board remains much as it was before, with the four directorates represented by their Directors General.

2.7 PROGRAM RESOURCES

Fleet Maintenance and Fleet Procurement operate annually with operating (vote 1) and capital (vote 5) funds. Prior to 2012-13, the expenditures for Fleet Maintenance and Fleet Procurement were included with Fleet Operational Capability. Therefore, we are unable to report on expenditures for the two sub-programs alone. The following table presents the data available for 2011-12 for the Fleet Operational Readiness program.

Table 1. Fleet Operational Readiness Resources

Fleet Operational Readiness Resources
2011-12 Resources
Financial Resources
($ millions)
Planned Total Authorities1 Actual
Fleet Maintenance 47.1 47.1 416.92
Fleet Procurement 129.6 137.1
Fleet Operational Capability 248.3 282.2
Total 425.0 466.4 416.9
Human Resources (FTEs) Planned Actual Difference
Fleet Operational Readiness 2,817 2,809 -8

1. Excludes amount deemed appropriated to Shared Services Canada
2.  Expenditures for all three sub-programs

Source: Departmental Performance Report 2011-12

2.8 RISK PROFILE

The following risks are aligned with the Coast Guard’s 2009 Corporate Risk Profile and reflect risks relevant to Fleet Operational Readiness as a whole.

  • Investment in Asset Base – There is a risk that the Coast Guard will be unable to procure and maintain the needed asset base, in a timely fashion, in order to deliver mandated services. The age of the existing assets is an inherent source of risk which needs to be addressed by increased maintenance and timely replacement.
  • Human Resources – There is a risk that the Coast Guard will be unable to sustain a sufficient and representative workforce with the appropriate competencies in order to adequately support, deliver and manage services.
  • Service Delivery – There is a risk that the Coast Guard will not meet the expectations and evolving needs of Canadians and its stakeholders with available resources. The ability to provide operational platforms is dependent on having the appropriate human resources, strengthened management and adequate fleet and land based assets.

3. METHODOLOGY


3.1 SCOPE

The evaluation of the two sub-programs, for the five-year period 2007-08 through 2011-12, was undertaken by the Department’s Evaluation Directorate between   November 2012 and January 2014. The evaluation covered National Headquarters and the Coast Guard’s three regions: Western, Central and Arctic, and Atlantic. Senior program management assisted throughout the evaluation. In addition to participating in interviews, they provided documents and statistical data for the evaluation team to review, identified other interviewees and stakeholders to contact, and provided feedback to the evaluation team on such items as the evaluation plan, survey questions and the evaluation report.

3.2 EVALUATION APPROACH AND DESIGN

The evaluation used a logic model combined with multiple lines of evidence drawing on both qualitative (e.g., interviews, program documents) and quantitative data (e.g., administrative data, survey). The various sources of input provided the basis for corroborating the evidence (triangulation) to arrive at valid findings and conclusions. The logic model used for this evaluation is from the April 2012 performance measurement strategy developed for Fleet Operational Readiness.

Neither an experimental nor quasi-experimental design would be appropriate, as they are contra-indicated for this program. First, the program is not one such that fleet assets can be randomly assigned into two groups, one which benefits from program initiatives and one which does not. Second, experimental-type designs are very time-consuming and resource intensive beyond what is appropriate and necessary to produce a timely evaluation of the program at reasonable cost. Rigour in a non-experimental design is achieved by virtue of using the combination of elements described above, i.e., the logic model and analysis based on multiple lines of evidence and both qualitative and quantitative data.

3.3 KEY ISSUES AND EVALUATION QUESTIONS

The evaluation matrix in Annex A presents the key issues investigated (relevance, effectiveness, efficiency and economy), the specific evaluation questions addressed and the corresponding lines of evidence used for each question. The evaluation team developed the questions based on Treasury Board’s Policy on Evaluation, a review of program documents and evaluation planning discussions with key program personnel.

3.4 DATA COLLECTION

The evaluation drew on the following lines of evidence to gather qualitative and quantitative data:

  • A document and website review examined various materials from the two-subprograms, the Canadian Coast Guard as a whole, Fisheries and Oceans Canada, including the previous evaluation and audits, and other federal government and external sources.
  • Administrative data was sourced from Coast Guard and other departmental publications, as well as spreadsheet information provided by the programs.
  • A literature review of fleet operations and management, based on 30 sources of information, contributed to addressing relevance and economy questions.
  • A comparative analysis of large vessel procurement programs was used to identify some best practices regarding the efficiency and economy of procurement processes, as well as assess similarities and differences in program design and delivery, and highlight lessons learned. The analysis compared Fleet Procurement with four other models: National Defence’s Joint Support Ship Project; the United States Coast Guard; the Norwegian Coast Guard; and the Australian Customs and Border Protection Service.
  • Interviews were held with 23 interviewee respondents: 9 for Fleet Maintenance and 14 for Fleet Procurement. These included interviews with a Deputy Commissioner, Directors General, Directors, and chiefs or senior managers in each region including the National Capital Region. The majority of headquarter interviews were conducted in person, with regional personnel interviewed by phone.  
  • A media scan contributed to addressing relevance and immediate outcomes.
  • An online survey of program staff (excluding junior and administrative staff) provided limited feedback. The survey had a response rate of 32%, with 19 respondents out of 59 invited.
  • A case study was undertaken focusing on the refit activities of a mid-sized vessel that was deemed appropriate considering its age and size.  The case study addressed evaluation questions having to do with efficiency and economy and followed the vessel over its five-year refit regulatory cycle between 2007-08 and 2011-12. While no single vessel can be representative of the range of Coast Guard vessels, the one chosen for the case study is considered fairly typical for the purposes of examining refit activities.

3.5 ANALYTICAL METHODS

The data from each evaluation method detailed above was summarized in individual line of evidence summaries and reports to address each of the evaluation issues and questions contained in the evaluation matrix. The results from each line of inquiry were then integrated for each question and analyzed together to determine the overall findings and arrive at valid conclusions. In other words, findings were determined by triangulating the multiple lines of evidence. Conclusions were then drawn from the findings.

3.6 LIMITATIONS AND MITIGATION STRATEGIES

Although the evaluation encountered some challenges and limitations that are outlined below, these limitations were mitigated, as much as possible, through the use of multiple lines of evidence and triangulation of data. This approach was taken in order to demonstrate reliability and validity of the findings and to ensure that conclusions and recommendations are based on objective and documented evidence to the fullest extent possible.

Survey of Program Staff

Even though the response rate was approximately 32%, the total number of survey respondents was low. A total of 19 surveys were completed out of the possible 59 staff members invited to participate in the survey.

The evaluation team attempted to increase the rate of responses by leaving the survey open for 12 weeks to ensure that program staff were given sufficient time to respond. In addition to the initial email announcing the survey launch, three reminder emails were sent at approximately one month intervals. The timing of the reminder emails was chosen to best reach program staff who had previously been out of the office. Although the reminder emails from both the Evaluation Directorate and Coast Guard management generated a few more responses, the overall response rate remained low.

Consequently, the survey responses were approached in a relatively conservative manner and were not subjected to overly complex analysis. Moreover, the evaluation team opted not to rely on the survey results as a lead line of evidence noting that nothing in the results was contrary to findings from other lines of evidence. As such, the survey results served as supporting evidence as opposed to a direct line of evidence in the evaluation.

Comparative Analysis

The comparative analysis was limited by the availability of relevant public literature (which was not often comparable between countries or programs). In addition, the inability of key contacts to participate in this study due to confidentiality or workload demand as well as the inability to actually reach or otherwise engage key contacts in order to secure their participation were other key limitations. Lastly, the value of the comparative analysis methodology is the applicability of the comparisons. Consequently, in order to mitigate the limitations of the comparative analysis, we restricted its analysis to the information on lessons learned or best practices that could be transferred to the Canadian context.

Key Informant Interviews

One possible limitation for the evaluation is that interviewees may overly emphasize the positive aspects of their program, thus limiting the credibility of information provided. We mitigated the potential for self-interest bias by structuring interview questions so that interviewees were not always commenting on areas where they have a vested self-interest and by compelling them to make trade-offs or rank alternatives. In addition, in order to mitigate the potential for bias, comments and findings were balanced with data from the other lines of evidence obtained during the evaluation.

4. MAJOR FINDINGS


This section combines information from all the lines of evidence used and presents the findings according to the broad evaluation issues of relevance and performance which covers effectiveness, efficiency and economy.

4.1 RELEVANCE

Overall Findings

The evaluation examined whether the Fleet Maintenance and the Fleet Procurement sub-programs continue to be relevant. Specifically, it looked at whether they address an essential need, whether they align with the federal government’s role and priorities, and whether they align with departmental priorities. The evidence indicates that on all three accounts both sub-programs continue to be very relevant and essential.

CONTINUING NEED FOR THE SUB-PROGRAMS


Key Findings: There is an unequivocal need for Fleet Maintenance and Fleet Procurement. The Coast Guard’s civilian fleet obligations, its aging vessels, federal budget commitments for procuring new vessels and increasing marine activity in Canada’s Arctic demonstrates the need for both sub-programs.


Canada’s Civilian Fleet Obligations

The Canadian Coast Guard owns, manages and operates a variety of marine assets in order to fulfill its mandate and obligations as the sole operator of the Government of Canada’s civilian fleet. This fleet includes over 35 large vessels, about 80 small vessels, various small craft and 22 helicopters. These assets are used to deliver a wide range of federal programs and services, for example:

  • Aids to Navigation ‒ Coast Guard vessels are used to deploy and retrieve buoys and maintain other aids.
  • Icebreaking ‒ Canada has more floating ice on its oceans and lakes than any other nation. Coast Guard handles about 1,500 requests a year for icebreaking support.
  • Waterways Management Services ‒ Commercial shipping and other vessel operators rely on Coast Guard services, such as keeping channels dredged, to ensure safe and efficient use of Canada’s commercial waterways.
  • Maritime Security ‒ Coast Guard provides and operates four vessels used by the Royal Canadian Mounted Police (RCMP) for security enforcement on the Great Lakes–St. Lawrence Seaway. Other on-water safety and security support is provided as needed, for example, during the 2010 Olympic and Paralympic Games.
  • Search and Rescue (SAR) ‒ Coast Guard works with the Department of National Defence in delivering the federal government’s SAR program. CCG is responsible for the maritime component and co-ordinates about 75% of all SAR responses, responding to more than 6,000 maritime incidents annually. Fleet hovercrafts and lifeboats are used to provide logistical support, conduct searches, transport people to safety and tow disabled vessels.
  • Environmental Response ‒ If a polluter in Canadian waters is unknown, unwilling or unable to respond to an incident, Coast Guard becomes the On-Scene Commander managing the clean-up.
  • Compliance and Enforcement ‒ Mid-shore patrol vessels are used by DFO’s Compliance and Enforcement program to assist in managing fisheries.
  • Community re-supply ‒ Coast Guard vessels are used to deliver supplies to communities in Canada’s Arctic.
  • Ocean science research ‒ DFO scientists and others rely on Coast Guard vessels to conduct marine research.
  • Hydrographic surveys ‒ Coast Guard vessels are an essential part of undertaking hydrographic surveys in Canadian waters.

The Coast Guard has formalized its obligations to use its vessels in support of various programs in service level agreements, memoranda of understanding and other agreements with clients and partners. These include, for example, the Coast Guard’s Operations Directorate; DFO’s Ecosystems and Oceans Science sector, and Ecosystems and Fisheries Management sector; other federal departments and agencies, such as Environment Canada and the RCMP; Nunavut; and international partners such as the United States Coast Guard.

Fleet Maintenance and Fleet Procurement, along with Fleet Operational Capability, are integral to ensuring marine assets are available to support Coast Guard’s obligations and delivery of programs. Coast Guard assets must be able to operate for extended periods of time safely, consistently and effectively and often in harsh and hazardous conditions.

Aging Vessels

An older fleet has implications in terms of both maintenance and procurement requirements. Almost all large vessels and just over half of small vessels were beyond the halfway mark by 2007, as shown in the following table. By 2012, all large vessels and 70% of small vessels were beyond the midpoint.

Table 2. Age of Fleet Vessels

Age of Fleet Vessels
  2007
(%)
2012
(%)
Large Vessels (Design life 25 to 45 years: average 35)
Equal to or beyond average mid-life (≥18 years) 95 100
Equal to or greater than 25 years 55 78
More than 35 years 25 25
Small Vessels (Design life 15 to 20 years: average 18)
Equal to or beyond mid-life (≥ 9 years) 55 70
More than 20 years 31 28

In the intervening years since the vessels were first built, platform requirements have changed. For example, each of its large vessels must be multi-functional, and new polar icebreakers must have increased capabilities to operate further north and for longer periods of time than currently possible, while providing greater icebreaking capabilities as well.

Maintaining Fleet Assets

As with any capital asset, maintenance is necessary to ensure the usefulness of an asset over the long term and to achieve a good return on investment. Transport Canada’s Marine Safety requirements set the regulatory parameters for vessel maintenance.  Approximately 70% of all Coast Guard vessel maintenance is to ensure regulatory compliance. Vessels that do not comply cannot be available for operations.

The most frequent types of asset maintenance are preventive and corrective maintenance. Preventive maintenance is any planned action required to prevent or reduce the likelihood of system or equipment failure. Corrective maintenance involves unplanned work required to correct unexpected system or equipment breakdowns. As vessels age, more breakdowns occur and maintenance costs increase. According to the document review, considerable investments have been made since 2003 in maintaining fleet assets. These investments involve major repairs, refurbishments and refits.

The ongoing need for major repairs, refit and vessel life extensions is reflected in the funding provided for fleet maintenance work over the past decade. Since 2003, the Coast Guard has received an additional $47 million annually to use in refurbishing its capital assets, it converted about $23 million from its operating budget for refits, and it received $175 million through the 2009 Economic Action Plan, part of which was to cover five vessel life extensions and major repairs on large vessels.

According to the document review, breakdowns continue to occur with aging vessels despite these investments. A 2010 Life Cycle Assets Management evaluation found that unplanned maintenance expenses had increased while the budget for preventive maintenance expenditures declined. The evaluation described the gap between the two as being significant.

Federal Commitments to Procure New Vessels

Eventually, old capital assets must be replaced in order to support continuing operations. Renewal of the Coast Guard’s fleet of vessels and helicopters is a long-term endeavour, requiring years to address immediate procurement needs and then to ensure ongoing replacement in future years. To that end, the Coast Guard developed a 25-year plan for 2006 to 2030. The plan was subsequently revised to 30 years for the period 2011 to 2041. The plan lists each vessel to be replaced and the planned target year for replacement. Vessels will not be replaced on a one-to-one basis, and overall there will be a reduction in the number of vessels and vessel types.

The federal government announced in its February 2006 through 2008 budgets a total of $1.4 billion for the Coast Guard to invest in fleet recapitalization over the next nine years. These monies are being used to procure 14 large vessels, specifically nine new mid-shore patrol vessels, four offshore science vessels and a polar icebreaker. The icebreaker is the largest vessel initiative and was announced in the 2008 federal budget. The 2009 Economic Action Plan included monies to procure 98 small vessels and crafts, and the 2010 federal budget provided $27.3 million over five years for a new hovercraft. In addition, the Coast Guard has an annual capital budget of $129 million to acquire small vessels and craft. Most recently, $5.2 billion was announced in the 2012 budget to acquire other large and small vessels and to renew the Coast Guard’s helicopter fleet. As this last allocation is outside the scope of this evaluation, these projects are not addressed further in this report.

Increasing Marine Activity in Canada’s Arctic

Throughout much of Canada’s North, shipping and other marine traffic are to a large extent the only transportation option available. There are virtually no roads, no rail lines other than at Churchill, and air service is expensive.

The Arctic Council’s Arctic Marine Shipping Assessment 2009 Report cites icebreakers, search and rescue capability, ship repair and salvage as some of the infrastructure components needed for a safe and efficient waterway. Arctic nations, both individually and collectively, are legally responsible for providing the necessary infrastructure in order to prevent loss of life and property and environmental damage. Furthermore, marine insurance companies consider polar icebreakers and marine salvage capability to be risk mitigators. Their availability substantially reduces risk to vessels and correspondingly the financial risk to owners and insurers. In essence, these risk mitigators are required for economic growth and development in the North. According to the report, Canada and other nations lack the basic infrastructure needed to support increased marine activity.

While it is not expected that the Northwest Passage will become a shipping route of choice anytime soon due to navigational difficulties and high operational costs, bulk shipping related to mining, community resupply and cruise shipping on Canadian Arctic waters are expected to continue increasing in the immediate future to at least 2020.

With increasing marine traffic in the North, there is an increasing need for fleet infrastructure and services. The Coast Guard currently owns and operates seven icebreakers in the Arctic. One of these is dedicated to Arctic science research. The others manage aids to navigation and communication towers, escort commercial ships, open up harbours, conduct SAR missions, respond to environmental concerns, and support research and Canadian sovereignty efforts.

As Arctic activity increases, the role of the Coast Guard is evolving. The push for greater sovereignty presence and security in the Arctic has implications for the provision of icebreakers and their capabilities. As well, Canada extended its jurisdiction in Arctic waters up to 200 nautical miles in August 2009 with an amendment to its Arctic Waters Pollution Prevention Act. This effectively doubled the previous area of jurisdiction within which Canada enforces its environmental laws and shipping regulations.

Increasing use of Canadian Arctic waters increases the risk of accidents. Although it is widely known that sea ice has been melting more rapidly than anticipated, Arctic waters are not expected to become entirely free of ice. In fact, less ice poses more hazards in terms of navigation. The ice that remains is old ice pushed south from the polar ice cap. It is harder than concrete, and small icebergs low in the water can be difficult to spot. If a ship hits ice at too great a speed, serious damage can result. For example, a Russian tanker loaded with diesel fuel became disabled when it collided with an ice floe on September 4, 2013 along the Northern Sea Route, off the coast of Russia. Over a week later it was still adrift, awaiting assistance from icebreakers to clear a path back to port.

Less ice can also tempt less capable ships and less experienced crews, increasing the potential for accidents. These accidents are not limited to encounters with ice. For example, three vessels in 2010, two tankers and the cruise ship Clipper Adventurer, all ran aground within the space of a month.

In August 2010, the Clipper Adventurer rammed into an underwater cliff near Kugluktuk at full speed. According to a Transportation Safety Board (TSB) report released in April 2012, the Clipper Adventurer not only lacked essential equipment, but its crew did not fully comply with established company procedures, nor did they make themselves fully aware of known hazards in the area. Further, the crew made subsequent decisions that could have had disastrous results, putting its passengers (average age reportedly 65), its crew and the environment at risk. The CCGS Amundsen arrived two days after the incident, having travelled more than 500 kilometers to reach the cruise ship.

The TSB report notes that during a seven-year period in the 1980s one ship made four Arctic cruises. In the seven-year period preceding the 2012 report, seven ships made 105 Arctic cruises, and seven cruise ships were still operating in Canadian Arctic waters in 2011. The report states that passenger vessels are a high risk as “an emergency in the Arctic could leave passengers and crew stranded for an extended period of time in a harsh environment.”3

While Canadian oil and gas resources are expected to be moved out by pipeline and not tankers, Canada’s Arctic waters can still be subject to oil spills from operations in adjacent waters, as well as from tankers in Canadian waters delivering fuel to Arctic communities. Within a week following the Clipper Adventurer incident, a fuel tanker ran aground near Gjoa Haven, while another one had run aground near Pangnirtung about a month earlier. Neither incident resulted in oil leaking into the environment. Still, as of September 2012, Canada was no more prepared than other countries to handle an Arctic water oil spill according to the head of the Arctic Council’s Arctic Monitoring and Assessment Program.4


3 Marine Investigative Report M10H006: Grounding, Passenger vessel Clipper Adventurer, p.33.

4 “Russia, world’s worst oil polluter, now drilling in Arctic”, CBC News, September 24, 2012.

ALIGNMENT WITH FEDERAL ROLE AND PRIORITIES


Key Findings: The Coast Guard derives its mandate for a civilian fleet from federal legislation. Fleet Maintenance and Fleet Procurement align with the Government of Canada’s outcome area of a safe and secure Canada, Canada’s Northern Strategy and the interests and priorities of other federal departments.


Governing Legislation

Both the legislation and input from program staff indicate that the Canadian Coast Guard is the appropriate owner, manager and operator of Canada’s civilian fleet. Federal legislation gives the Minister of Fisheries and Oceans Canada and the Coast Guard responsibilities relevant to owning and maintaining a civilian fleet. The Constitution Act, 1867, the Oceans Act and the Arctic Waters Pollution Prevention Act have a bearing on the Coast Guard with respect to owning and operating a civilian fleet.

The Constitution Act, 1867, s. 91 (9, 10) confers on the federal government exclusive jurisdiction to govern all matters related to navigation, navigational aids (such as beacons, buoys and lighthouses) and shipping on the country’s oceans and inland waterways. The Oceans Act (SC 1996, c. 31, s. 41-43) gives the Coast Guard its specific mandate. More specifically, the Act gives the Minister of Fisheries and Oceans responsibility for ensuring safe, economical and efficient movement of ships in Canadian waterways through the provision of such services as aids to navigation, icebreaking, channel management, marine search and rescue, and environmental response. The Minister is responsible for supporting other federal organizations by providing ships, aircraft and other marine services. The Minister is also responsible for marine science and may maintain and operate ships expressly for ocean science research. The Arctic Waters Pollution Prevention Act (RSC, 1985, c. A-12) has a bearing on the extent of Canada’s jurisdiction in Arctic waters and hence the area to be covered by the Coast Guard with respect to its Environmental Response capabilities. 

Both interviewees and survey respondents consider that it is appropriate for the federal government to own and manage a civilian fleet. Such a fleet contributes to maritime safety and security while providing a platform in support of other government programs such as marine science research. The programs that are supported by the Coast Guard’s fleet are not within the Department of National Defence’s mandate, nor is it the private sector’s role to provide such services as search and rescue or sovereign presence.

Alignment with Federal Priorities

Fleet Maintenance and Fleet Procurement contribute to federal government priorities. Both ultimately align with the Government of Canada’s outcome area for a safe and secure Canada, one of 16 outcomes identified in the federal government’s whole-of-government framework. They also align with two of Canada’s Northern Strategy priorities, presented in the federal government’s 2007 Speech from the Throne: Exercising Our Arctic Sovereignty, and Promoting Social and Economic Development. They contribute by ensuring icebreakers are available in support of Canada’s sovereignty presence and through such missions as escorting other vessels, transporting dry cargo and fuel to isolated northern communities that are not connected by road or rail, conducting science research, and supporting search and rescue missions.

Coast Guard vessels support the interests and priorities of other federal departments and agencies such as Environment Canada, Natural Resources Canada, National Research Council Canada, Aboriginal and Northern Affairs Canada and Health Canada. Coast Guard vessels are used to assist the Department of National Defence and the RCMP in sovereignty and security exercises and operations. For example, the Coast Guard participated with other federal departments in a Canadian Forces-led northern Arctic sovereignty and emergency preparedness exercise. During the 2010 Olympic and Paralympic Games, Coast Guard vessels were deployed to assist with marine security. Finally, the Coast Guard’s participation in the National Shipbuilding Procurement Strategy supports the Government of Canada’s priority in stimulating the economy by creating over 15,000 jobs across the country in the Canadian shipbuilding industry.

ALIGNMENT WITH DEPARTMENTAL PRIORITIES


Key Findings: Both sub-programs align with DFO’s strategic outcome of Safe and Secure Waters. They also contribute to the department’s two other strategic outcomes.


The document review and interviews confirm that Fleet Maintenance and Fleet Procurement are aligned with DFO’s strategic outcome of Safe and Secure Waters. CCG’s Fleet Operational Readiness program, through Fleet Maintenance and Fleet Procurement, is responsible for providing safe, reliable, available and operationally capable vessels, air cushion vehicles, helicopters and small craft in support of the department’s mandate and other federal government on-water requirements.  

Fleet Maintenance’s role is to ensure the availability and reliability of Coast Guard vessels by providing expert technical and life cycle asset management advice, assisting with vessel modifications in support of client projects, providing preventive, predictive and corrective maintenance, and disposing of assets no longer required. Fleet Procurement’s role is to plan for and manage over the long term the acquisition of new assets designed to meet operational requirements.

In addition to the two sub-programs aligning with DFO’s strategic outcome of Safe and Secure Waters, Coast Guard assets are integral to the delivery of DFO programs that align with the department’s two other strategic outcomes, namely Economically Prosperous Maritime Sectors and Fisheries, and Sustainable Aquatic Ecosystems. For example, Coast Guard vessels are used to deploy and retrieve buoys in support of marine navigation and by DFO’s Compliance and Enforcement program in support of fisheries. As interviewees mentioned, DFO could not fulfil its marine science mandate without a fleet. For example, it could not undertake fisheries research or hydrographic surveying. In essence, Fleet Maintenance and Fleet Procurement support all three departmental strategic outcomes.

4.2 EFFECTIVENESS

Overall Findings

The evaluation examined whether vessels and ACVs were maintained and new vessels were acquired (immediate outcome) and whether vessels were reliable and a modern fleet available (intermediate outcome) in support of the federal government’s on-water needs and priorities (long-term outcome). It also investigated whether external factors or challenges likely had an impact on the results and whether Fleet Maintenance or Fleet Procurement resulted in any unintended outcomes. In summary, an older civilian fleet was available to support federal programs and priorities to the extent possible, taking into account limitations due to the maintenance requirements of older vessels and the lack of a modern fleet even with some new assets procured.

IMMEDIATE OUTCOMES


Key Findings Fleet Maintenance: Vessels were maintained primarily to meet regulatory requirements.

Key Findings Fleet Procurement: Some new assets were procured, and there is evidence that life cycle management is being considered at the procurement stage.


FLEET MAINTENANCE: EXTENT VESSELS AND ACVS WERE MAINTAINED

Need to Focus on Outcome

Fleet Maintenance’s indicator for its immediate outcome focuses on operational activities rather than the immediate outcome of having done maintenance work. Specifically, the indicator is the percentage of maintenance work milestones met. These milestones, for example, address contracting activities, such as the number and percent of projects that started on time or were late, were posted on Merx on time or late, were awarded on time or late, and so forth.

We appreciate that such information may be a useful and necessary part of project management. These activities contribute to achieving outputs, in this case maintained, refitted and refurbished vessels, ACVs and small craft. However, focusing on activities is too detailed for evaluation purposes and not central to assessing outcomes.

The immediate outcome of having completed maintenance work (output) is the availability of operationally ready vessels, ACVs and small craft. The extent to which this immediate outcome has been achieved (i.e., the extent to which vessels and ACVs are maintained to meet program requirements) would, it seems, be reflected in relatively few breakdowns and hence little need for unplanned, corrective maintenance.

It would seem appropriate then to consider such indicators as follows in assessing the immediate outcome for Fleet Maintenance:

  • percentage of operational days required for unplanned maintenance;
  • percentage of operational delays due to equipment breakdowns (if not part of the preceding percentage); and
  • extent of corrective maintenance required.

Unplanned Maintenance Requirements: Up, Down or Consistent?

We found some data available for the first two indicators suggested above. The findings for each are discussed in turn in the following sections. The findings as a whole are inconsistent, presenting a somewhat conflicting picture: either i) the time required for maintenance, including unplanned maintenance, remained consistent or ii) there has been an increasing need for maintenance, especially unplanned maintenance.

1) Maintenance Time

Although Fleet annual reports and Departmental Performance Reports include some data pertinent to the evaluation, an inconsistent and incomplete story emerges when we look at the available data and all lines of evidence. For example, vessel availability (vessels are assigned or ready to be assigned for operational duty) has been relatively consistent year over year since at least 2004-05 at about 66 to 68% of the time. From 2007-08 to 2010-115, vessels were unavailable 12% of the time due to maintenance, with unplanned maintenance accounting for 2 to 3% of this, as reflected in the following table. In essence, there is no statistically significant change in the amount of time required for maintenance.

Table 3. Vessel Availability Annually

Vessel Availability Annually
Available Unavailable Total
Assigned Un-assigned Sub-total Lay-up / Winter Maintenance Other Sub-total
Planned Unplanned
2007-08
67% 1% 68% 19% 9% 3% 2% 32% 100%
2010-11
65% 1% 66% 20% 10% 2% 2% 34% 100%

Sources: Fleet Annual Reports 2007-08 and 2010-11

Given that some unexpected maintenance problems are always likely to occur, then the actual presence of unplanned maintenance in itself is not problematic. The question is at what percent does it become an issue?

The following graph shows that although maintenance days were up in 2008-09 and 2009-10, there is little difference between 2007-08 and 2010-11. This reinforces the data in the preceding table. In fact the number of maintenance days in 2010-11 is slightly lower.

Figure 2. Days Required for Maintenance Days Required for Maintenance Graph

Source: Fleet Annual Report 2010-2011


5 A CCG Fleet Annual Report was not issued for 2011-12.

A couple of factors have a bearing on the figures in the latter two years. First, funding through the Economic Action Plan announced in early 2009 enabled the Coast Guard to complete vessel life extensions on five vessels by March 31, 2011, extending their service life by 10 to 15 years. In addition, the Coast Guard repaired or refit 35 vessels for more cost-effective operation, extended service life and improved reliability and availability to deliver programs. Second, several older vessels were replaced by new ones in 2010-2011. A subsequent drop in maintenance days in the ensuing years makes sense. However, more than one or two years of data is required to assess the overall trend.

Other data and lines of evidence indicate there has been a continuing increase in the need for unplanned maintenance. The 2007 February Status Report of the Auditor General of Canada expresses considerable concern about the Coast Guard’s aging fleet. It notes that vessels were becoming increasingly unreliable, which impacted the Coast Guard’s ability to support delivery of client programs. The 2010 Life Cycle Asset Management (LCAMS) evaluation report notes that more corrective maintenance (i.e., unplanned maintenance) is required with aging vessels and needing to keep them in service well beyond their design life. Aging assets require longer and more frequent maintenance, which affects their availability. That evaluation found that corrective maintenance expenditures increased between 2004 and 2008, while preventive maintenance expenditures decreased. A lack of sufficient Operations and Maintenance (O&M) funding for preventive maintenance and life cycle management work meant vessel maintenance was focused more on regulatory and corrective work.

The refit case study for this evaluation, which looked at the maintenance work undertaken for one vessel, found that most of the work focused on meeting regulatory requirements. Refits have been kept to a minimum and essentially exclude preventive work due to financial constraints. The vessel’s five-year maintenance plan makes few references to life cycle maintenance work. Interviewees for the case study noted that this practice likely affects the vessel’s reliability.

While the additional maintenance work done as a result of the 2009 Economic Action Plan and the acquisition of some new vessels may result in a decline in the maintenance time required going forward, especially unplanned maintenance, the majority of vessels remain well beyond their halfway point of their design life, if not well beyond their endpoint. For example, CCGS Amundsen was taken out of service at the end of its 2011 Arctic mission when cracks were discovered in four of its six engines. This had a significant impact on planned science research projects. Major repairs were completed in May 2013, with the vessel resuming active service in July 2013. Comments from interviewees reinforce the perspective that maintenance continued to be a pressing issue for the period 2007-08 through 2011-12 and even now.

2) Operational Delays

Operational delays due to equipment breakdown appears to be another indicator relevant to the immediate outcome.  In Fleet Annual Report 2007-2008 the total reported for the year was 134 lost days, and for the previous four years it varied between about 100 and 140 days. Presumably this is in addition to the planned and unplanned maintenance days reported in figure 2.  

The 2007-08 annual report notes for the period 2003-04 through 2007-08 that equipment breakdown increased by 28.85%. We were not able to determine the trend for 2007-08 through 2011-12, as statistics on equipment breakdowns were not reported in subsequent annual reports or otherwise available to us. We also noted that the number of unplanned maintenance days is not always included in the annual reports. For example, the 2007-08 and 2009-10 reports do not include this, but the 2010-11 report includes figures for 2009-10 and 2010-11.

FLEET PROCUREMENT: EXTENT TO WHICH ASSETS WERE PROCURED

Long-term Initiative

Fleet Procurement is a relatively new sub-program in Coast Guard, having been introduced in 2009-10. It takes a considerable amount of time to initiate the preliminary work and get to the conceptual design stage. Initial documents address such aspects as vessel or helicopter needs, project management, risk management and technical specifications. It then takes time to prepare tenders, go through the bidding process and have procurement contracts in place, or to plan and schedule construction through the newly introduced National Shipbuilding Procurement Strategy. Looking at a five-year period therefore provides a relatively narrow view with respect to Fleet Procurement outcomes.

Procurement of vessels, small craft and helicopters is also contingent on industry being ready and available to proceed with design and construction work. The National Shipbuilding Procurement Strategy has implications for the Coast Guard’s large vessel projects in terms of timelines and results achieved over the short and long term, given that the Coast Guard needs to coordinate with the shipyard and other federal departments (i.e., the Department of National Defence and Public Works and Government Services Canada) to determine the priorities in terms of vessel construction.

Need to Focus on Outcome

As with Fleet Maintenance, the performance indicator for the immediate outcome is not quite on target. The indicator identified in the performance measurement strategy ― percentage of vessels delivered versus planned ― is really an output indicator (number of units obtained). An immediate outcome of having purchased vessels, for example, would be the availability of operationally ready vessels that comply with specified fleet and client operational requirements.

As with Fleet Maintenance, Fleet Procurement had data on the percentage of critical project milestones met, which likewise focuses on operational activities. This information may be useful to the sub-programs, but it does not shed light on immediate outcomes in terms of key results.

In the absence of data specific to the immediate outcome, we looked at procurement projects underway (outputs). We also looked for evidence of life cycle management being taken into consideration at the procurement stage, in accordance with a recommendation in the 2010 LCAMS evaluation.

Vessel Procurement Projects

Federal budgets from 2005 to 2010 allocated a total of over $2.1 billion to the Coast Guard for vessel procurement and maintenance. This includes $175 million provided as part of the Economic Action Plan to acquire small vessels and crafts by end of March 2011, as well to cover vessel life extensions and refits. In addition, the Coast Guard provided internal funding for two near-shore vessels and an ACV. For the period evaluated, the Coast Guard had procurement projects underway as outlined in the following table.

Table 4. Procurement Projects 2007-08 to 2011-12

Procurement Projects Table
Description # Original
Delivery Date1
Actual or
Revised Date
Large Vessels
Mid-shore Patrol Vessels 9 2011-2013 2012–2014
Offshore Fishery Science Vessels 3 2010 2014, 2015
Offshore Oceanographic Science Vessel 1 2012 2016
Polar Icebreaker 1 2017 2017
Small Vessels
Near-shore Fishery Research Vessels 3 2010-11 2012-13
Specialty Vessels 3 2006, 2009, 2011 2009, 2010, 2013
Small Craft, ACVs, Helicopters
Search and Rescue Lifeboats 5 2010-11 2010-11, 2011
Environmental Response Barges 30 2010-11 16 in 2010/11;
14 in 2012
Air Cushion Vehicles 2 2009; 2013-14 2009; 2013-14
Other Small Craft 60 2010-11 2010-11
Helicopters 0 - -
  1. Original delivery dates for Fleet Procurement projects are per initial Preliminary Project Approval submissions.

At the time of writing this evaluation report, the Coast Guard was in the process of examining the projected delivery date for the polar icebreaker due to a federal government decision to give priority to a naval vessel ahead of the icebreaker. The shipyard does not have the capability to build vessels for both the Coast Guard and National Defence in tandem.

Evidence of Life Cycle Maintenance Considerations

As noted in the 2010 LCAMS evaluation report, an important part of procurement includes appropriate training, documentation, manuals and maintenance plans for new vessels to ensure their longevity without incurring excessive life cycle maintenance costs. Following the 2010 evaluation report, ITS Management had formal agreements (MOUs) in place with Operations by April 2012 to ensure costs associated with asset support and maintainability requirements are included in all project estimates, including Memoranda to Cabinet.

By early 2013, Vessel Procurement and ITS had introduced monthly management meetings to address life cycle issues arising. As well, the Coast Guard created positions to support life cycle management and in-service support as part of vessel procurement projects. These positions are intended to ensure ITS is actively involved in asset acquisitions for life cycle asset management purposes.

INTERMEDIATE OUTCOMES


Key Findings Fleet Maintenance: Vessel reliability is declining.

Key Findings Fleet Procurement: Overall, the Coast Guard’s fleet continues to approach the end of its useful life.


FLEET MAINTENANCE: AVAILABILITY OF A RELIABLE FLEET

Declining Reliability

Vessel condition scores are a key element in assessing whether a reliable fleet is available. Table 5 shows that large vessels, and to a lesser extent small vessels, declined in their condition scores between 2009-10 and 2011-12. Large vessels scored 54.1 overall in 2009-10 and small vessels, 74.0. The score for large vessels dropped by 23.2% over the next two years, while the overall score for small vessels decreased by 9.3%.

Table 5. Condition Scores: Large and Small Vessels  

Condition Scores
Performance Indicator Vessel Targets Baseline 2009-10 March 31, 2012 % Change
% change in condition score compared to baseline

Large – 64.4

Small – 65.8

54.1

74.0

41.5

67.1

-23.2%

-9.3%

Other data likewise indicates a significant decrease in the condition of vessels between 2009-10 and 2011-12, with a corresponding jump in a need for major repairs. The percentage of vessels requiring significant refurbishment remained fairly consistent. As shown in table 6, 97% of large vessels and close to half of small vessels, 47.6%, required major repairs or significant refurbishment as of 2012. The balance were in good condition.

Table 6. Repairs or Refurbishment Required

Repairs or Refurbishment Required
  2009 (%) 2012 (%)
Large Vessels
Good condition or only minor repairs required 52.5 2.8
Major system repairs required 27.5 72.2
Significant equipment or system refurbishment required 20.0 25.0
Small Vessels
Good condition or only minor repairs required 81.4 52.4
Major system repairs required 12.8 41.5
Significant equipment or system refurbishment required 5.8 6.1

Input from both the document review and interviews suggest that budget restrictions have been a key contributing factor. There has been a lack of sufficient O&M monies for ongoing preventive maintenance, and interviewees noted that maintenance budgets have not kept pace with inflation. As a result, regulatory and corrective maintenance requirements have taken precedence over preventive maintenance work.

The foregoing results are congruent with Fleet Procurement data that shows the remaining operational life of all large and small vessels is on average around 25%, well below a 50% target. As well, an increase in the number of reported unsatisfactory conditions corroborates these findings. An unsatisfactory condition includes technical problems, breakdowns or deficiencies with systems or equipment that may affect the safe operation of machinery or the safe and efficient delivery of the program. Between 2007-2008 and 2010-11, the number of unsatisfactory conditions reported nearly doubled, from 168 to 324, as shown in the following graph.

Figure 3. Reported Unsatisfactory Conditions

Unsatisfactory Conditions Graph

Source: Fleet Annual Report 2010-11

FLEET PROCUREMENT: AVAILABILITY OF A MODERN FLEET

Remaining Operational Life Low

The extent to which a modern fleet is available is based on the remaining operational life of vessels and helicopters. The Coast Guard currently has a high percentage of old assets and a few new ones. As such, the overall remaining operational life of these assets is relatively low. Table 7 below shows the results for 2010-11 and 2011-12.

Table 7. Procurement Intermediate Outcomes

Procurement Intermediate Outcomes
Performance Indicator Target   2010-11
(%)
2011-12
(%)
% operational life remaining 50%

Large vessels
Small vessels
Helicopters

23
27
10

21
29
6

In the years ahead, the intermediate outcome of procuring new vessels will gradually shift the remaining operational life to a better balance between old and new. A 50% target for remaining operational life appears reasonable, as there will always be some vessels beyond their midlife point balanced by newer vessels that have yet to reach the midpoint.

LONG-TERM OUTCOME


Key Findings: Operationally ready vessels were available in support of federal programs and priorities, although to a somewhat limited extent.


The Fleet Operational Readiness logic model shows the two sub-programs contributing to a strategic result, namely, the federal government has access to an operationally ready fleet to facilitate its on-water needs and priorities. For evaluation purposes, this is the long-term outcome, given that strategic outcomes are at the departmental level. The strategic outcome for the program and sub-programs is Safe and Secure Waters. The results show that Fleet Maintenance and Fleet Procurement contributed to an operationally ready fleet being available for a wide range of services in support of federal programs, although not without some unexpected setbacks as noted previously.

The assessment of progress towards the long-term outcome will also be addressed in the Fleet Operational Capability evaluation to be reported on in 2014-15.

EXTERNAL FACTORS, CHALLENGES


Key Findings: The availability of additional funding, shipyard capacity and shipbuilding priorities, an aging fleet and a competitive labour market emerged as the key external factors and challenges affecting program results.


Reliance on Additional Funding

Since 2003, Fleet Maintenance has received roughly an additional $47 million annually to assist with required maintenance work. In short, its budget has been insufficient in meeting maintenance requirements for the past decade. These additional monies still were not enough to cover all maintenance requirements. Canada’s 2009 Economic Action Plan included $175 million, part of which was designated for major repair work. The outcome findings reported above indicate that adequate maintenance funding continues to be an issue. This is reinforced by DFO’s 2011 audit which states that the Coast Guard’s capital budget is insufficient for ongoing maintenance needs. 

Approval of the Fleet Renewal Plan, which covers both vessel life extensions and procurement of new vessels and helicopters, has been a major factor in influencing program outcomes. The funds approved for Fleet Procurement are in addition to its established budget. The commitment of such funds underlines the significance of these investments by the Government of Canada in support of the Coast Guard.

While the Canadian Coast Guard funds the purchase of some small vessels and craft from its annual capital budget, the following figures show the extent to which it relies on approval of additional funding for asset procurement. Since 2005, the federal government has invested an additional $6.8 billion in the Canadian Coast Guard’s fleet:

  • $1.4 billion in budgets 2005, 2007, 2008 and 2010 ― to acquire nine mid-shore patrol vessels, four scientific research vessels (three for fishery science, one for oceanographic science), a hovercraft and Canada’s first polar icebreaker;
  • $175 million in Canada’s 2009 Economic Action Plan ― for major repair work on 40 large vessels and to acquire 98 new small vessels and craft; and
  • $5.2 billion as part of Canada’s 2012 Economic Action Plan ― to procure up to 10 large vessels, 18 to 21 small vessels and up to 24 helicopters; to provide additional funding for the polar icebreaker project; to extend the lives of 16 Coast Guard vessels; and to complete mid-life modernizations on two hovercraft.

Shipyard Capacity and Shipbuilding Priorities

Interview respondents noted that the ability of shipyards to respond to the scale of the Fleet Renewal Plan could impact long-term delivery timelines. Informants also noted that the National Shipbuilding Procurement Strategy represents both new challenges and new opportunities for the Coast Guard. There is a need to coordinate shipbuilding priorities with the Department of National Defence, Public Works and Government Services and the shipyard. As a result, the timing determined for certain Coast Guard vessels before the procurement strategy was introduced is now subject to revision.

Maintaining an Aging Fleet

Both the document review and interviews revealed that repairing and refurbishing older vessels can pose a serious challenge. As equipment and vessels are kept in service beyond their expected life, economical repair and readily finding replacement parts become increasingly problematic. The scarcity of some parts may result in considerable effort having to be expended to find replacements, through unconventional sources or suppliers or perhaps even having to pay a premium for custom-made replacements. Given the rapidity of technological development, electronic equipment and associated software specifically designed for vessels can be difficult if not impossible to obtain. Within a relatively short period of time, manufacturers may no longer produce certain parts due to advances in technology, or the original equipment manufacturer may no longer be in business. Unconventional suppliers may include oversea scrapyards or eBay. If a suitable spare part cannot be found, a much more extensive overhaul or complete system replacement may be required to keep a vessel operational.

Changes in delivery times for new vessels to replace aging ones, therefore, have implications in terms of interim maintenance and refit costs for existing vessels that now need to be kept in service longer than anticipated. This is especially true for vessels already well beyond their expected design life. Effective maintenance management decisions cannot be made when delivery times change considerably and it is not known whether older vessels need to be kept in service for a short while longer or considerably more time. For example, the estimated useful life of the Coast Guard’s heavy icebreakers is 30 years. One of them, the CCGS Louis S. St-Laurent, is already well beyond its design life having been built 44 years ago in 1969. By the time its replacement is delivered, it could be well beyond 50 years.

Competitive Labour Market

A number of interviewees noted that with the increased investment in maintenance and procurement activities, the Coast Guard is facing an increasingly competitive labour market for recruitment and retention of a limited labour pool of skilled employees. Most notable are the difficulties in finding qualified marine engineers. The Coast Guard is competing for limited staff resources with the two major shipyards identified in the National Shipbuilding Procurement Strategy and with industries that support those shipyards.

4.3 EFFICIENCY AND ECONOMY

Overall Findings

The evaluation examined whether Fleet Maintenance and Fleet Procurement have the appropriate governance and management processes and systems in place to support program delivery, whether roles and responsibilities are clear, and whether the program is using its resources efficiently. Progress has been made in achieving greater consistency in the organizational structure and setting the stage for greater consistency to be demonstrated in practice. Both sub-programs make use of best practices and lessons learned. No economical alternatives emerged, but there are some inefficiencies to be addressed.

GOVERNANCE AND MANAGEMENT STRUCTURE


Key Findings: Recent changes and other efforts towards greater consistency now need to be proven in practice. The performance measurement strategy needs some adjusting.


Organizational Changes Introduced

The Coast Guard’s overall organizational structure has undergone some significant changes since 2007-08. Prior to this, the Auditor General repeatedly called for greater consistency within the Coast Guard. According to the 2007 February Status Report of the Auditor General of Canada, a standard organization structure was largely in place at headquarters by mid-2006, but implementation in the regions had yet to occur for various reasons. The Coast Guard’s own 2006 internal review likewise had identified a need for greater consistency, having found a wide range of operational practices.

As a result of these observations, the Coast Guard introduced a new Fleet Operational Readiness program as part of the department’s PAA in 2008-09. All three sub-programs (Fleet Operational Capability, Fleet Maintenance and Fleet Procurement) were formally part of the department’s PAA as of 2009-10.

In the current structure, the Fleet Maintenance sub-program is delivered by Integrated Logistic Support, one of four branches within Integrated Technical Services (ITS). Integrated Logistic Support is the common technical services provider for ITS, which includes the technical program planning and performance function for Fleet Maintenance. As well, ITS includes the Marine Engineering branch which did not exist in 2008. The responsibilities of ITS remain largely the same.

The Deputy Commissioner, Vessel Procurement is responsible for Fleet Procurement activities, supported by a Director General, Major Projects, a Project Manager, Polar Icebreaker, a Director, Business Support and a Director, Engineering Support. Fleet Procurement will continue to evolve as it proceeds with projects currently underway and new ones are brought online in keeping with the Fleet Renewal Plan and available funding. With this sub-program having been introduced in 2009-10, and given the complexities of major capital acquisitions and the subsequent introduction of a National Shipbuilding Procurement Strategy, it is too soon to arrive at any definitive findings and conclusions regarding the governance and management structure for Fleet Procurement.

In addition to these PAA changes, the Coast Guard consolidated its regional structure in 2012-13 from five regions to three. Most elements of the new structure came into effect October 1, 2012, with the entire structure fully implemented by March 31, 2013. As part of the recent consolidation, the Coast Guard standardized the organizational charts for the three regions, with some minor variances.

Both Fleet Maintenance and Fleet Procurement interviewees were positive about the recent organizational changes. According to them, well-defined governance structures and roles and responsibilities are now in place, along with the necessary rigour for good governance. As well, they believe there is good collaboration between the Fleet Operational Readiness sub-programs, particularly at senior management levels. Further progress can be made through continuous improvement efforts and using the Vessel Maintenance Management Manual and the Centres of Expertise that have been put in place.

Still, there appears to be room for improved communication. The Auditor General’s 2007 status report stated that the Fleet Renewal Plan was outdated and unrealistic. The Coast Guard reportedly consulted extensively with others when it subsequently revised the plan. However, the 2011 DFO audit found that not everyone responsible for maintenance was aware of the future asset requirements that had been identified.

The majority of interviewees think that their sub-program is efficient to a considerable extent in the following four areas, with one exception as noted under the third item:

  • service planning and delivery;
  • collaboration with partners and clients;
  • process for human resource planning and management ― Fleet Procurement respondents on average consider their sub-program to be efficient only to a moderate extent in this area; and
  • process for asset planning and management.

Some Fleet Procurement interviewees cited the small number of staff in relation to the size of capital investment as being of concern. They questioned if they had the right project management balance in terms number of staff in relation to the size of Fleet Procurement’s capital investments. 

Moving Towards Standardized Maintenance Procedures

Although Fleet’s 2007-08 annual report indicates national operating procedures and policies were in place for all aspects of its operations, considerable work in fact remained to be done towards achieving consistencies in its procedures and practices, as clearly indicated in the February 2007 Status Report of the Auditor General of Canada and a 2011 DFO audit. For example, the 2007 Auditor General report states that no standardized maintenance procedures and manuals were in place. The report notes that maintenance failures have been partly due to a lack of materiel management systems and standard maintenance procedures and manuals. The 2011 audit found no standards in place for various systems used to track on-board and stored inventory.

Subsequent to the Auditor General’s report, the Coast Guard undertook a vessel maintenance management review. The review resulted in several recommendations, following which the Coast Guard proceeded with corrective actions beginning in 2008-09. Recommendations from the review and the 2010 LCAMS evaluation led, for example, to the creation of three centres of expertise introduced in late 2009-10, one a marine engineering group, another vessel maintenance management and the third being materiel identification to strengthen inventory management, each having a mandate for continuous improvement.   

The centres contribute to operations by being focal points for addressing issues arising. For example, Marine Engineering can assist with maintenance in the regions, by looking at problems arising, determining the extent of their occurrence across fleet assets, whether the problem has to do with the hardware or system itself (e.g. a pump) or the way it is installed.

Further to the creation of a Marine Engineering unit within ITS, another element arising from the Vessel Maintenance Management Review has been the development of a Vessel Maintenance Management Manual, issued in March 2011 and posted on the Internet. The manual is intended to delineate more clearly the roles and responsibilities of various Coast Guard personnel with respect to maintenance and identify standard procedures. The US Coast Guard has also focused on standardizing procedures as a means of dealing with aging vessels that are approaching or have exceeded their expected design life.

Having a maintenance manual that consolidates information in one document is a first step. It can then take considerable time and effort before consistency is achieved in practice. For example, the manual documents roles and responsibilities for first, second and third line maintenance. The 2011 DFO audit found evidence of discrepancies with the manual in practice, specifically, second line maintenance was not being done by Coast Guard personnel due to limited time and capacity to do the work. Given the large number of employees involved in operating and maintaining Coast Guard vessels, it will take some time for the manual to be fully ingrained in practice throughout fleet operations. Marine Engineering is aware of the need for continuing and considerable effort in promoting the manual and its active use.

Given the recent nature of these developments, it is too soon for any substantive evidence of greater consistency occurring in practice as a result of the organizational changes and new procedural documents and systems.

Standardizing Fleet Procurement Procedures

The 2011 DFO Audit of Fleet Asset Management found that vessel procurement projects were not using a consistent approach and standard procedures for project management. The impact of not having a consistent framework in place for implementing vessel procurement projects had contributed to weaknesses in areas such as project risk assessment, project cost estimates and client liaison. The audit recommended that a project management framework be developed. The framework, which was completed in March 2012, received approval from the Deputy Commissioner, Vessel Procurement in September 2012.

Performance Data: Getting the Metrics Right Requires Some Further Work

As of April 2012, Fleet Operational Readiness had a performance measurement strategy in place. This is a considerable step forward. The Auditor General’s 2007 status report indicates that clear, concrete, realistic and consistent performance expectations and data for the Coast Guard’s fleet operations has been an issue going back to 2000. DFO’s 2009 Evaluation of Fleet Operational Readiness similarly had concerns over the accuracy and reliability of data. They noted discrepancies between information they obtained from a fleet information system and results presented in a Fleet annual report which led to opposing conclusions. In the course of conducting this evaluation, program staff were not able to readily provide some of the performance data we requested.

In accordance with Treasury Board’s Policy on Evaluation, the purpose of the performance measurement strategy is to ensure there is performance data available that is relevant to evaluations. This information is needed to assist in demonstrating to elected officials, senior management and other interested Canadians whether a program is achieving its intended outcomes in support of a departmental strategic outcome.

In the absence of such a strategy for 2007-08 through 2011-12, we applied the new one for this evaluation. It became evident in the course of the evaluation that the logic model requires some adjustment, as do some of the performance indicators. For example, outputs should reflect the main purpose of a program (e.g., # of vessels maintained, # of vessels procured), not production of plans and reports. The need for such adjustments is not entirely surprising. According to a 2005 survey conducted by the Organisation for Economic Co-operation and Development (OECD), many OECD governments found performance measurement far more difficult than anticipated. Their experience and that of industry indicates it can take years to get the metrics right, possibly five to seven years if not longer. Still, as noted above, the need for good fleet performance data has been an ongoing issue for close to 15 years.

COMPLEMENT, DUPLICATE OR OVERLAP?


Key Findings: Fleet Maintenance and Fleet Procurement do not duplicate or overlap any other federal programs. The Coast Guard’s fleet as a whole complements other federal programs.


Inasmuch as the Coast Guard is the sole owner, manager and operator of Canada’s civilian fleet, there is no duplication with other federal department or agency programs. A majority of interviewees noted that the Coast Guard’s fleet does not duplicate or overlap with other federal programs, it complements them. For instance, while the Department of National Defence owns and operates Canada’s Navy, the Navy’s mandate fundamentally differs from the Coast Guard’s civilian fleet. The Navy does its maintenance in-house, whereas the Coast Guard uses Alternative Service Delivery methods for most of its maintenance. Where other government departments such as the RCMP, Environment Canada and Parks Canada have small craft vessels, these assets are complementary to the Canadian Coast Guard’s. 

The literature review revealed that many nations adhere to a regime of cooperation and coordination between their maritime agencies (e.g., navy and coast guard) in order to fulfil a range of missions to protect their nations, citizens and interests on the water. Interviewees and survey respondents indicated that Canada’s situation is similar: the Canadian Coast Guard and Royal Canadian Navy complement each other rather than duplicate.

A comparative analysis further underscored the complementary nature of the Coast Guard and the Navy. Both are taking part in the largest procurement sourcing arrangement in Canadian history, namely the National Shipbuilding Procurement Strategy. The Navy requires combatant (warship) and some non-combatant vessels, whereas the Coast Guard will be engaging the shipyards to build a polar class icebreaker, mid-shore patrol vessels, an offshore oceanographic science vessel and offshore fisheries science vessels.  None of the Navy vessels is in keeping with the Coast Guard’s mandate, and vice versa. With respect to vessel procurement, the comparative analysis found that Canada’s multi-department governance and oversight structure, its centralized procurement function and its specific project management functions are similar to those of other countries. As well, most countries use a competitive bidding process to select contractors for construction of new vessels.

OPPORTUNITIES FOR IMPROVING EFFICIENCY


Key Findings Fleet Maintenance: Fleet Maintenance needs to continue pursuing greater consistency and efficiencies.

Key Findings Fleet Procurement: There is a concern as to whether Fleet Procurement is sufficiently staffed in relation to the work required. Assessing the adequacy of project management support in relation to the scope and value of procurement projects would contribute to determining Fleet Procurement’s ability to operate efficiently.


Continuing Need to Improve the Asset Management System

According to Fleet reports, improvements made in 2007-08 meant that maintenance planning and fiscal management could proceed in a more rigorous fashion. These improvements included, for example, a new refit authority consolidating all refit resources into a single budget and the establishment of national policies and procedures for all aspects of its operation. Other evidence shows, though, that improvements needed to be made in inventory management and costing, and there is still room for improvement.

The February 2010 Evaluation of the Life Cycle Asset Management Services Program recommended an inventory management system by asset class and region, consistent with DFO policies on inventory management, be phased in. This system is for the purpose of tracking, managing and valuing Coast Guard inventory holdings. At the time of the report, the Coast Guard noted it had computerized inventory management capacity in place, specifically referencing its Asset Management System. This system is comprised of MAINTelligence™, which is used for large vessels, and MAXIMO™, which is for small vessels. Further to the recommendation, the Coast Guard introduced a Shore Based Inventory Management Guide by March 2012. It subsequently reported in March 2013 that it had deployed inventory management practices and processes on its vessels and will continue to do so on additional ships as required.

The 2011 DFO Audit of Fleet Asset Management indicates that assessing maintenance requirements and estimating associated costs required improvement. The audit report noted that without integrated information systems for maintenance management and materiel management, the Coast Guard did not have complete and accurate information to adequately support decision making. A lack of integrated maintenance and cost information systems had been previously noted in the Auditor General’s 2000 report and again in the Auditor General’s 2007 status report.  In response to the 2011 DFO audit, the Coast Guard committed to developing a bridge between MAINTelligence™ and MAXIMO™ or to convert to a single management system for all of its assets.

Program staff observed that data is still not being consistently entered into the Asset Management System. They also noted that the management system is not yet linked to the financial system. Survey results likewise underscore these findings. Respondents noted that record keeping and documenting lessons learned needed to be improved. Results from the case study reinforce this and indicate that a formal process is being put in place to capture best practices and lessons learned. The case study and survey results suggest that MAINTelligence™ and MAXIMO™ should be used for maintenance planning, scheduling and comparing actual expenditures against budget estimates.

The US Government Accountability Office observed in a July 2012 report, Legacy Vessels’ Declining Conditions Reinforce Need for More realistic Operational Targets, that a high-quality, reliable cost estimate providing well-documented, comprehensive and accurate cost information would assist the US Coast Guard in effectively allocating its resources in a constrained federal budget environment.

Other Opportunities

According to survey respondents, efficiencies could also be improved if there were more accountability and communication between senior vessel staff and shored-based employees responsible for maintenance. As well, they noted that due to budget and time constraints, temporary repairs are made rather than permanent ones, although it is recognized that in the long term a permanent repair is more cost-effective.

The document review reinforces a need for specialized training in electronics and mechanical equipment. In addition, new engines should come with diagnostic software and appropriate training to contribute to cost savings. For example, crew members who received training in overhauling a main engine realized savings of approximately $200,000. Such training reduces dependence on external suppliers for more expensive maintenance and repairs.

Staff Resources for Procurement Projects

There are concerns within Fleet Procurement about the possible under-resourcing of project management support activities. As the majority of vessels are well beyond their mid-life point and many are already beyond their planned operational life, vessel procurement will remain a key activity in the years ahead. Given the size and complexity of upcoming projects, the Coast Guard is committed to ensuring that it has the appropriate resource levels in light of future requirements.

USE OF BEST PRACTICES AND LESSONS LEARNED


Key Findings: Both sub-programs make use of best practices and lessons learned to improve their functioning and delivery of services. The Vessel Maintenance Management Manual, cataloguing of parts, a Central Project Coordination Office and improved costing methodology are key examples.


Fleet Maintenance Initiatives

Interviewees mentioned the Vessel Maintenance Management Manual as an example of a best practice in that it supports a standard way of doing vessel maintenance across the Coast Guard. The manual is intended for all employees and contractors responsible for any aspect of vessel maintenance. The key aspects of the framework are that, in one comprehensive document, it assists those planning the maintenance activities, those carrying out the maintenance activities, and those using the management tools that support

the maintenance activities. They also noted a collaborative initiative with the Department of National Defence to standardize the cataloguing of vessel parts in alignment with North Atlantic Treaty Organization materiel standards. The materiel coding standard will help to standardize the format used to identify all government materiel assets and will enable departments to communicate better and carry out their transactions electronically. The Coast Guard’s Centre of Expertise for Materiel Identification, located in Halifax, introduced this cataloguing system about two years ago.

In addition, interviewees noted the importance of having a Central Project Coordination Office as a source for guidance and documentation related to managing and implementing refit, maintenance as well as capital projects within the organization. The office acts as a hub by receiving information on the various refit, maintenance and capital projects and distributing to senior management at the Coast Guard’s Investment Management Committee.  The literature review reinforces the benefits of having information available centrally to better inform management decisions.

Fleet Maintenance also uses a project close report to capture lessons learned, issues and challenges encountered. After major work periods, maintenance managers review extras incurred to inform future work specifications. The goal is to reduce the number of extras when conducting similar work in the future.

Fleet Procurement Initiatives

The literature notes the importance of establishing a dedicated procurement directorate, as such units create a learning environment and expertise within the purchasing organization. This reinforces the Coast Guard’s decision to create a Vessel Procurement Sector and a Fleet Procurement sub-program. The comparative analysis highlighted the benefits of long-term procurement arrangements for shipbuilding projects and consequently being in a position to engage industry, partners and clients early in the procurement process. With the creation of the National Shipbuilding Procurement Strategy, the Coast Guard is able to implement this best practice when negotiating its shipbuilding contracts.

Like Fleet Maintenance, Fleet Procurement is documenting lessons learned in an effort to achieve greater consistency between projects and to encourage continuous improvement across all projects. According to interviewees, regular project updates and formal quarterly reporting, combined with reporting against the Coast Guard’s three-year business plans, have all contributed to an increase in sharing lessons learned and leveraging greater efficiencies. In addition, the document review indicated that lessons learned from procurement projects have subsequently been used to improve the costing methodology.

MINIMIZING USE OF RESOURCES


Key Findings Fleet Maintenance: For budgetary reasons and due to the advanced age of fleet assets, regulatory compliance and corrective maintenance took precedence to the detriment of preventive and predictive maintenance.

Key Findings Fleet Procurement: The planned reduction in the number of vessel designs will contribute to achieving standardization and greater efficiencies in equipment, systems, maintenance and crew training, and deployment of vessels over the long term.


Program Investment

As noted in section 2, Program Profile, budgetary figures were available only for Fleet Operational Readiness as a whole, and not by sub-program, for 2008-09 through 2010-11. Sub-program data available for 2011-12 is presented in the table below. According to this data, Fleet Maintenance accounted for 10% of the total approved budget (total authorities), Fleet Procurement accounted for 29%, and Fleet Operational Capability accounted for the balance of 61%.

Table 8. Fleet Operational Readiness Approved Budget for 2011-12

Fleet Operational Readiness Approved Budget for 2011-12
($ millions)
Sub-Program Total Authorities % of Total
Fleet Maintenance 47.1 10.1%
Fleet Procurement 137.1 29..4%
Fleet Operational Capability 282.2 60.5%
Total 466.4 100%

Source: Departmental Performance Report 2011-12

The financial data available for Fleet Operational Readiness is presented in the table below. Figures for 2008-09 are incomplete as they do not include Employee Benefit Plan monies and omit salary dollars that are transferred from two DFO sectors. Figures for 2009-10 onwards reflect technical adjustments made to include these monies, which explains part of the increase for 2009-10 figures compared to 2008-09.

Changes in approved capital budgets and expenditures account for a significant portion of the fluctuations over the five years. Specifically, expenditures for the middle two years are higher due to additional funding from the 2009 Economic Action Plan for maintenance and procurement initiatives to be completed by March 2011. Capital expenditures for 2009-10 increased sharply by 74.7% over 2008-09. They increased a further 10.4% in 2010-11, before dropping by 22.9% in 2011-12.

Table 9. Fleet Operational Readiness Budgets and Expenditures ($ millions)

Fleet Operational Readiness Budgets and Expenditures ($ millions)
  2008-09 2009-10 2010-11 2011-12
Total Budget and Expenditures
Voted Authorities 314.4 440.6 455.0 466.4
Expenditures 274.1 433.7 461.6 416.9
% Change 1 6.4% (9.7%)
  1. As the 2008-09 figures do not include all salary and Employee Benefit Plan figures, the percentage change for 2009-10 is omitted.
Capital
Expenditures 109.1 190.6 210.4 162.2
% Change 74.7% 9.7% (22.9%)

The majority of interview respondents thought that the value and the benefit derived from the two sub-programs were greater than the cost of program funding and administration. They did not identify other, more economical alternatives with respect to delivering either program.

The Cost of Minimizing Maintenance

As noted in the 2010 LCAMS evaluation and again in the current evaluation, regulatory compliance and corrective maintenance of necessity took priority, with little in the way of available resources for preventive and predictive maintenance. Findings from the document review, survey and case study all underscore the downside of this approach. Specifically, the lack of preventive and predictive maintenance reduces the life expectancy of an asset and would be more cost-effective in the long term.  However, given the available budget and the age of the asset base, the program is left with little choice other than to prioritize safety and regulatory compliance. The case study illustrates the implications of this approach, revealing that the lack of resources to support preventive refit work necessitated emergency repairs, affecting vessel availability and resulting in more costly repairs.

As noted in the following section, procurement of new vessels is part of the answer to achieving some greater economy with respect to maintenance costs. Interviewees also observed that as vessels are old and nearing the end of their operational life, they require more money than is optimal to spend on maintenance.  This is true even with the recent vessel life extensions.

Achieving Economies through Fleet Renewal

The literature suggests that the economic efficiency of a fleet can be improved by adding new vessels when the benefit exceeds the additional cost. With the majority of Coast Guard vessels past their midpoint and many nearing or beyond the end of their expected design life, more money than is optimal is required for maintenance and vessel life extensions. As new vessels become available, Coast Guard operations and maintenance will become more economical. A majority of those interviewed observed that the fleet needs to reach a steady state where the average age for vessels is much lower than at present. 

The Coast Guard aims to achieve economies with its vessels in other ways, specifically through effective use of vessel and system designs. For example, new vessels will be designed and equipped for multi-tasking. This means that they will be able to adapt to more than one program function over their operational life. In this way, the vessel design contributes to maximizing vessel use. Both the document review and interviewees identified the importance and value in shifting from vessels built for single-purpose to vessels designed and built for multi-tasking. As well, these same sources noted the value of following sound engineering principles and practices during the design phase so that vessels are reliable and able to operate without failure. Moreover, designing vessels for ease of maintenance, including servicing and repairs, will also contribute to minimizing life cycle costs, that is, the cost incurred over a vessel’s life.

The Fleet Renewal Plan for 2011 to 2040 also calls for reducing the number of vessel types from over 30 to 14. This will allow for greater standardization of equipment and systems across the vessels. In turn, this contributes to facilitating crew and maintenance training. Fewer designs also make it easier to reassign vessels and redeploy crews.

5. CONCLUSIONS AND RECOMMENDATIONS


5.1 RELEVANCE

This evaluation concluded that both Fleet Maintenance and Fleet Procurement are integral to the continuing availability of Canada’s civilian fleet, and there is a legislated mandate for the Coast Guard to have such a fleet. The sub-programs align with federal and departmental roles and priorities, and they contribute to supporting the priorities of other federal organizations. Canada’s Northern Strategy and increasing marine activity in Canadian Arctic waters, combined with an inadequate infrastructure to respond to Arctic marine disasters, underscore the need for stronger fleet presence in the North, and hence a greater need for Fleet Maintenance and Fleet Procurement.

5.2 EFFECTIVENESS

Although a conclusion to be drawn from the findings is that vessels are available, the Coast Guard’s aging fleet requires considerably more maintenance than would otherwise be necessary for a younger fleet. This is supported by the fact that there is clear evidence that vessel reliability has declined, leaving vessels more susceptible to breakdowns and needing more maintenance as reflected in the intermediate outcome.

Fleet Procurement made some progress in acquiring new capital assets and initiating work on several major procurement projects. Given the long timelines for such capital projects, reaching the goal of having a modern fleet remains, understandably, well into the future. Progress to some extent is subject to the National Shipbuilding Procurement Strategy, which requires Fleet Procurement to prioritize and sequence its shipbuilding projects with those of the Department of National Defence. Progress is also likely to be affected by the fact that there is stiff competition with industry for access to a very small pool of highly skilled workers.

5.3 EFFICIENCY AND ECONOMY

Progress has been made in achieving greater consistency in the organizational structure and setting the stage for greater consistency to be demonstrated in practice.  Recent organizational changes and other initiatives, such as the Vessel Maintenance Management Manual, are aimed at achieving standardization and much greater consistency throughout Coast Guard operations. The results now need to be proven in the immediate years going forward and beyond.

It is expected that Fleet Maintenance will realize greater efficiencies and economy from further improvements in its operations as follows:

  • Demonstrated consistency in practice ― e.g., extensive compliance with the Vessel Maintenance Management Manual and consistency in data entry, record keeping and documenting lessons learned; and
  • A fully integrated asset management and financial information system or systems ― outstanding need since at least 2000; in addition to tracking information, the system(s) should be useful for maintenance planning and scheduling.

There will be a need to ensure that efforts toward greater consistency continue and subsequent results are monitored and documented to support reporting for program, departmental, auditing and evaluation purposes.

The Fleet Renewal Plan is an example of a best practice, as it addresses both a need to keep old vessels operating while acquiring new ones. Typically during times of fiscal constraint, funding is not available for both, leading to greater inefficiencies and reduced cost-effectiveness over the long term.

The performance measurement strategy, including the logic model, should be adjusted to ensure the required information to support programming decisions, reporting as well as future evaluations is available. In short, the performance measurement strategy needs to be adjusted with regards to performance measures identified for the Fleet Maintenance and Fleet Procurement sub-programs. Some of the performance data is operational in nature and not central to assessing outcomes.

Recommendation 1: The Commissioner should ensure that the Fleet Operational Readiness program builds on lessons learned during the early stages in implementing its performance measurement strategy pertaining to Fleet Maintenance and Fleet Procurement. The strategy, including the logic model, should be reviewed and revised to ensure it incorporates appropriate performance indicators, outputs and outcomes to inform ongoing program management, program decisions and evaluations, and to facilitate an understanding of the program’s overall impact.

ANNEX A: MANAGEMENT ACTION PLAN


Management Action Plan
Recommendation

Rationale: The performance measurement strategy, including the logic model, should be adjusted to ensure the required information to support programming decisions, reporting as well as future evaluations is available. In short, the performance measurement strategy needs to be adjusted with regards to performance measures identified for the Fleet Maintenance and Fleet Procurement sub-programs. Some of the performance data is operational in nature and not central to assessing outcomes.

Recommendation 1: The Commissioner should ensure that the Fleet Operational Readiness program builds on lessons learned during the early stages in implementing its performance measurement strategy pertaining to Fleet Maintenance and Fleet Procurement. The strategy, including the logic model, should be reviewed and revised to ensure it incorporates appropriate performance indicators, outputs and outcomes to inform ongoing program management, program decisions and evaluations, and to facilitate an understanding of the program’s overall impact.

Strategy

Fleet Procurement sub-program

In 2013-14, Vessel Procurement (VP) developed a draft Performance Measurement Strategy (PMS) which includes a revised logic model and updated performance measures for the Fleet Operational Readiness program.  VP will review its draft PMS in the context of this evaluation and lessons learned to finalize its PMS for inclusion into the Fleet Operational Readiness PMS.

Fleet Maintenance (FM) sub-program

As part of DFO’s Performance Measurement Strategy Implementation Plan, Integrated Technical Services (ITS) developed in 2012-13 a comprehensive PMS for both the Shore-based Asset Readiness (SBAR) program and the Fleet Maintenance (FM) sub-program comprising of measures and indicators for each of its major asset management service offerings.  Following the development of the PMS, ITS detailed an action plan which sees the full assessment of its services using the PMS over 5 years covering the period between formal program evaluations.  Given that the majority of measures in the PMS for both programs are similar and that the FM sub-program has been evaluated during FY 2013-14, the Technical Executive Board (TEB) has agreed to commence the implementation of the SBAR program PMS during FY 2013-14 and deferred the FM PMS to FY 2014-15.  This will allow ITS to refine the FM PMS based on lessons learned following the first year of implementation of the SBAR program PMS.

The implementation of the PMS indicates the start of the formal and ongoing management activity of program performance assessment within ITS.  This ongoing management activity will produce annual performance reports to TEB to support senior management in making informed decisions on areas for improvement in program services and delivery.

Management Actions Due Date (by end of month) Status Update:  Completed / On Target  / Reason for Change in Due Date Output
(Fleet Procurement) Submit a draft PMS for the Fleet Procurement program to Evaluation for comment End of  March 2014   Draft PMS
(Fleet Procurement) Revise and finalize a new PMS End of June 2014   Final PMS
(Fleet Maintenance) Revise PMS End of June 2014   Final PMS
(Fleet Maintenance) Commence the implementation of the PMS as per the action plan. NOTE: reporting on the status of implementation will be done via the DFO’s quarterly reporting of the PMS action plan to the CFO. End of June 2014   Communication informing of the start of program performance assessment as part of ongoing management activity.

ANNEX B: EVALUATION MATRIX


Evaluation Matrix
Issue/Questions Indicator Line of Enquiry
Inter views Fleet Manage ment Survey staff Docu ment & file Review Lit Review, Comp arative Analysis Case Studies Fleet  Admin istrative Data Media
Relevance
1.
Is there a continued need for the Fleet Operational Readiness Program, specifically Fleet Maintenance and Fleet Procurement?
  • What are the needs of Canadians regarding the Fleet Operational Readiness Program? Have they changed over time/over the last 5 years?
  • Key informants attest to the need for and importance of the Fleet Operational Readiness Program (i.e., importance of its mandate and the objectives of the program; are the components still relevant?).
  • Fleet utilization data/SLAs/MOUs confirm demand for services by partners and clients
checkmark checkmark checkmark     checkmark checkmark
2.
Is there a current role for Federal Government intervention, and is the Program consistent with Government of Canada priorities?
  • Evidence of appropriate federal role and responsibility.
  • Activities of programs support stated federal roles and responsibilities.
checkmark checkmark checkmark        
3.
To what extent are the mandate and activities of the Fleet Operational Readiness Program aligned with DFO priorities and objectives?
  • Evidence that the program is aligned with departmental mandates, priorities and strategic outcomes
  • Key informants attest to the ability of the programs to fulfill departmental mandates, priorities and objectives.
checkmark checkmark checkmark        
Performance
Effectiveness
4. To what extent has the Fleet Operational Readiness program achieved its immediate outcomes?
a.
Fleet Operational Capability: To what extent were vessels, ACVs, helicopters and small craft operational with the required certificated professionals on board in order to meet program requirements?
  • # of operational days planned  versus available
  • Opinions of key informants, and survey respondents
  • Review of key documents
Part of the 2nd evaluation focusing on Fleet Operational Capability
b.
Fleet Maintenance: To what extent were vessels and ACVs maintained to meet program requirements?
  • % of critical maintenance milestones achieved versus planned
  • Opinions of key informants and survey respondents
  • Review of key documents
checkmark   checkmark   checkmark checkmark checkmark
c.
Fleet Procurement: To what extent were new large vessels, ACVs, helicopters and small craft procured to meet program requirements?
  • % of vessels, ACVs, helicopters and small craft delivered versus planned
  • Opinions of key informants
  • Review of key documents
checkmark   checkmark     checkmark  
5.  To what extend has the Fleet Operational Readiness Program achieved its intermediate outcomes?
a.
Fleet Operational Capability: The extent there was an operationally capable fleet that had the capacity to respond to the operational needs and requirements of the GOC?
  • # of operational days versus planned days
  • # of operational days delivered vs. planned
  • Existence of Fleet Safety and security manuals
  • Existence of SLAs/MOUs
  • Opinions of key informants & survey respondents
Part of the 2nd evaluation focusing on Fleet Operational Capability
b.
Fleet Maintenance: The extent there was a reliable fleet that responded to the requirements of the GOC?
  • Condition rating for the fleet of large vessels remains within acceptable risk tolerance for reliability, availability and maintainability.
  • Condition rating for the fleet of small vessels remains with acceptable risk tolerance for reliability, availability and maintainability.
  • Opinions of key informants
checkmark   checkmark     checkmark  
c.
Fleet Procurement: The extent there was a modern fleet that responded to the operational needs and requirements of the Government of Canada?
  • Remaining Operational Life for Large Vessels
  • Remaining Operational Life for Small Vessels
  • Remaining Operational Life for Helicopters
  • Opinions of key informants and survey respondents
checkmark   checkmark     checkmark  
6. To what extent has the Fleet Operational Readiness Program achieved its strategic outcome?
a.
The extent the Government of Canada has access to an operationally ready fleet to facilitate its on water needs and priorities for safety, security, sovereignty, science, federal presence, and other maritime requirements?
  • Evidence to confirm that an operationally capable fleet is available that responds to the needs and requirements of the Government of Canada.
  • Opinions of key informants and survey respondents
checkmark   checkmark        
7.
Are there any external factors and/or challenges that may have impacted the results of the Fleet Operational Readiness Program?
  • Evidence that the Fleet Operational Readiness Program has established systems in place to monitor and mitigate internal/external risks
  • Evidence of internal/external factors or risks that have influenced the achievement of program outputs
  • Documented management actions to address the influence of internal/external factors or risks
checkmark   checkmark checkmark      
8.
Are there any unintended outcomes, positive or negative, that can be attributed to the Fleet Operational Readiness Program?
  • Identification and assessment of unintended positive and negative outcomes reported by staff, stakeholders and/or observed by the evaluator (high profile or recurring items or issues)
checkmark            
Efficiency & Economy
9.
To what extent is the governance and management structure clear and functioning adequately in order to support the achievement of intended outcomes?
  • Evidence that the governance and management structure are clear and functioning adequately
  • Opinions of key informants and survey respondents
checkmark   checkmark checkmark      
10.
To what extent do the activities of the Fleet Operational Readiness program complement, overlap or duplicate with other programs of DFO, or other Federal, Provincial and Territorial Governments?
  • Evidence that the programs are unique and or similar to other programs across Canada
  • Evidence that the programs complement other programs within DFO or across Canada
  • Perception of key informants regarding the program overlap/duplication with other programs
checkmark checkmark checkmark checkmark      
11.
To what extent could the efficiency / economy of the Fleet Operational Readiness Program be improved?
  • Breakdown of program budget between operating expenditures and program activities, if possible
  • Key informant views of delivery modifications to the program that would make it more cost-effective 
checkmark checkmark checkmark checkmark checkmark    
12.
Does the Fleet Operational Readiness Program demonstrate use of best practices and or lessons learned in the design and implementation of their activities?
  • Evidence that program performance is being monitored and is used for decision-making.
  • Evidence that program outputs have incorporated best practices and or lessons learned
checkmark checkmark checkmark checkmark      
13.
Is the Fleet Operational Readiness Program operating in a way that minimizes the use of resources to achieve their intended outcomes?
  • Evidence regarding the economy of the program compared to other similar programs (e.g. ratio of overhead cost to program expenditures; # of staff members employed to deliver the program).
  • Evidence of approaches that could result in more effective use of resources
checkmark checkmark checkmark checkmark