CANADIAN COAST GUARD COLLEGE EVALUATION REPORT
- Table of Contents
- Executive Summary
- 1. Introduction
- 1.1 Purpose of the Evaluation
- 1.2 Parameters
- 1.3 Report Structure
- 2. Program Profile
- 2.1 College Mandate
- 2.2 Program Activities
- 2.3 Program Stakeholders
- 2.4 Program Resources
- 2.5 Governance
- 2.6 Program Logic Model
- 3. Methodology
- 3.1 Scope
- 3.2 Evaluation Approach and Design
- 3.3 Key Issues and Evaluation Questions
- 3.4 Data Sources
- 4. Major Findings
- 4.1 Relevance
- 4.2 Effectiveness
- 4.3 Efficiency
- 4.4 Economy
- 5. Conclusions
- ANNEX A : Evaluation Matrix
- ANNEX B : Program Statistics at a Glance
- ANNEX C: Management Action Plan
This report presents the results of an evaluation of the Canadian Coast Guard College program. The evaluation assessed the College’s relevance and performance, the latter including effectiveness, efficiency and economy, in accordance with the Treasury Board of Canada’s Policy on Evaluation. The evaluation covers a five-year period from 2006-07 to 2010-11. The Department of Fisheries and Oceans’ Evaluation Directorate conducted the evaluation between April 2011 and May 2012. This is the first evaluation in keeping with Treasury Board’s Policy on Evaluation. Two previous in-depth reviews, undertaken in 2005 and 1996, looked at whether the College should be kept open and how it could operate more efficiently and economically. There was also a departmental assessment of the College operations in 2003.
The College is part of the Department’s Program Alignment Architecture aligned with the strategic outcome of Safe and Secure Waters. The College, in operation since 1965, became part of the Department in 1995 when the Canadian Coast Guard left Transport Canada to merge with Fisheries and Oceans Canada. For the next 10 years, the College reported to the Department’s Human Resources. This changed in 2005, when the College came into a direct reporting relationship with the Coast Guard.
The College develops and delivers marine training essential to Coast Guard operations. The training basically falls into one of two categories, the first being student training that produces graduates. The College’s Officer Training Program and its Maritime Communication and Traffic Services student training fall into this category. The second category includes all short-term specialized training courses, including any refresher courses for field staff. Marine Maintenance and Equipment Training, Search and Rescue training and Environmental Response training are in this category.
The College’s budget averaged about $11 million annually over the five-year evaluation period, and the College employed about 95 to 110 full- and part-time employees. The Officer Training Program represents the largest portion of the budget at 44%. Maritime Communication and Traffic Services student training accounts for 8%, and the specialized training courses are 12%. Administration and Operations, Campus Services and Galley comprise the remaining 36% of the budget.
Three Coast Guard directorates — Fleet, Maritime Services and Integrated Technical Services — are the College’s clients. Three other key stakeholders provide further direction and input on the College’s training: Transport Canada accredits the Officer Training Program (OTP); Cape Breton University ensures the Officer Training Program meets its academic requirements as it confers a Bachelor’s degree on Officer Training Program graduates; and Industry Canada sets requirements for part of the Maritime Communications and Traffic Services (MCTS) training.
The evaluation used a non-experimental design incorporating a logic model, multiple lines of evidence, and qualitative and quantitative data. This design is appropriate where experimental and quasi-experimental designs are contraindicated, due to being inappropriate, as in this case (cannot randomly assign people to training and control groups), and given that they are very time-consuming and costly to undertake. The College is a relatively small program, representing about 2.5% of the Coast Guard’s entire budget. This calls for a very efficient and modest evaluation design. The design employed met this requirement, while ensuring the evaluation was rigorous to arrive at valid findings and conclusions. Also, more attention was given at times to the College’s largest program, the Officer Training Program, and to a lesser extent MCTS, to keep the evaluation modest.
The evaluation drew on the following lines of evidence: program and related information review; literature review; administrative data; two comparative analyses; 20 interviews; and a survey (63 respondents). The lines of evidence posed a few limitations and challenges. None were prejudicial to the validity of the evaluation results as strategies were used to mitigate these limitations.
The literature review revealed there is no agreement as to how to measure the quality of post-secondary educational institutions. However, the evaluation incorporated four key indicators that were identified: type of learning (cognitive, attitudinal or skill-based); how learning is assessed, using the widely accepted Kirkpatrick model (participant feedback, tests and actual impact on the job); graduation rates, the one universally accepted key indicator of an educational institution’s overall performance; and administrative cost, typically used as a measure of financial performance. The evaluation was also informed by the previous reviews undertaken in 1996, 2003 and 2005.
Evaluation Findings and Recommendations
The College continues to be relevant and is aligned with federal responsibilities. It addresses a range of unique marine training needs essential to Fleet, Maritime Services and Integrated Technical Services. Fleet’s vessels, equipment and operations differ markedly from commercial operations such that its officer training needs differ. The Coast Guard has sole responsibility for maritime communications and vessel traffic services in Canadian waters, and it looks to the College to provide related training. Integrated Technical Services relies on the College to provide training in the maintenance and repair of communications equipment, systems and vessels, as this training is not available elsewhere. The Office of the Auditor General has called for greater consistency in Coast Guard operations. Having a facility that purposely serves the Coast Guard’s specialized and unique training requirements contributes to fostering consistency.
The Coast Guard has identified workforce issues as one of its priorities. Like the marine industry in Canada and worldwide, it faces severe shortages of marine engineers and navigators now and in the years ahead. Marine schools in Canada have a difficult time attracting students, probably due in part to a general negative perception of marine industry careers. This weighs in favour of the Coast Guard recruiting and training its own candidates. The Coast Guard also has a commitment to developing a representative workforce, which requires having training candidates from employment equity groups.
Provincial ministers of education do not perceive the education of Canadian Coast Guard personnel as falling under their jurisdiction (see: “The Federal Contribution” in a posting entitled Education in Canada: An Overview on the Council of Ministers of Education, Canada’s website, under “Canada-wide information”).
The College has been successful in achieving immediate outcomes, which are twofold: 1) knowledgeable, competent officers with demonstrated proficiency in their duties, and 2) employees who have acquired specialized knowledge. The College uses exams, practical exercises and simulations to assess learning and performance, as appropriate to each training program. As well, OTP cadets have two phases of at-sea training. OTP’s overall rigor stems from the requirements set by its three stakeholders: Transport Canada accredits the program; Fleet defines its unique requirements; and Cape Breton University ensures the necessary academic standards are met in order to confer a Bachelor’s degree.
The College’s graduation rates for its two officer student programs — 72% for OTP and 87% for MCTS — remained consistent with their rates for the period preceding this evaluation. These rates compare very favourably with a graduation rate of 69% for undergraduate programs. As noted previously, graduation rate is the one universally accepted key indicator of institutional performance.
The evaluation confirms that the College is realizing its intermediate and longer term outcome, specifically that College-trained personnel contribute to the delivery of Coast Guard programs. There is evidence that OTP graduates, having trained in Coast Guard operations, are at an advantage in being able to perform their duties immediately upon being hired by Fleet following graduation. Other new officers typically require a few to several months or even longer to familiarize themselves with Coast Guard vessels, equipment and procedures. Also, almost all survey respondents agreed that OTP graduates demonstrate leadership competencies necessary for advancement within the Coast Guard. It can be concluded that this is due to the College’s leadership training. Survey respondents also observed that, by virtue of the College’s training exceeding the basics required for officer certification, OTP graduates readily adapt to new challenges and technology.
Retention of OTP graduates beyond their mandatory service period provides another indicator with respect to long-term outcomes. According to a Coast Guard study undertaken in 2010, 60% of OTP graduates from 1990 to 2005 were employed with the Coast Guard or elsewhere in the Department at the time of the study. However, the evaluation was not able to obtain similar data on other ships’ officers or crew members or for the shipping industry in general for comparative purposes.
The return on investment for the federal government extends beyond the Coast Guard, as other federal departments seek to hire Coast Guard officers who are former OTP graduates. These officers have a reputation for discipline and analytical skills that began with their training at the College.
The Coast Guard directorates on the whole are satisfied with the training provided by the College. However, they and the College would benefit from feedback on the impact the training is having in the workplace. Post-training assessments would provide this. Employer satisfaction is considered an important part of assessing a post-secondary educational institution and adult training, yet it typically does not occur. The College and the Coast Guard have an ideal opportunity to ensure this type of assessment is done.
Recommendation 1 : The Commissioner should ensure the Coast Guard develops assessment tools, processes and an implementation plan to undertake post-training assessments of the Canadian Coast Guard’s various training programs.
The are a few persistent issues with no ready solutions: 1) it has proven difficult to attract instructors who can teach in both official languages; 2) rotation or some other means is required to ensure instructors remain current with field operations; 3) infrastructure support and life-cycle management continue to be pressing concerns; and 4) Fleet training needs, such as helping junior officers progress to higher levels of certification, exceed the College’s current capacity to deliver.
There is solid evidence that the College is diligent about operating efficiently. Previous reviews identified efficiency as the key component requiring improvement. Significant changes have now taken place to strengthen the College’s effectiveness and efficiency in achieving its outcomes. These changes began in 2005, when the College came into a direct reporting relationship with the Coast Guard. Its previous reporting relationship to the Department’s Human Resources had in effect isolated it from the rest of the Coast Guard.
The 2005 realignment led to an in-depth review of the College, culminating in several recommendations aimed at improving the College’s performance. The review resulted in a Transformation Plan, implemented over a three-year period from 2009-10 through 2011-12. A new governance structure and organizational changes have already contributed to improving the College’s effectiveness and efficiency. These changes also ensure that efficiency remains part of ongoing considerations. For example, business cases have replaced arbitrary, unilateral decision making. The full impact of these changes will become more evident going forward.
Roles and responsibilities between the College and the Coast Guard directorates are generally clear. A training governance model, being developed at the time of the evaluation, is expected to provide further clarification. However, the College and its clients would benefit from more direct and early communication with one another. Decisions about new assets or changes in the provision of Coast Guard services impact on the College. Likewise, the College may make decisions that impact on its need for guest instructors from the field or other Coast Guard support. There is insufficient discussion of proposed changes at an early stage.
With respect to recruiting and selecting OTP applicants, the College has a limited capacity to handle the intensive and heavy workload. Since 2007, when the College was given the responsibility for this work, the volume of applicants increased by 840% as of 2010-11, from about 285 initially to close to 2,700. As well, the Coast Guard has employment equity targets to meet, and the College can only effectively outreach to women and visible minorities in its immediate area. While Coast Guard regions provide some assistance, the process would be more effective and efficient, for both the College and the Coast Guard, if National Headquarters and the regions increased their participation.
The College provides an economical means of addressing various and unique Coast Guard marine training requirements. No options emerged as economical alternatives. The College’s programs are specific to Coast Guard needs, designed to provide training that to all intents and purposes other organizations do not or cannot provide.
Marine officer training at provincial marine schools and the College differ. While all students must meet the same Transport Canada requirements for officer certification, the Coast Guard has distinct academic and practical training requirements. The OTP course of studies addresses these, of necessity beginning in the first year. For example, OTP students take courses in the first year to prepare for their at-sea training on Coast Guard vessels in the second year. An analysis of university education costs indicates that the education cost per OTP student is in line with the cost of an undergraduate engineering program.
OTP graduates are required to provide four years of mandatory service, in return for coverage of their educational costs and related expenses. The Coast Guard has an obligation to Canadians to ensure that the mandatory service period is appropriate in length.
Recommendation 2 : The Commissioner should ensure the Coast Guard requires an appropriate length of mandatory service for OTP graduates, specifically whether four years or longer is sufficient, and determine whether other forms of reimbursement should be introduced when mandatory service cannot be fulfilled. Reasons in support of the chosen length of service and any other forms of reimbursement should be well documented.
Other training provided by the College is likewise specific to Coast Guard responsibilities and operations, such as MCTS, equipment and systems maintenance and repair, and maritime Search and Rescue.
The Canadian Coast Guard College develops and delivers marine training essential to the delivery of Coast Guard programs. Furthermore, its training programs do not replicate training available elsewhere, being specifically designed to meet unique requirements identified by the Coast Guard. The College’s ongoing relevance has been challenged and reaffirmed in reviews since at least 1996. Instead, the reviews found a need to improve governance and operations. The College has now addressed governance and made many other operational improvements. Still, a handful of resource issues persist that are largely beyond the College’s control.
This report presents the results of an evaluation of the Canadian Coast Guard College (the College). The evaluation is in accordance with the Treasury Board of Canada’s Policy on Evaluation, which requires all direct program spending to be evaluated every five years. The evaluation commenced in 2011-12, in accordance with the departmental evaluation plan, and was completed in May 2012. Recommendations stemming from the main findings are formulated to allow for improvements to the program where necessary and to inform future decision-making.
The evaluation focuses on the core issues of relevance and performance, the latter including effectiveness, efficiency and economy, as specified in Treasury Board’s Policy on Evaluation. It assesses whether there is a continuing need for the College and the extent to which the College has achieved its intended outcomes as outlined in the program’s logic model. The evaluation covers a five-year period from 2006-07 to 2010-11.
For those not familiar with the College, the next section provides a brief overview of the College’s mandate, activities and so forth. Section 3 describes the evaluation methodology, followed by a discussion of the main findings in section 4 and conclusions in section 5. Annex A presents the key issues, questions, indicators and data sources used for the evaluation.
The following sections may be read for a brief account of the evaluation and the results:
- executive summary – brief overview of the evaluation, the findings and the recommendations
- conclusions – section 5 provides a succinct summary of the overall conclusions.
Section 4 on the main findings is structured to facilitate scanning. Each subsection (relevance, effectiveness, efficiency and economy) begins with a short description of what was examined and very briefly states the overall finding. Key findings are highlighted, and subheadings are used in the evidence sections to further guide readers. Evidence summaries are included where there are several findings relevant to a given evaluation question.
The purpose of the College is to develop and deliver training, or oversee delivery, that supports Canadian Coast Guard-mandated programs in maritime safety, security and environmental protection. The type of training required and its significance are reflected in the following brief summaries of Canadian Coast Guard (CCG) programs and activity:
- Aids to Navigation — CCG maintains over 17,000 aids to navigation, which include lighthouses, buoys and the Differential Global Positioning System, among others.
- Icebreaking — Canada has more floating ice on its oceans and lakes than any other nation. CCG Fleet handles about 1,500 requests a year for icebreaking support.
- Waterways Management Services — Commercial shipping and other vessel operators rely on CCG services to ensure safe and efficient use of Canada’s commercial waterways.
- Marine Communications and Traffic Services — Twenty-two centres provide marine and safety communications, screen vessels entering Canadian waters, regulate vessel traffic and respond to marine distress calls.
- Maritime Security — CCG participates in three Marine Security Operations Centres, monitoring approximately 1,000 vessels daily operating within 2,000 nautical miles of Canada’s shores. CCG also operates vessels used by the Royal Canadian Mounted Police for security enforcement on the Great Lakes–St. Lawrence Seaway.
- Search and Rescue (SAR) — CCG works with the Department of National Defence in delivering the federal government’s SAR program. CCG is responsible for the maritime component and co-ordinates about 75% of all SAR responses, responding to more than 6,000 maritime incidents annually.
- Environmental Response — If a polluter in Canadian waters is unknown, unwilling or unable to respond to an incident, CCG becomes the On-Scene Commander managing the clean-up.
The College’s training encompasses the following components:
- CCG Officer Training Program (OTP) — This is a four-year program in which officer cadets specialize in either navigation or marine engineering. Graduates receive professional certification from Transport Canada and a Bachelor of Technology in Nautical Sciences from Cape Breton University.
- Marine Communications and Traffic Services (MCTS) training — The College delivers a 25-week basic training program for MCTS students and refresher training for MCTS officers. MCTS training is also provided to OTP cadets, and there is a three-week vessel traffic services course for Coast Guard personnel who monitor and organize maritime traffic.
- Marine Maintenance and Equipment Training (MMET) — The Coast Guard provides a 27- to 36-month Marine Electronic Development (MELDEV) training program to bring new electronic technology recruits to a working level specific to Coast Guard needs. Experienced CCG technologists also require training on new equipment and systems and refresher courses.
- Rescue, Safety and Environmental Response training — Both DND and CCG personnel receive SAR training at the College, and environmental response training is provided to CCG and industry personnel either at the College or in the regions.
Three Coast Guard directorates — Fleet, Maritime Services and Integrated Technical Services — are the College’s clients. Other DFO programs and National Defence benefit from the College’s facilities, expertise and training. Transport Canada, Cape Breton University and Industry Canada are among the College’s major external stakeholders. Transport Canada accredits marine schools in Canada. It sets curriculum and quality assurance requirements for navigation and marine engineering training, administers certification testing and issues officer certification. Cape Breton University, under an agreement with the College, confers a Bachelor of Technology in Nautical Sciences degree to OTP graduates. This arrangement dates back to June 1995. Industry Canada sets certification standards and accredits institutions for Global Maritime Distress and Safety System training, a component of the College’s MCTS training. It is the issuing authority for Radio Operations Licences, which it has delegated to the College.
Various industries depend on Coast Guard services, such as commercial shipping, fishing, ferry and tourism operators, and oil and gas offshore industries. Likewise, marine researchers, other federal departments, recreational boaters and the Canadian public at large all depend on Coast Guard services, which in turn rely on training provided by the College.
The College includes specialized training facilities, equipment and simulation environments pertinent to field operations. It has a residential facility, with the capacity to house and train up to 250 personnel concurrently, on-site recreational and sports facilities, and a library housing the most extensive marine interest collection in the country. Public access to the College’s physical training facilities is permitted. This was a condition of Treasury Board’s approval in 1982 for construction of these facilities. The College employed about 95 to 110 full- and part-time employees between 2006-07 and 2010-11. As well, it had 77 to 153 full-time equivalent OTP cadets annually. The College’s expenditures are presented in Table 1. Building Management is shown separately, below the annual totals, as responsibility for this was returned to DFO’s Real Property Gulf Region effective April 1, 2009.
|Salaries||5,370.30||5,974.20 1||6,712.50||7,781.20 2||7,430.10|
|Other O & M||4,182.10||4.451.90||4,586.90||4,090.70||4,435.10|
|Building Management 3||3,786.70||5,580.10||5,217.70||347.70||0 $|
1College did not cover cadet salaries prior to 2007-08 (previously covered by Fleet).
2 Transformation Plan called for an increase in staff resources.
3Responsibility transferred to Real Property, Asset Management Services 2009.
The College was established in 1965, three years after the creation of the Canadian Coast Guard. Responsibility for the Coast Guard and the College transferred from Transport Canada to DFO in 1995. From then until 2005, the College reported to DFO’s Director General, Human Resources. In 2003, CCG became a line organization within DFO, and the College subsequently came under the direct control of the Commissioner of the Coast Guard in 2005. As of September 2010, the College’s Executive Director reports to the Deputy Commissioner of Operations, who in turn reports to the Commissioner of the Canadian Coast Guard.
The logic model shown in figure 1 was developed for this evaluation.
Figure 1. Canadian Coast Guard College Logic Model
The evaluation was undertaken by the Department’s Evaluation Directorate between April 2011 and May 2012. The evaluation involved contacts at the College, National Headquarters and external to Fisheries and Oceans Canada, as well as operational staff in five Canadian Coast Guard regions: Newfoundland and Labrador; Maritimes; Quebec; Central and Arctic; and Pacific. College staff assisted throughout. In addition to participating in interviews, they provided input on documents to review, other key informants and stakeholders to contact, statistical data and feedback on such documents as the evaluation plan, a survey and the evaluation report.
An overall challenge for the evaluation was the scope of the program. The College is an institution in itself, with accommodation and training facilities, governance issues, financial systems and so forth. More than the training programs alone need to be assessed for a rigorous evaluation. Furthermore, the College offers two distinct streams of training: one produces graduates, and the other is comprised of adult training courses. Although the scope is complex, the College is a relatively small program within the Department. Therefore, the evaluation needed to be conducted in an efficient manner relative to program size while being sufficiently robust to investigate the complexities. The lines of evidence were chosen accordingly, plus more attention was given to OTP as it represents 44% of the College’s total budget.
The College embarked on a major, three-year transformation in 2009-10, to be completed in 2011-12. The evaluation therefore assessed a program in a state of flux rather than one that is well established in all of its processes. The full scope of the changes cannot be captured in this evaluation, as any organization undergoing significant changes will have a “settling in” period beyond implementation.
The evaluation used a logic model approach combined with multiple lines of evidence drawing on both qualitative (e.g., interviews, program documents) and quantitative data (e.g., administrative data, survey). Some of the qualitative and quantitative input made it possible to draw a few comparisons between College-trained and non-College-trained officers. In essence, this is a non-equivalent group design. The various sources of input provided the basis for corroborating the evidence (triangulation) to arrive at valid findings and conclusions. Neither an experimental nor quasi-experimental design was chosen, as they are contra-indicated for this program. Not only would it be impractical, given that individuals cannot be randomly assigned to receive or not receive training, but the relatively small size of the program indicated a need for an efficient and modest evaluation design, precluding a considerably more time-consuming and costly approach inherent with quasi-experimental and experimental designs.
The evaluation matrix in Annex A presents the key issues and evaluation questions addressed, as well as the lines of evidence used for each question. The questions were established in keeping with Treasury Board’s Policy on Evaluation, a review of key program documents and an evaluation planning session with key program personnel.
The evaluation drew on the following lines of evidence:
- An information review covered material from the College, Coast Guard, DFO and other federal government sources, and the International Maritime Organization’s website.
- A literature review on assessing the quality of post-secondary educational institutions and adult training courses contributed to developing the evaluation’s methodology and confirming some of the information to be gathered.
- The evaluation used administrative data from the College, Fleet, Marine Services and Integrated Technical Services.
- The evaluation included two comparative analyses, one comparing the College’s OTP with other marine schools and the other comparing educational program expenditures.
- Interviews were held with 20 key informants: 10 College personnel, eight CCG national staff (Fleet, Maritime Services, ITS), and two marine sector representatives external to DFO. The majority of interviews were in person, with only two by phone. Interviews with College personnel were conducted as part of a site visit, which also afforded the opportunity to tour the facilities.
- A survey was undertaken with selected Fleet personnel to assist in evaluating OTP outcomes, with 63 responses received: 36 Commanding Officers, 24 Chief Engineers and three Maritime Superintendents. The survey had an estimated response rate of 50%, based on 125 invitations sent (some e-mail addresses were for more than one respondent).
The interviews and survey afforded insight into the perceptions, opinions and experiences of key individuals involved directly or indirectly with the College. Both provided qualitative and quantitative input. The survey contributed to assessing employer satisfaction with OTP graduates, a post-secondary education performance indicator not usually assessed according to the literature review. The terms in the following table have been used in some instances to report the proportion or frequency of responses in place of numerical data.
|Percentage of Respondents||Proportion Terms||Frequency Terms|
|80-99 %||Almost all||Almost always|
|50-79 %||Many||Often, usually|
|1-9 %||Almost none||Almost never|
With a non-experimental design, rigour is achieved by using multiple lines of evidence and drawing on both qualitative and quantitative data. The logic model added further rigour to the evaluation by clearly differentiating between the College’s two main streams of training (officer training and specialized training courses) and by identifying the immediate and long-term intended outcomes to be used in evaluating the results. As well, the evaluation was strengthened by obtaining comparative statistics. This input contributed to assessing the College’s OTP and MCTS training results. In summary, the various lines of input provided the basis for corroborating evidence (triangulation) to arrive at valid findings and conclusions about the College’s contribution to outcomes.
Considerable care must be exercised in selecting other organizations, training and related data for comparative purposes to ensure comparisons are appropriate. For example, the OTP may be compared with provincial marine institutes on some accounts but not all, given that the OTP is a four-year university program whereas navigation and marine engineering training at provincial marine schools leads to a diploma. Only one of the provincial schools offers a Bachelor’s degree as an option. Where necessary, caveats have been noted in the comparative analyses.
Over half the respondents were Commanding Officers, and most responses were from Eastern Canada (Quebec and East Coast regions). To determine if the results were biased in favour of either of these factors or both, the results for a) Chief Engineers and b) Central and Arctic and Pacific regions were checked against the overall responses for any statistically significant differences using t-tests. No significant differences emerged for Chief Engineers at the 95% confidence level, but there were a few for the Central and Arctic and Pacific regions, which are reported in the findings. The survey results were also considered in tandem with input from the other lines of evidence to arrive at valid findings and conclusions.
The evaluation examined whether the Canadian Coast Guard College meets an essential need and serves unique training requirements, whether it is aligned with departmental strategic outcomes and federal priorities, and whether its mandate is in keeping with federal jurisdiction. The evidence indicates the College satisfies these requirements.
Key Findings on Continuing Need: The College addresses essential marine training requirements unique to the Canadian Coast Guard. Fleet faces a serious shortage of ships’ officers and has increased its demand for OTP graduates. The Coast Guard’s continuing evolution and changes in the marine industry, such as rapid technological changes, indicate a continuing need, if not increasing, for the College’s training programs.
Unique Training Requirements
The College is the focal point for providing highly specialized marine training in support of Coast Guard programs. It addresses a range of unique training requirements for all three Coast Guard directorates: Fleet, Maritime Services and Integrated Technical Services.
Fleet vessels and operations differ markedly from commercial operations. For example, its vessels differ in design and function from tankers, cargo ships, cruise ships and other commercial vessels. Coast Guard vessels have smaller, diesel electric engines, which allow for close proximity work not possible with direct drive engines used in commercial vessels. CCG navigators must be trained to work in conditions in which other vessels do not operate, such as in shallow waters, and for unique functions (e.g., setting buoys, icebreaking). The Coast Guard has equipment and systems that are unique to its operations. No one other than Maritime Services requires Marine Communication and Traffic Services training, as the Coast Guard is the sole provider of these services. Integrated Technical Services has a need for specialized training as the Coast Guard deals heavily in communications. Community colleges focus on meeting mainstream requirements, training technicians who are strong in electronics but not in communications technology.
The evidence indicates that training for Fleet differs from the training provided at provincial marine schools, which focuses on commercial operations. Graduates from any marine school, including the College, must meet the same Transport Canada requirements to receive officer certification. While some of the training is the same across all marine schools, the College differs in focus, curricula, and simulator and practical training. The requirements that are unique to Fleet are grouped into six areas:
- Certification Requirements — OTP cadets complete exams required for more senior ranks, up to Master (navigation) and 1st Class (marine engineering) levels, as part of their four-year program
- Discipline Skills — examples include CCG operations (icebreaking, SAR, dynamic positioning, buoy tending), steering in heavy weather, CCG towing policy, helicopter operations, salvage techniques in SAR operations, small boat maintenance, etc.
- General Knowledge — Fleet Safety and Security System, administrative procedures, knowledge of DFO
- Language and Communication — bilingual skills
- Leadership Skills
- Physical Fitness.
Provincial marine schools do not prepare students from the outset for future senior rank and leadership positions. The focus is on training students to meet the basic requirements for entry-level positions. There is no need to ensure that graduates meet a bilingual profile or any physical fitness requirements. Their three- or four-year programs lead to a diploma, with the exception of one school which offers an optional fourth year leading to a Bachelor’s degree. Otherwise, the College is the only Canadian marine school whose navigation and marine engineering training leads to a Bachelor’s degree. For the Coast Guard, a Bachelor’s degree is an important credential for its more senior-ranking and executive positions.
The Coast Guard has sole responsibility for marine communications and vessel traffic services in Canada. As such, no one else in Canada has a demand for MCTS training and the equipment needed for it, specifically a simulator. MCTS officers regulate traffic movements, broadcast safety information, coordinate communication between ships and others, and they are responsible for detecting distress situations and ensuring a response.
The College provides specialized equipment training through its MMET courses. Coast Guard technical specialists maintain and repair a wide range of equipment and systems. They also need to be trained in checking equipment installations. Sea-going technicians need to be trained in doing maintenance at sea, which is not taught at community colleges. As new systems and technology are introduced, experienced technicians need to update their knowledge and skills, and the necessary training typically is not available elsewhere. The marine industry continues to face rapid technological changes.
National Defence and the College jointly deliver a SAR Mission Coordinator Course. The College, augmented by National Defence, delivers a Maritime Search Planning Course and a SAR Mobile Facilities/On-Scene Coordinator Course. As noted previously, the Coast Guard is responsible for maritime SAR delivery, and the majority of all SAR responses are for maritime incidents.
In addition to the above evidence, previous reviews attest to the College’s fundamental role in supporting the delivery of Coast Guard services and programs. The College was considered for closure for budgetary purposes in a 1996 review by Public Works and Government Services Canada and the Department. However, the review concluded the College should be kept. The College needed to operate more efficiently and economically, for example by reducing overhead and increasing the use of its facilities. Financial and management systems in particular were found to be inadequate, and the College needed a clear statement of purpose. Failing progress in these areas, the possibility of closure was to be revisited in three to four years. According to a 2005 Review of the Canadian Coast Guard College, undertaken on behalf of the Coast Guard, all of the main recommendations from the 1996 review were acted upon to varying degrees. Some were not fully implemented and time changed a number of the deliverables.
A 2003 departmental assessment concluded the College was needed, as did the 2005 review which was charged with considering various options for the College, including closure. The key reasons for keeping the College according to the 2005 review were as follows: the College addresses many different specialized training needs of the Coast Guard, not just training for marine officer certification; provincial marine schools have difficulty attracting students; the Coast Guard is obligated to have a representative workforce, which requires targeted recruitment and training efforts; the Coast Guard’s rapid and substantial evolution creates a major impact on its training needs; and the Coast Guard is evolving into a national institution. The review recommended that the College be kept and renewed for these reasons. The same reasons are still valid today.
The Office of the Auditor General in a number of reports has called for much greater consistency in Coast Guard operations and its training, and the need for the Coast Guard to function as one national institution. This can only be enhanced by having a facility that purposely serves the Coast Guard’s very specialized and unique training requirements.
Addressing workforce issues is one of the Coast Guard’s key priorities as indicated in its three-year Business Plan documents. These plans identify the key risks and strategic challenges the Coast Guard faces, and the priorities indicate how the Coast Guard plans to respond to them. Since 2008, the Coast Guard has had a companion Strategic Human Resources Plan document that details the issues and the initiatives to address them.
The Coast Guard faces significant staffing needs due to the pending retirement of many employees. The Strategic Human Resources Plan documents identify five occupational groups essential to Coast Guard operations that are particularly at risk of attrition. As of 2008, about 28% of employees in these groups were eligible to retire over the next five years. These include two groups for which the College provides training and that continue to be of concern, namely Ships’ Officers, Radio Operations (MCTS Officers) and Electronics (marine technologists).
The Coast Guard also has a serious shortage of 3rd Class marine engineers. At the same time, the Coast Guard is being asked to fulfil an increasing role in meeting federal priorities, such as a stronger national presence in the North. Currently, the Coast Guard is in the process of acquiring five new vessels, which will require approximately 40 more ships’ officers. The addition of new vessels also has implications for MMET, as technical personnel must be up to date with new systems and equipment to ensure proper maintenance and repairs.
To help address some of these challenges, Fleet recently increased the annual target intake for OTP cadets by a third, from 48 (September 2006-09) to 64 (September 2010). The College responded by increasing actual intake from an annual average of 39 for 2006-09 to 57 in September 2010, a 46% increase.
While the College is the sole source of ships’ officers trained specifically for Coast Guard operations, there are two other sources of ships’ officers: external hires, and ships’ crew who become ships’ officers. For the period evaluated, the Coast Guard hired altogether close to 300 new ships’ officers from all three sources. Each source accounted for about a third of the hires. The 300 excludes term and casual hires, but includes seasonal staff and mid-level officers from the latter two sources in addition to indeterminate staff and junior officers. All OTP graduates are hired as indeterminate staff and junior officers. As separate numbers for new seasonal employees or those hired at more advanced ranks were not available, it was not possible to determine the actual extent to which the College is a source of indeterminate, junior officers compared to the other two sources. However, if we were able to exclude seasonal and mid-level hires from the 300, then the College would prove to be a greater source of those hired over the five-year evaluation period.
It would be advantageous for future evaluations if the Coast Guard could readily provide statistics on the number of indeterminate, junior officers hired from each source, as well as the number of indeterminate staff at mid-level or higher ranks hired externally or from ships’ crew.
The Marine Industry at Large
According to a 2008 federal study, the marine industry in Canada faces acute difficulties in attracting, training and retaining a skilled workforce. There are alarming shortages for officers, with the need for marine engineers being the number one priority, followed closely by deck officers. This shortage is thought to be due in part to a negative perception of marine industry careers.
Although the global economy has faced severe challenges of late, marine activity is expected to increase over the long term. When the world economy is strong, it becomes harder for the Coast Guard to recruit and retain staff, as officers with shipping companies earn considerably more than Coast Guard officers, possibly double the income. Graduates from provincial marine schools are more likely to opt for the private sector given its better compensation.
These factors weigh in favour of the Coast Guard recruiting and training students to become Fleet officers.
In short, the College provides training that is both unique and essential to Coast Guard operations. Previous reviews support this finding as they likewise have confirmed a need for the College. Through its training programs, the College is ideally positioned to help the Coast Guard achieve greater consistency as called for by the Auditor General. There is an ongoing need for the College’s training programs over the long term.
Key Findings on Alignment with Federal Priorities: The College is well aligned with the Department’s strategic outcomes and federal priorities. Furthermore, it contributes to meeting federal obligations required under international conventions.
Alignment with DFO and Federal Priorities
The College provides training that supports the Coast Guard’s mandate in marine safety and environmental protection and its participation in marine security activities. The College is directly aligned with the Department’s strategic outcome of Safe and Secure Waters, and by virtue of the training it provides, it supports the Department’s other strategic outcomes. For example, OTP cadets receive training in setting and retrieving buoys, which is part of the Coast Guard’s Marine Navigation program under the strategic outcome of Economically Prosperous Maritime Sectors and Fisheries. Environmental Response training and the use of Fleet vessels for marine research, which depends on having trained ships’ officers, contribute to the strategic outcome of Sustainable Aquatic Ecosystems.
As indicated in the Department’s Report on Plans and Priorities, its three strategic outcomes contribute to Government of Canada outcomes, such as a safe and secure Canada and national prosperity through global commerce (shipping handles more than 90% of global trade). Furthermore, Speeches from the Throne between 2007 and 2011 have reiterated the need for a federal role and presence in Canada’s North. As noted by an external stakeholder, the Coast Guard is Canada’s visible presence in the North. The College also contributes to other federal organizations, such as National Defence, in fulfilling their mandates.
The College contributes to implementing Canadian international obligations by providing training as required under international conventions ratified by Canada. The United Nations’ International Maritime Organization (IMO) is responsible for safety of navigation, protection of the marine environment from shipping activities and marine security. International conventions adopted under the auspices of the IMO and ratified by Canada set the parameters for national legislation and related regulations.
The IMO’s treaty Standards of Training, Certification and Watchkeeping for Seafarers (STCW) prescribes the minimum standards that must be met or exceeded. STCW came into force in 1984, with major revisions in 1995 and 2010. Some of the most recent changes, which came into force January 1, 2012, address new training and guidance requirements relating to modern technology, marine environment awareness, leadership and teamwork, and personnel serving on ships operating in polar waters. As a signatory to the STCW, Canada must have a system in place for training and certifying Canadian seafarers. Transport Canada is responsible for accrediting Canadian marine schools, including the College, and certifying officers. Schedules 1 and 2 of the Canada Shipping Act, 2001 (S.C. 2001, c. 26) list various international conventions, with implementation assigned respectively to the Minister of Transport and the Minister of Fisheries and Oceans. STCW is listed in Schedule 1.
The International Association of Marine Aids to Navigation and Lighthouse Authorities recommends to its members, Canada being one, to have marine traffic services, standard communication systems and related training.
The College, its clients and Transport Canada all contribute to ensuring the College remains aligned with federal priorities and meets the required standards.
The Constitution Act, 1867 provides the Parliament of Canada with exclusive legislative jurisdiction over navigation, shipping, beacons, buoys and lighthouses. Pursuant to this jurisdiction, Parliament enacted the Oceans Act which provides for the Coast Guard’s mandate and the powers, duties and functions of the Minister of Fisheries and Oceans, including those as the minister responsible for the Canadian Coast Guard. The Coast Guard provides services in support of safe, economical and efficient movement of ships in Canadian waters, such as aids to navigation, marine communications and traffic management services, icebreaking services and channel maintenance. Other services include the maritime component of the federal search and rescue program, marine pollution response and support to other federal departments, boards and agencies through the provision of vessels, aircraft and other marine services. The Coast Guard meets most of these responsibilities through its Fleet and Maritime Communications and Traffic Services Centres, which rely on the College to provide essential training to their officers, as well as to Canadian Coast Guard technologists.
Provincial ministers of education do not perceive the education of Canadian Coast Guard personnel as being within their jurisdiction (see: “The Federal Contribution” in a posting entitled Education in Canada: An Overview on the Council of Ministers of Education, Canada’s website, under “Canada-wide information”).
The evaluation looked for evidence that demonstrates the College is contributing to its immediate and longer term outcomes. With respect to immediate outcomes, the evaluation asked what evidence is there that students have acquired the necessary competency, knowledge of Coast Guard operations and proficiency to graduate, and what evidence is there that training course participants have acquired specialized knowledge. How the College assesses learning is central to answering this. With respect to longer term outcomes, the evaluation looked for evidence that College-trained personnel contribute to delivery of Coast Guard operations. The evaluation focused on OTP for the most part, as it is the College’s largest program, accounting for 44% of the College’s budget, and some evidence was more readily available for this training. However, the evaluation did obtain some input relevant to the College’s other training programs. Overall, the evidence indicates the College is effective in achieving immediate and longer term outcomes.
College Assessment Comprehensive and Rigorous
The criteria for evaluating the College’s approach to assessing immediate outcomes are based in the literature review undertaken for the evaluation. Formal testing (e.g., exams, tests, practical exercises and simulations) assesses learning in cognitive abilities, attitude or skills. Learning can also be assessed by different means. The most basic is trainee feedback, with participants sharing their thoughts about the training. Formal testing is more rigorous, while an assessment of the impact of training on the job would be the most definitive in assessing training results. Post-secondary education is typically assessed by formal means such as exams. In contrast, learning from adult training courses is often based only on what participants thought about the training. The impact of training on job performance is rarely assessed. Using this criteria, the College proved to be quite comprehensive and rigorous in its assessment of competency, knowledge and skills, as reflected in Table 3.
|Training Program||Changes Assessed||How Assessed|
|Cognitive||Attitude||Skill||Trainee Feedback||Exams, Testing, etc.||On-the-Job Performance|
Both OTP cadets and MCTS students write exams and are tested using simulators. OTP cadets have two at-sea training phases, 60 weeks in total for navigation and 34 for marine engineering. The Coast Guard’s Officer Code of Discipline, drill and parade training, and leadership and shipboard discipline courses are examples of training content designed to instil in officer cadets the Coast Guard’s culture and ethos of Safety First, Service Always. SAR exercises are as realistic as possible, with Joint Rescue Command Centre personnel and local media and emergency services participating. Both programs have exit interviews with students, and MCTS students complete a survey as they proceed through their training.
OTP cadets have 40 hours of courses per week, plus study time, and they are expected to be involved in community activities on the weekends. Navigation students must achieve 60% to 90% in their studies, depending on the specific course. Marine engineering students must achieve 60% in all of their courses. MCTS students are required to achieve at least 80% in their exams and simulations. Between 2006-07 and 2010-11, the College graduated 95 OTP cadets and 108 MCTS students. All OTP graduates obtained their Transport Canada officer certification. Data on OTP grade point averages were not readily available, due to cumbersome record keeping in the past. The College was in the process of developing an improved system for capturing student data at the time of the evaluation. The graduating class average for MCTS for the five-year period was 93%.
Training requirements for marine schools have become more stringent. Transport Canada has a new requirement for instructor training as part of a marine school’s quality assurance. The College and Cape Breton University together have developed training for College instructors and university professors. The objective is to help them acquire the necessary teaching skills to become good instructors.
The College undertook a major OTP curriculum review in 2010-11 in consultation with Fleet, Transport Canada and Cape Breton University. The review looked at quality management, academic rules and Fleet’s competency requirements distinct from those required for Transport Canada certification. Nine documents comprise the College’s OTP Course of Studies detailing the various requirements that must be met (e.g., pass marks, attendance criteria), course descriptions and course content.
All MMET courses have written and practical exams, except one which requires only a written exam. A practical exam involves three test scenarios, each assessing the trainee’s ability to successfully find the fault, use proper fault finding procedures and successfully deal with the problem. Any safety oversight (e.g., not taking off all rings and watch) is an automatic failure. Trainees must pass two of the three test scenarios. Trainees complete feedback forms at the end of a course. The Memorandum of Understanding for this training indicates that the College may conduct post-training assessments with course participants and supervisors after trainees have been back at their jobs for a while. However, no post-training assessments of this nature have been undertaken.
SAR trainees participate in a full simulation exercise and trainees are required to pass an exam at the end of each course, as they are will be Mission Controllers at Joint Rescue Command Centres. Trainees are constantly monitored during the simulation to see how they react under pressure.
Environmental Response trainees likewise have simulations, exercises and practical assessments, plus the instructor observes the functioning of the team as a whole. There is a written exam which leads to the successful trainee being designated as a Pollution Response Officer by the Minister of Fisheries and Oceans, in accordance with the Canada Shipping Act, 2001. That Act provides Pollution Response Officers with broad powers with respect to marine pollution or a threat of marine pollution.
Table 4 presents a summary of the training output statistics for the evaluation period.
|OTP Graduates||MCTS Graduates||MMET Training Seats 1||SAR||ER4|
1ITS counts training seats not individuals as an individual may take several courses during a year.
2Excludes OTP cadets.
3National Defence personnel were the only other federal employees trained by College personnel.
4CCG personnel and industry.
There is no agreement among institutions, governments and other stakeholders as to how to define or measure the quality of post-secondary educational institutions. However, an institution’s graduation rate appears to be the one universally accepted key indicator of institutional performance.
For OTP, 72% of students from the 2002 through 2006 intake years graduated between June 2006 and June 2010. This is consistent with the graduation rate for the preceding four years (no graduating class in 2001 as there was no intake in 1997). MCTS had a graduation rate of 87% over the five-year evaluation period, which is consistent with the preceding five years. The College had no graduation target rates for the period evaluated but it introduced targets in 2011-12: 70% for OTP, and 90% for MCTS. The evaluation results confirm the targets are reasonable.
The College’s OTP graduation rate compares very favourably with those from a federal government longitudinal Youth in Transition Survey. The study, which has followed a large sample of youth since 1999, collected data in 2008 that shows the youths’ graduation rate from undergraduate programs as being 69%. The results are based on 12,360 responses.
The evidence presented indicates that the College graduates students who are competent, knowledgeable about Coast Guard operations and programs, and have demonstrated their proficiency to proceed either as Fleet officers or with field training at MCTS centres. Fleet, Transport Canada and Cape Breton University requirements all contribute to the rigor of the College’s largest training program, OTP. Both OTP and MCTS students must demonstrate their learning and abilities in a number of ways. The College’s graduation rates are a further indication of the College’s effectiveness, as is the fact that 100% of OTP graduates obtain Transport Canada officer certification. Finally, the evidence indicates that the College is as diligent in assessing the immediate outcome of its training courses. Participants must demonstrate they have acquired specialized knowledge through tests, exercises or simulations.
Key Finding on Intermediate and Long-Term Outcomes: College-trained personnel contribute to the delivery of Coast Guard programs and more.
Contribution to Delivering Coast Guard Programs
OTP graduates must serve a minimum of four years with the Coast Guard Fleet in exchange for their training. For the evaluation period 2006-07 through 2010-11, Fleet hired all 95 OTP graduates. All graduates join Fleet at the entry level for officers, as they do not have the necessary sea time yet to qualify for higher ranks.
OTP cadets are trained in conditions unique to Coast Guard operations, such as navigating in shoal waters against currents, tide and wind. Officers trained elsewhere do not have this training, nor are they familiar with operating open barges, fast rescue craft and other Coast Guard vessels. Due to OTP’s specific focus on Coast Guard operations, graduates are at an advantage in being ready to perform their duties when hired. Other new officers require a period of familiarization according to many of those surveyed and some interviewees other than College staff. Of those surveyed, 58% indicated that non-OTP officers require four to 12 months for familiarization, and a further 15% indicated the process can take one to three years or more (n=53).
Survey respondents also noted other outcomes of the College’s training. Almost all agree that OTP graduates demonstrate leadership competencies necessary to advance within the Coast Guard (n=62). As well, OTP graduates are seen as adapting to new challenges and technology by virtue of the College’s academic training exceeding the basics required for officer certification.
The College is the only source of MCTS officer recruits. That is, the MCTS centres would not be able to operate without the training delivered by the College. Of 108 MCTS students graduating between 2006-07 and 2010-11, 101 (94%) passed their on-the-job training. Of these, 98 were with DFO as of December 2011. Almost all, 95, were with MCTS centres, and the other three were employed elsewhere within the Department.
On the whole, all three Coast Guard directorates are satisfied with the training provided by the College. If a training course or component is not being provided, it is because the Coast Guard has not asked for it.
Concerns from the Field
Almost all survey respondents perceived gaps in the training OTP cadets receive. Most would like to see more practical, hands-on training, such as in machine shop, refrigeration, boilers, hydraulics and electronics, and with barges and workboats. Some mentioned health and safety training, in particular respiratory protection fit-testing. Knowledge of various codes, systems and administration could be stronger, some examples being International Safety Management, security training, iFleet, collective agreements and administration manuals.
Cadets would also benefit from more emphasis on communication skills (writing, reporting and public speaking). As well, leadership and management skills need some more attention to ensure graduates understand the dynamics of interacting with and supervising experienced crew members. Finally, some OTP graduates have unrealistic expectations. For example, some expect advancement to happen quickly. Some lack a commitment to the basic maintenance work that needs to be done, such as cleaning and painting.
As mentioned previously, it is anticipated that improvements will be evident going forward.
Retention of Graduates
Retention of graduates, although beyond the College’s control, is an appropriate indicator with respect to intermediate and long-term outcomes. According to a study undertaken by Fleet in 2010, 60% of the 1990 to 2005 OTP graduates were employed with the Coast Guard or the Department as of summer 2010. This excludes those still serving their mandatory service. The College graduated 12 classes from 1990 to 2005. Seven had retention rates above 50%, averaging 74% overall, and five had retention rates of 31 to 50%, averaging 41%. The evaluation was not able to obtain comparable data on the retention of other Coast Guard officers or ships’ crew, or Canadian mariners in general, for comparison.
Retention of College graduates is an important indicator of the long-term impact and return on investment for the government. For future evaluations, it would be beneficial to have retention data as reported above. To facilitate this, it would be advantageous for the College to have an alumni registry of OTP and MCTS graduates that is kept up to date. It would also be useful for future evaluations if the Coast Guard could provide data on the retention of non-OTP officers for comparison with retention of OTP graduates.
Other Benefits of the College
Training at the College provides networking opportunities for students and other Coast Guard personnel that otherwise would not be possible. OTP and MCTS students benefit from cross-training opportunities between the two programs. Each group gains an appreciation of the other’s perspective and role. Opportunities are created for OTP cadets to meet visiting Coast Guard officers. Employees on training courses are able to network and share their best practices. This also fosters a network of contacts to call on in the future.
According to interviewees, other federal departments, public utilities and industry seek to hire Coast Guard personnel who are former OTP graduates. These employees’ reputation for being very organized, competent and capable in troubleshooting and systematic analyses begins with their training at the College. As well, small vessels comprise 80% of all vessels in Canada, and Coast Guard officers have expertise in this area. Transport Canada, the Transportation Safety Board and Public Works and Government Services Canada have all recruited Coast Guard employees who were former OTP graduates. Likewise, public utilities and maritime companies have hired Coast Guard personnel who trained at the College. For example, the North American head office of Lloyd’s of London, one of the world’s largest ship underwriters, is headed by a College graduate. The alumni registry mentioned above could capture information on College graduates who are working elsewhere in the federal government (numbers and organizations), further illustrating the government’s return on investment in the College, as well as those who are working elsewhere.
Marine Security Operations Centres benefit over the long term from the College’s MCTS training program. The Department and the Coast Guard participate in operating the three centres with National Defence, the Royal Canadian Mounted Police, the Canada Border Service Agency and Transport Canada. The Coast Guard’s role includes collecting and analyzing vessel traffic information and other data related to marine activities. MCTS officers become particularly adept at vessel identification and are therefore purposely targeted by the Marine Security Operations Centres for recruitment.
Resource and Capacity Issues
The College faces some key challenges with no immediate or easy answers:
- It has proven particularly difficult to attract bilingual instructors who can teach in both official languages. Bilingual officers can readily pick from a range of attractive employment options, none of which necessarily includes the College.
- Many instructors have been teaching at the College for some time and lack recent field experience. The College continues to grapple with how best to keep them current. Guest instructors can augment their teaching, but there is a need to rotate College instructors back to the field. The 2005 review similarly recommends that the Coast Guard consider measures to facilitate rotation of Coast Guard officers from the field into instructor positions and back again.
- Infrastructure support and a life-cycle management plan continue to be pressing needs.
- Fleet training needs exceed the College’s current capacity to deliver. For example, the Coast Guard faces a significant loss of 3rd Class marine engineers due to mid-career changes in employment. Helping 4th Class officers progress to higher levels of certification involves new approaches, such as distance learning and coaching, but these require time and resources to develop, implement and maintain.
- The College does not always have the same equipment (simulators) as field operations, which means more training is required in the field.
- Some courses require frequent updates. It is problematic when the College does not have the funds to cover updates in both official languages.
Interviewees from the Coast Guard directorates and the College said it would be beneficial to assess the impact training has had in the workplace. This information is an important part of assessing the performance of post-secondary educational institutions and adult training programs, although typically no such follow up is ever done. The College and the Coast Guard are in an ideal position to assess the actual impact of training, at some point after the training. Such feedback would contribute to further strengthening the College’s training by informing course design and content. It would also expand performance monitoring of the College’s training, increasing the scope of assessment from quite comprehensive to very comprehensive.
Recommendation 1 : The Commissioner should ensure the Coast Guard develops assessment tools, processes and an implementation plan to undertake post-training assessments of the Canadian Coast Guard College’s various training programs.
The evidence shows that College-trained personnel are directly involved in the delivery of Coast Guard programs over the intermediate and long term. There is also some evidence that ships’ officers who attended the College have an advantage at the outset over those who did not. Post-training assessments and a College alumni registry would provide useful evidence and feedback on intermediate and long-term outcomes.
The evaluation examined whether the College has the appropriate governance, processes and systems in place to support program delivery, whether roles and responsibilities are clear between the College and its clients, and whether the College is using its resources efficiently. The College recently introduced many governance and operational changes that have significantly improved its operations, and the College is diligent about efficient use of resources. There is a need, though, for improved communication amongst Coast Guard programs.
Key Finding on Operations: The College’s new governance structures, processes and systems have strengthened operations in support of program delivery.
Recent Organizational Changes
The College has undergone a major transformation initiative to improve its effectiveness and efficiency. The 2008-09 Transformation Plan, which is being implemented over three years from 2009-10 to 2011-12, sets out the governance and organizational structure now in place. These changes reflect recommendations from the 1996, 2003 and 2005 reviews, but most particularly from the last review. Both the 1996 review and the Department’s 2003 assessment called for a Board of Governors for the College. A strong governance structure is an important component that contributes to efficiencies. The 2005 review made various recommendations concerning governance, staffing and operations. The Transformation Plan addresses many of them, although not all. For example, the plan does not address training of ships’ crew employees to become ships’ officers, instead focusing on other key initiatives to be accomplished over the three years.
The College’s new governance structure includes an Advisory Council, an Academic Council and three sub-committees: Academic Standards, Knowledge Management, and Courseware and Curriculum. The College and Cape Breton University are represented on each other’s governing body. The University’s Dean of Science sits on the College’s Academic Council, and several professors are involved with the College’s academic sub-committees. The new organizational structure includes a Quality Assurance and Academic Excellence Unit, created in 2009-10. The unit is responsible for ensuring OTP meets the needs, standards and requirements for academic excellence as set out by Fleet, Transport Canada and Cape Breton University. Another organizational change involved splitting the former Department of Nautical Sciences into three faculties (Navigation; Marine Engineering; and Arts, Sciences and Languages), as the former structure proved to be too broad. While it is too soon to know the full impact of these changes, there are some initial results which are presented in the next section.
The College undertook a major OTP curriculum review in 2010-11 in consultation with Fleet, Transport Canada and Cape Breton University. The review looked at quality management, academic rules and Fleet’s competency requirements distinct from those required for Transport Canada certification. New policies were developed based on MCTS training policies which have been in place for many years. The review also addressed a recommendation from the 2005 review to determine how OTP cadets could best achieve second language proficiency. The report, issued in May 2011, identified program changes that could be made, which have since been implemented, to support cadets’ second language acquisition to BBB levels. Coast Guard executives require a CBC linguistic profile, and OTP is the basic training ground for this cadre. It is less expensive and disruptive to provide language training to officer cadets than to do this when they are full salaried employees.
The College is a member of the Canadian Association of Marine Training Institutions, which links it with other marine schools in Canada. It is also a member of the Association of Canadian Community Colleges and the Society for Teaching and Learning in Higher Education. The latter is dedicated to advancing teaching and learning technologies. These partnerships contribute to the College staying on the forefront of developments in marine training and educational instruction.
Before and After the Plan
The Transformation Plan has its roots in a significant change taking place in 2005. In that year, the College came back into a direct reporting relationship with the Coast Guard, having previously reported to DFO’s Human Resources since 1995. The Coast Guard proceeded with an in-depth assessment of the College, resulting in a report with several recommendations to improve operations. This subsequently led to the development of the Transformation Plan.
Interviewees at the College and National Headquarters spoke of the College as being isolated from the rest of the Coast Guard prior to 2005. The College looked like the Coast Guard, but it had its own subculture which conflicted with the Coast Guard’s. The change in the reporting relationship has had a positive effect.
Arbitrary processes and unilateral decisions have been replaced with procedures. Issues are delegated to sub-committees to develop proposed courses of action. Business cases are required, based on an analysis of the situation and the requirements. Prior to the Transformation Plan, the College had no analytical capacity, whereas now it has staff to assist with analyzing and synthesizing information. Where items used to fall by the wayside, someone is now responsible, and people who need to be informed are now kept informed.
Responsibility for MCTS training rests solely with the Coast Guard, unlike OTP which has three stakeholders (Fleet, Transport Canada and Cape Breton University). It is Coast Guard policy that all MCTS officers be certified. A MCTS Officer Training and Certification Policy details how MCTS students attain certification and the criteria for post-certification training.
MCTS training requirements are overseen by a Training Advisory Group comprised of staff from National Headquarters and the College. The group provides advice and recommendations to MCTS senior management and the College on standardized national training and related matters. The College is developing a MCTS refresher course as a result of MCTS’ ongoing monitoring of training requirements. It is also developing an instructor training course for MCTS employees who instruct trainees at MCTS centres. This evolution in MCTS training indicates that the College and its client have the necessary processes and systems in place to support program delivery.
With respect to vessel traffic services, the directives of the International Association of Marine Aids to Navigation and Lighthouse Authorities (IALA) are another component that shapes MCTS training. In order for the Coast Guard to maintain its IALA accreditation, it is required to audit its MCTS training for vessel traffic services every five years to ensure compliance with IALA guidelines. An audit was scheduled for 2011-12. A review of MCTS student training is scheduled for 2012-13 to ensure training is up to date in accordance with operational needs, and current rules and regulations. These requirements contribute to ensuring the appropriate structures and processes are in place to support program delivery.
The College has introduced a new governance structure and other operational changes as part of a major Transformation Plan. Although these changes are quite recent, with some still underway at the time of the evaluation, initial evidence suggests that these changes have been beneficial. Other evidence indicates that the College has partnerships and processes in place that contribute to keeping its training programs current.
Key Findings on Roles and Responsibilities: These are clear for the most part, but the College and its clients would benefit from more open communication between them. The College and the Coast Guard would both benefit from a more efficient and effective process for recruiting and selecting OTP applicants that involves the regions and National Headquarters.
Both College and Coast Guard directorate interviewees consider the roles and responsibilities between them to be clear. On average, they gave a score of 3.9 out of 5, where 3 is moderately clear and 4 is clear (n=13). The Coast Guard directorates are the College’s clients, responsible for identifying training requirements. The College’s role is to develop training, advise on delivery and, in many cases, deliver the training.
Still, there is room for improvement. Not all Coast Guard program staff fully understand that they are responsible for specifying what needs to be taught. The many changes that have taken place over the last three years have caused confusion for some directorate staff about the College’s role and services.
There is a need for more direct and early communication between the College and its clients. It takes time to develop curricula and, if necessary, engage trainers. Current lead times are insufficient. For example, the College should be involved when equipment or vessel specifications are being developed for a request for proposals. Likewise, the College needs to discuss with its clients planned changes that can impact on field operations and staff. The College would also benefit if the directorates were clear among themselves about program or service changes and how training or new equipment costs are to be covered.
Communication between the College and officers supervising at-sea cadets could also be improved. Close to three-quarters of survey respondents are clear about their role in providing direction and feedback to OTP cadets. Only half, though, are clear about their role in providing feedback to the College. Some suggested more direct, informal contact and communication with the College, perhaps through more frequent and direct interaction with the Sea Training Officer. Some also feel that the College has not always been attentive to their feedback in the past.
Work on establishing a proactive training governance model was underway in 2011-12. The framework will assist in addressing how the College can further contribute to the Coast Guard. It will provide a process for identifying and developing training, and it is expected to contribute to clarifying roles and responsibilities. Nevertheless, this should not diminish other efforts to improve early and direct communication.
Recruiting and Selecting OTP Applicants
The College lacks the capacity to handle the breadth of work and intensive peak periods inherent in recruiting and selecting OTP applicants. It was given this responsibility in 2007 and dealt with 285 applications on average in the first two years. This figure mushroomed in the next two years to 2,681 in 2010-11, an increase of 840%. In contrast, National Headquarters and the regions look after all of the work involved in recruiting and selecting MCTS applicants.
The College has been working with the Public Service Commission and the Department’s Gulf Region to streamline the process. Still, the need for transparency requires a more rigorous process than in the past. The College must document why applicants with certain physical limitations are excluded. A few years ago they would have been screened out without such documentation.
The regions provide some assistance, but the College and the Coast Guard as a whole would benefit from a more structured and efficient process. It is not a matter of hiring more resources, but achieving a better balance in sharing the work involved. For example, the regions could assist with outreach to women and visible minorities across the country. The College can do this only in its immediate area. It would be beneficial for the College and Fleet to develop a process for National Headquarters and the regions to be more involved in facilitating and undertaking outreach to potential OTP applicants and in selecting suitable candidates. The 2005 review noted the importance of a proactive campaign to attract and recruit strong candidates, noting this should be a joint effort between Fleet and the College, with the regions heavily involved in the selection process. The 2005 review also recommended that recruitment efforts target a significantly elevated employment equity representation until target levels are achieved in the organization.
Environmental Response Training
The Coast Guard’s role and that of the College with respect to environmental response training have been raised in recent audits by the Office of the Auditor General and the Department’s Internal Audit Directorate. Industry is responsible for oil spills, and federal governance and the roles of various departments still require clarification. As such, the Coast Guard has not been in a position to provide direction to the College on environmental response training. Given that the audit reports are specific on the issues and shortcomings, and that the matter had yet to be resolved at the time of the evaluation, there are no further points or definitive results to comment on at this time.
1OAG (2010) Chapter 1 – Oil Spills from Ships, 2010 Fall Report of the Commissioner of the Environment and Sustainable Development; DFO (2010) Audit of The Canadian Coast Guard – Environmental Response Services
The evidence indicates that roles and responsibilities, although fairly clear, could be strengthened. Specifically, more effort needs to be given to open communication within the Coast Guard. The College and Fleet would benefit from more involvement on Fleet’s part in recruiting and selecting OTP candidates. Not only has the workload for this activity increased substantially since 2009, but the College is not in a position to outreach as necessary to women and visible minorities across the country.
Key Finding on Use of Resources: The College is efficient in its use of resources.
As shown in the chart that follows, two-thirds of expenditures are for the training programs, including OTP cadet salaries. The remaining third is for administration and other support services.
Figure 2: Distribution of College Expenditures
Post-secondary institutions often look at administrative cost as a measure of financial performance. According to a couple of sources, administrative costs for post-secondary institutions average about 20 to 30%. At 11%, the College appears to be doing very well. However, neither source specified what is included in administration costs. For example, does it include campus services? Without knowing how other institutions define their administrative costs, it is not possible to make comparisons and draw valid conclusions.
In order to determine whether the College’s operating expenditures are reasonable, it would be beneficial for the College to seek input on other institutions’ operating costs, such as those of other federal training facilities or post-secondary educational institutions. This input could be used to assess the College’s administrative expenditures from time to time. As there is no standard definition of what constitutes administrative costs and perhaps other operating costs, the College needs to ensure any comparison is based on exactly the same item. For example, another institution might include campus services or perhaps other items as part of its administrative costs.
Efficiency Part of Ongoing Operations
Efficiency is an integral part of College operations. In addition to being a significant element of the Transformation Plan, efficiencies were again looked at in the Department’s 2010-11 Strategic Review. And they remain part of ongoing considerations. For example, when staff leave or a contract is up for renewal, the College reassesses whether it is making the most efficient use of resources. It recently determined that it could save $70,000 by hiring a physical education instructor instead of contracting for these services, as it had done for over 10 years. At the time of the evaluation, staff were working on a business case to engage fourth-year OTP cadets as night duty officers in place of casual staff. This will allow the College to have daytime duty officers who can also help with administrative work.
The evaluation sought to determine whether there are any viable alternatives to the College’s training programs that would be more economical. Of the options considered, no suitable alternatives emerged. The College appears to be a viable and economic means of meeting the Coast Guard’s need for various types of highly specialized training.
Key Finding on Economical Alternatives: The evaluation found no viable economic alternatives to the Coast Guard College or any of its programs.
No Viable Alternatives
No viable options to the College and its training programs were forthcoming. The College develops and delivers training unique to Coast Guard requirements. College simulators provide a training context that mimics MCTS and Fleet equipment, vessels and operations. Provincial marine schools provide training and simulations relevant to industry needs. It is not feasible for others to invest in MCTS and Fleet simulators to meet the unique needs of one employer and the relatively small number of students and trainees involved. Likewise, it would not be feasible for other marine schools to incur the costs inherent in designing and delivering related training.
Part of the College’s role is to advise on the best means of delivering training. It may be done at the College, at another educational institution or in the field by a College instructor or a third party, such as a consultant. The College is not the delivery site by default and it makes use of other educational institutions as appropriate. For example, the College’s MMET courses do not replicate community college training, and OTP cadets take their marine emergency training at a provincial marine school.
OTP represents 44% of total expenditures, the largest portion of the College’s budget (excluding building management). It is therefore appropriate to consider more closely whether there are viable options for this program.
A couple of decades ago, the College scaled back OTP from fours years to three. This proved unsatisfactory. Fewer students were able to meet Coast Guard requirements within the shorter period, so the program returned to four years. The Coast Guard wants high-calibre graduates with solid academic grounding, bilingual capabilities and leadership skills to best position them for future advancement to senior ranks.
Reliance on provincial marine schools as a prime or sole source of Fleet officers would be more costly for the Coast Guard. A period of training specific to Coast Guard operations would have to be done with employees at full salary rather than with students on a cadet salary. This would include training related to operations, vessels, bilingualism, leadership and further academic studies necessary for higher levels of officer certification.
Alternatively, the College might recruit students who have been at a marine school for one or two years. However, marine schools are having difficulty recruiting students, especially for marine engineering. Seagoing careers are not highly regarded, according to a 2008 study on Canada’s marine industry. If the Coast Guard is viewed more favourably, then it is at an advantage recruiting directly into its own training program.
Another option might be for the Coast Guard to send its recruits to other marine schools for one or two years. However, training requirements specific to Fleet are interwoven throughout the four years, with culture, ethos and living and working closely together part of each year’s learning experience. OTP cadets begin training in their first year in preparation for at-sea training in their second year. The following list is a sampling of the subjects specific to Fleet that are covered in the first year: Harassment Free Workplace training; Stress and Sexual Safety Education; familiarization with Coast Guard operations and the responsibilities of a Fleet officer; the concepts and dynamics of leadership, team management on Coast Guard vessels; the dynamics and responsibilities of a follower; physical fitness; and seamanship skills and small boat operations. Furthermore, OTP graduates receive a Bachelor degree from Cape Breton University, and their academic studies must satisfy the university’s requirements, as well as those of the Coast Guard. Bringing in OTP cadets from other marine schools becomes less straightforward.
National Defence sends students to a provincial marine school for training in marine engineering, but not navigation. Staff at a nearby military base oversee the students and provide military training as required. Military staff are exempt from Transport Canada officer certification requirements. As such, the military need not be concerned with at-sea training, which is needed for Transport Canada officer certification, until after classroom learning is completed. A key question in considering whether this would be a viable option for training marine engineers for the Coast Guard is whether in fact the pros outweigh the cons. If there is no viable option to the College for its navigation training, one needs to look at what would be gained and what would be lost by contracting out a year or two of training for marine engineering. It is not readily apparent that the pros would outweigh the cons.
It is interesting to note that the Coast Guard’s approach to OTP recruiting and training reflects best practices noted in the 2008 federal study Marine Transportation, Ports and Ocean Technology Situational Analysis. They are as follows:
- Very rigorous recruiting methods and screening procedures
- Employer-paid tuition for all courses on any work-related course for any position
- Companies offered work placement for students who later became employees
- Specific recruitment problems addressed through targeted advertising.
Cost Per Student Analysis
The average annual education cost per OTP cadet was $30,339 for the five-year evaluation period. Variable costs (cadet salaries, books, uniforms, accommodation, travel and so forth) averaged $16,963 annually per student. Altogether, the annual cost per student was $47,302.
The average education cost per student for the College’s six-month MCTS training program was $21,739 over the five-year evaluation period. With other College education costs added in, such as books and other materials, the cost per student was $25,879. The Coast Guard regions, not the College, cover students’ living expenses and allowances. This figure does not include any costs associated with field training of graduates at MCTS centres.
The evaluation analyzed OTP education costs in comparison with undergraduate education costs, in particular engineering programs. The analysis drew on Statistics Canada data, plus other related information. As the College is training for a very specialized function and has a small number of OTP students, it is reasonable to expect higher education costs per student than for university programs with higher enrolments. The comparison does not include variable costs such as living expenses and books. The evaluation calculated the average annual education cost per undergraduate engineering student as being roughly $22,000 to $25,000. The College’s annual cost of $30,339 per student appears to be quite reasonable.
Length of Mandatory Service
OTP graduates are required to provide four years of mandatory service upon graduating. This is in return for coverage of their educational costs and related expenses, such as accommodation, meals, uniforms and so forth. Given a federal government obligation to ensure its programs maximize the return on investment for taxpayers, the Coast Guard needs to ensure that the length of mandatory service is appropriate.
National Defence has a minimum mandatory service period of five years for its military college graduates. Graduates of the United States Coast Guard Academy likewise have a mandatory service period of five years. No comparable data was readily available for other countries researched, which in addition to the United States included Australia, Ireland, Japan, New Zealand, South Africa and the United Kingdom. Aside from the United States and Japan, the coast guards in these other countries appear to be dedicated solely to search and rescue. The government organizations that provide services comparable to the Canadian Coast Guard appear to be maritime authorities, customs and border services, and transport departments. No specific information could be found as to which marine schools are used for training for civilian government vessels and any mandatory service requirements.
Recommendation 2 : The Commissioner should ensure the Coast Guard requires an appropriate length of mandatory service for OTP graduates, specifically whether four years or longer is sufficient, and determine whether other forms of reimbursement should be introduced when mandatory service cannot be fulfilled. Reasons in support of the chosen length of service and any other forms of reimbursement should be well documented.
The Canadian Coast Guard College is an essential and viable program. It develops and delivers training, not available elsewhere, that is necessary and unique to Coast Guard operations. The College’s relevance has been confirmed in previous reviews and again reaffirmed in this evaluation. Where the College required improvement was in its governance and operations.
The year 2005 marked the first significant change in the College’s recent history, with it being directly realigned as part of the Coast Guard. The next significant development began in 2009 with the College’s three-year implementation of its Transformation Plan. The changes introduced appear to have had a tremendous and positive impact on College operations, serving to strengthen its effectiveness and efficiency. The full results of these changes will become more evident in the years ahead.
The workload for the College has increased. In order to plan effectively and manage resources efficiently, it needs to be involved early on in discussions regarding operational or equipment changes. The College similarly needs to discuss changes with others that have the potential to impact on field operations. The College and its clients at National Headquarters and in the field would benefit from more direct and open dialogue. Recruiting and selecting OTP candidates could be a more efficient and effective process with greater assistance from National Headquarters and the regions.
There is evidence that the return on investment for the federal government extends beyond the Coast Guard and DFO over the long term. Other federal departments purposely recruit Coast Guard officers mid-career who trained at the College. Public utilities and industry have likewise benefited. An alumni registry would contribute to future assessments of longer term outcomes. Post-training assessments would likewise provide evidence on outcomes, plus feedback that could contribute to improving training. Most educational and training institutions do not assess employer satisfaction with the training, but the College and the Coast Guard are in a unique position to do so. For these reasons, the evaluation recommends that preparations and planning be undertaken for post-training assessments.
No economic alternatives were identified that could satisfactorily replace the training provided by the College. In light of the federal government’s obligation to taxpayers, the Coast Guard should reassess whether the length of mandatory service it requires for OTP graduates is sufficient and document supporting reasons for the length of time chosen.
|Issues/Questions||Indicators||Document review||Admin. Data||Lit. Review / Comparative Analyses||Interviews||Fleet Survey|
|College Mgmt./ Staff||CCG Clients||External Stake holders|
|1.1 Is there a continued need for the CCG College?||CCG College meets an essential need and serves unique training requirements||x||x||x||x||x||x|
|1.2 To what extent are College objectives aligned with federal priorities and with DFO strategic outcomes?||College mandate and objectives are in keeping with federal priorities and DFO strategic outcomes||x||x||x|
|1.3 Is it appropriate for the federal government to be delivering this program?||Mandate is in keeping with federal jurisdiction||x||x||x||x||x||x|
|OTP and MCTS graduates demonstrate competency, knowledge and proficiency in accordance with assessment criteria||x||x||x|
B. Specialized Training2.2 To what extent have trainees demonstrated that they have acquired specialized knowledge?
|Trainees demonstrate acquisition of specialized knowledge in accordance with assessment criteria||x||x||x||x||x|
Intermediate/Long-term2.3 To what extent have College-trained personnel contributed to delivery of CCG programs?
OTP / MCTS
Specialized Training Courses
2.5 Are there any unexpected results, positive or negative, as a result of the College�s programs?
3.1 To what extent does the College have appropriate governance, processes and systems in place to support program delivery?
3.2 Are roles and responsibilities clear and appropriate?
3.2 3.2 Could resources be used more efficiently in achieving outputs?
|4.1 Is there a more economical way of training personnel or acquiring trained personnel?||
|# Applicants||Not available||319||247||1018||2681||4265|
|# Qualified for aptitude test (met academic requirements)||Not available||172||171||160||394||897|
|# Qualified for pool||Not available||53||45||60||83||241|
|Diversity targets considered in selection: Yes/No||Not available||Y||Y||Y||Y||-|
|Attrition (by cohort)||14||10||6||9||10||49|
|Intake – maximum 12 students per class (# classes)||27 (3)||27 (3)||30 (3)||21 (3)||19 (2)||124|
|Attrition (by cohort)||4||4||3||4||1||16|
|1. No intake in 2004; therefore, no graduates in 2008.|
Source: CCG College
|MMET: Training Seats Provided||108||152||134||150||121||649|
|SAR (CCG and DND)||27||24||54||26||32||163|
|ER (CCG and industry)||45||0||27||33||32||137|
Recommendation 1 :
The College, along with its program clients in Fleet, Maritime Services, and Integrated Technical Services, will develop an initiative to establish a post-training assessment methodology. The purpose of such post-training assessments will be to assist programs during the identification and development of Coast Guard operational competencies. The post-training assessments will become a tool to ensure the delivery of reliable and relevant training and will form an integral part of the College’s performance measurement framework.
|Management Actions||Actions Completed||Actions Outstanding||Target Date||Supporting Evidence|
|Consult with three Coast Guard Program clients (Fleet, Maritime Services. Integrated Technical Services) to define objectives and develop a Terms of Reference for a post-training assessment methodology.||December 15, 2012|
|Initiate work on developing a methodology.||January 2013|
|Present preliminary work at a pre-conference session of the Society for Teaching and Learning in Higher Education to obtain academia input for further development of the post-training assessment methodology.||July, 2013|
|Test methodology.||December 2013|
|Present data collection and evaluation methodologies to the CCG College Advisory Council.||February 28, 2014|
|Submit post-training assessment plan to CCG-Management Board.||April 1, 2014|
Recommendation 2 :
To undertake a comprehensive review of the length of mandatory service/reimbursement model for Canadian Coast Guard Officer Training Program graduates to ensure value for money and provide a rationalization which considers other Government of Canada training programs and their respective length of mandatory service/reimbursement models.
|Management Actions||Actions Completed||Actions Outstanding||Target Date||Supporting Evidence|
|Canadian Coast Guard to identify an OPI to undertake a review of mandatory service/reimbursement model for the Officer Training Program.||November15, 2012|
|Consult with Legal Services, CFO and Unions to identify potential impacts to changes in mandatory service requirements/reimbursement.||November 2012 – June 2013|
|Collect data from other Federal Government training programs on length of mandatory service/reimbursement.||February 2013|
|Evaluate impact changes to length of mandatory service may have on CCG –OTP recruitment levels.||December 2013|
|Present findings to CCG-Management Board.||March 30, 2014|
- Date de modification