Evaluation to Support Canadian Coast Guard Force Generation

Evaluation Report
March 2019

Evaluation Division, Planning, Results and Evaluation Directorate

Fisheries and Oceans Canada

Table of Contents

• 1.0 Evaluation Context
  • 1.1 Scope of Evaluation
• 2.0 Evaluation Methodology
  • 2.1 Lines of Evidence
  • 2.2 Limitations of Methodology
• 3.0 Evaluation Findings
  • 3.1 Crewing Factor Methodology
    • 3.1.1 Current CCG Method and Key Gaps
    • 3.1.2 Methodology for Calculating a New Crewing Factor for the CCG
    • 3.1.3 Demand Analysis
    • 3.1.4 Supply Analysis: Training
    • 3.1.5 Supply Analysis: Leave
  • 3.2 Certificates of Competency
    • 3.2.1 Gaps
    • 3.2.2 Challenges Upgrading
    • 3.2.3 Solutions to Address Challenges
  • 3.3 Personnel Development Governance
    • 3.3.1 Definition
    • 3.3.2 Responsibility for Activities
    • 3.3.3 Mechanisms for Coordination and Information Sharing
• 4.0 Conclusions and Recommendations
• Appendix A: Evaluation Questions
• Appendix B: Management Action Plan

NOTE:The Evaluation to Support the Canadian Coast Guard’s Force Generation has an alternative report format that uses graphics and visuals, which is available in pdf format

1.0 Evaluation Context

1.1 Scope of Evaluation

The evaluation to Support CCG Force Generation was designed as a developmental evaluation ^{Footnote 1} to gather information to support the CCG’s Force Generation Project. The evaluation was intended to build on the Evaluation in Support of the CCG’s Seafarers Establishment (2017-2018) and explored three key issues (see Appendix A for a list of evaluation questions).

The scope of the evaluation did not include: the calculation of a new crewing factor; an assessment of the magnitude, appropriateness, or quality of personnel development-related products and tools; or personnel development activities funded through the Oceans Protections Plan.

2.0 Evaluation Methodology

2.1 Lines of Evidence

Focus Groups

• Held one NHQ focus group with 10 participants to consult on the methodology for developing a crewing factor.
• Held eight regional focus groups (Victoria, St. John’s, Quebec City, and Sarnia) to discuss the methodology for developing a crewing factor and to determine how the model could be applied to the CCG; and to discuss certification gaps and alternatives.
• Participants totaled about 100 personnel from across the CCG and primarily included personnel from fleet who work ashore.

Mapping Exercise

• Administered an excel-based template to all Regional Directors and NHQ Directors (approximately 50 individuals) to gather information on their level of involvement with activities related to recruitment, training, career management, and wellness.

Interviews

• Conducted interviews with 32 individuals in the regions and NHQ to discuss level of involvement with activities related to recruitment, training, career development, and wellness; and to understand the level of coordination and information sharing with respect to these activities.

Data Analysis

• Gathered and examined data from MariTime related to certificates of competency, exemptions, and training.

Document Review

• Reviewed relevant documents related to the CCG Force Generation Project, alternative practices for crewing methodology in other jurisdictions (domestic and international), and certification requirements.

2.2 Limitations of Methodology

Focus Groups

• Participants were invited to attend the focus groups based on their experience with the topics being discussed. Operational requirements may have limited the ability of some personnel to participate. This was particularly the case for seagoing personnel, as few were able to attend. To mitigate this, former seagoing personnel, who are currently in shore-based positions, were included to the extent possible.

Mapping Exercise

• Templates were not received from all directorates within NHQ. This did not have a significant impact on the results as templates were received from all regional directorates and the evaluation team conducted targeted follow-up with NHQ directorates that have a key role in personnel development to ensure those templates were completed^{Footnote 2}.
• The information provided in the mapping templates is a subjective assessment of each directorate’s level of involvement in the activities included in the template. This could have resulted in some inconsistency in the information provided. To mitigate this, interviews were conducted with regional directors and NHQ directors to clarify the information provided in the template.

Data Analysis

• It was not possible to use MariTime data to assess the gaps in certification and training, as the data are not reliable (i.e., they are incomplete and inconsistent). This was confirmed through analysis and validation with CCG program representatives and focus group participants in all regions. Program documentation also noted that the certification and training modules in MariTime have not been validated for quality on a regular basis due to the lack of business rules and training for officers and clerks
• Instead of using MariTime, regions maintain separate records (e.g., excel spreadsheets, human resource plans) to track and manage training and certification gaps and needs. There are variations in how this is being done across the regions.
• The evaluation had to rely on qualitative information to understand gaps in certification and training.

3.0 Evaluation Findings

3.1 Crewing Factor Methodology

3.1.1 Current CCG Method and Key Gaps

CURRENT CCG CREWING FACTOR CALCULATION

The CCG currently calculates the number of personnel required (N) based on the number of positions in the crewing profiles for vessels and a crewing factor determined in 1995^{Footnote 3}.

GAPS IN THE CURRENT CREWING FACTOR

Based on an analysis of crewing factor models, the results from the 2017-2018 evaluation, and consultation with CCG personnel, there are known gaps and limitations with the current calculation.

• Workload analysis on vessels has never been conducted. Therefore, the number of positions (P) in the crewing profiles may not accurately represent the demand for personnel on vessels.
• The current crewing factor considers only some leave provisions and does not consider the relief needed for training personnel. Therefore, it overestimates the availability of an individual employee, which results in underestimation of the number of personnel (N) required.
• Differences across seagoing units (e.g., size of vessel, position) have never been analysed. Therefore, the factor may not accurately reflect those differences.
• Some units indicated that they have never been able to staff to the factor of 2.5 due to resource constraints.

3.1.2 Methodology for Calculating a New Crewing Factor for the CCG

DEVELOPMENT OF A METHODOLOGY FOR CALCULATING A NEW CREWING FACTOR

The Evaluation in Support of the CCG’s Seafarers Establishment included best practices research to identify methodologies for developing crewing factors. The manpower requirements determination process, implemented by the United States Coast Guard (USCG), was found to have the most comprehensive and rigorous methodology. The evaluation team used the approach of the USCG to develop a methodology for calculating a new crewing factor for the CCG. This methodology was used for consultation with the CCG and was then revised to reflect a feasible approach that could be applied in the CCG context. The methodology has four components, which are shown below.

IMPORTANCE OF ESTABLISHING THE UNIT OF ANALYSIS

The crewing factor methodology should be applied to a CCG unit, which is a sub-group of CCG seagoing personnel with similar conditions related to workload, individual time, leave provisions and training needs. The CCG will need to determine appropriate units before implementing step 1a and 1b of the methodology. Demand and supply analysis will need to be completed for each unit. A detailed description of the methodology is provided below.

1a. Demand Analysis

Based on strategic documents, regulations, CCG mandate, program guides and levels of service:

• Establish CCG program requirements and unit work requirements; and
• Determine how much time it takes to complete the unit work, the total unit workload, in hours per year.

For the CCG, the methodology suggests five key categories of workload.

1b. Supply Analysis

Based on collective agreements, regulations and policies, and training needs and requirements:

• Determine the total assigned individual time (e.g., on-cycle working hours in one year);
• Establish the amount of time an employee is unavailable to do the workload of the unit, in hours per year; and
• Subtract unavailable time from the total assigned individual time to determine individual program time, in hours per year.

For the CCG, the methodology suggests an analysis of how much time is required for leave and training needs.

2. Calculation of Unit Personnel Required and Crewing Factor

The number of unit personnel required (N) is the ratio of the total unit workload and the individual program time.

For the CCG, the crewing factor (CF) of the unit is the ratio of the unit personnel required (N) and the number of positions (P) in its crewing profile (CF = N/P).

3. Implementation of the Crewing Factor

The implementation of the crewing factor requires coordinated support from other functions of the organization.

The timeline and the actions for implementation of the crewing factor are dependent on factors related to feasibility, risk, and priorities.

For the CCG, the crewing factor is largely based on human and financial resources, not all six elements shown here.

3.1.3 Demand Analysis

CATEGORIZATION OF WORKLOAD ON VESSELS

As noted previously, the crewing profiles establish the number of positions required on each vessel. However, because a workload analysis for fleet has never been conducted, the number of positions in the crewing profiles may not accurately represent the number of personnel that are needed to complete the workload.

The crewing profiles present some limitations to the demand analysis because it assumes that changes to the number of personnel aboard (e.g., to the crewing profiles) are possible, which is not likely the case in the short-term. Despite the limitations, there is still support for conducting a demand analysis because CCG personnel reported that the workload on vessels has been increasing and it would be useful to have a better understanding of the impact of this increase on CCG program operations.

If a demand analysis is to be conducted, the evaluation identified five workload categories, which could be used to describe the work on vessels. Note that some examples of the types of activities that could be included in these categories have been provided; but it is not an exhaustive list.

METHODOLOGICAL CONSIDERATIONS FOR DEMAND ANALYSIS

In conducting an analysis to determine how much time is required to complete all tasks and work activities within the responsibilities and requirements of the unit, the following would need to be considered.

Data Availability

• Given that an analysis of the workload on vessels has never been conducted before, and data on workload has not been consistently collected, the workload analysis would likely require new data collection. Potential sources to consider are documents (e.g., levels of service, logbooks, management tools), observations, reviews of other marine organizations, and qualitative information from experts in CCG operations.

Variability of Factors That Affect Workload

• Depending on the specific context, the volume and the categorization of workload may vary by size of vessel, department, program or vessel stage (i.e., operating, alongside, dry-dock). Some activities may be classified across several categories (e.g., maintenance, administration).
• It may be necessary to conduct a separate workload analysis for different units in order to reflect the operational, regional, and contextual specifics (e.g., seasonal units).
• Other factors, such as regulatory changes or composition of personnel, may also have an impact and should be considered (e.g., increased need for on-the-job training due to an increased number of new or temporary staff, increased administrative work due to new regulatory tracking requirements).

Recent Contextual Factors

3.1.4 Supply Analysis: Training

TRAINING CATEGORIES THAT HAVE THE BIGGEST IMPACT ON RELIEF

In the operational context of the CCG, training is an important aspect of professional development for personnel. However, it is also an important vehicle to ensure the safe and effective operation of vessels. The amount of time needed for seagoing personnel to take training is currently unknown. In addition, training now largely must occur during a person’s off-cycle (i.e., during their time off).

METHODOLOGICAL AND FEASIBILITY CONSIDERATIONS FOR TRAINING

In conducting a training analysis that allows enough relief to meet training needs and priorities, the following would need to be considered.

Identification of All Types of Training

• Each of the training categories that have been identified is very broad and includes a wide range of training courses, all of which would have to be taken into consideration in the analysis.

Data Availability

• There are currently no reliable data from CCG administrative systems to be used for a historical data analysis of the time spent on training. Thus, a collaborative effort of data analysis experts, national and regional program and training specialists, and the CCG College will be required to identify alternative ways to collect this information.
• Some documentation does exist, which would help inform the analysis of training needs, particularly for regulatory and CCG-specific training.

Variability of Factors that Affect Training Needs

• There may be differences in the type of training required across regions, vessels, departments, classifications, seniority, and over time. The variability will need to be considered in the analysis.

National and Regional Needs

• National training requirements as well as regional-specific training needs and priorities should be taken into consideration in the training analysis.

3.1.5 Supply Analysis: Leave

LEAVE CATEGORIES THAT HAVE THE BIGGEST IMPACT ON RELIEF

METHODOLOGICAL AND FEASIBILITY CONSIDERATIONS FOR LEAVE

In conducting a leave analysis to determine how much relief time is required to allow seagoing personnel to take leave, the following would need to be considered.

Data Availability

• Use of historical data for some categories of leave is feasible, but there are limitations to consider. For example, most leave categories were not used to the extent needed and requested due to operational constraints and lack of sufficient relief personnel; thus, historical data would underestimate the actual need. Qualitative analysis may be required to complement administrative data
• Some categories of leave will require data collection and/or modelling because they are recently added provisions and no historical records have been accumulated.

Variability of Factors that Affect Leave

• The analysis should ensure that the variability across departments, regions, seniority and over time are considered.

Leave Banks

• Seagoing personnel currently have large leave banks (e.g., annual leave, compensatory, and layday leave) that must be considered in the leave analysis.

Other

• Some categories of leave may require relief for longer than their duration due to logistics and scheduling issues (i.e., from an operational perspective, it is difficult to remove a person from a vessel for a short period of time).
• Requirements for accommodation (e.g., gradual return to work following an injury) and the related impact on relief should be considered.
• The leave analysis should apply a forward-looking approach to address changes in leave provisions that are very recent or anticipated in near future.

3.2 Certificates of Competency

3.2.1 Gaps

CERTIFICATION REQUIREMENTS FOR SEAGOING PERSONNEL

The International Maritime Organization (IMO) is a specialized agency of the United Nations with responsibility for the safety and security of shipping and the prevention of marine and atmospheric pollution by ships. The IMO is responsible for several conventions, including the Standards of Training, Certification and Watchkeeping for Seafarers (STCW), which establishes international requirements for seafarers.

GAPS IN CERTIFICATES OF COMPETENCY

Findings from the evaluation determined that the CCG currently is experiencing gaps primarily for four certificates of competency:

• Master Near Coastal;
• Chief Mate Near Coastal;
• Second Class Engineer; and
• Third Class Engineer.

It is anticipated that gaps for First Class Engineer certificates will also exist in near future due to the large number of eligible retirees.

These gaps were identified through qualitative discussion and analysis, as there are a number of limitations with respect to MariTime data. The certification data are not reliable and the system does not provide some of the contextual information that is important when analyzing information on certification^{Footnote 4}

3.2.2 Challenges Upgrading

MANAGEMENT OF TRAINING AND SEA TIME

• CCG uses the scholarship program to provide support to personnel to upgrade certificates (program varies by region); funding for scholarships has increased in the current fiscal year.
• Otherwise, the management of training and sea time is largely done ad hoc and on a case-by-case basis and is often done on an individual’s own time (i.e., on their off-cycle).
• There is a high reliance on an individual’s initiative to identify needs for training and sea time and to request support.
• Sea time can be challenging to obtain because of limitations on the vessel size in some regions, particularly in Western Region and Central and Arctic Region (Great Lakes Sector).

AVAILABILITY OF, AND ACCESS TO, TRAINING

• There is a lack of instructors for the training that is needed to upgrade certificates.
• Western region faces challenges accessing courses due to its distance from the College and the limited number of marine institutions in the region.
• Central and Arctic Region (St. Lawrence Sector) faces challenges accessing courses in French.
• There is little financial support available to personnel to take training (i.e., cost for training, time-off, travel time), thus requires personal investment.
• Training schedules often do not align with crew schedules.

PERSONNEL SHORTAGES

• There is a lack of people to provide relief to allow personnel to take required training during their on-cycle time, or participate in developmental assignments, which are required to upgrade certificates.

INDIVIDUAL / REGIONAL CIRCUMSTANCES

• CCG personnel are often required to invest their own time and money to upgrade certifications, which can present financial implications and impact the work-life balance of personnel. This can impact personnel’s ability to upgrade their certifications.
• Personnel may not be able to advance in their current region (i.e., some regions have higher-level positions than other regions, due to types of vessels being used) and thus do not see a value in upgrading certificates.

3.2.3 Solutions to Address Challenges

ROLE OF THE CCG COLLEGE IN TRAINING

There is a need to improve the availability of, and access to, training, particularly because it is a tool to maintain and improve personnel readiness and safe and effective operations. Expanding the role of the College was consistently raised in all regions as the most feasible solution to address the challenges related to training and upgrading certifications. This could include activities such as:

• providing more online or distance courses;
• having instructors travel to different regions to deliver courses;
• implementing a training program to assist ship’s crew with upgrading certifications to become ship’s officers;
• improving coordination with the regions and NHQ to determine training needs; and
• developing formal partnerships with marine institutions that could offer required training in different locations across the country.

For the College to expand its role in training:

• There would be resource implications, which may lead to the need for additional funding.
• Better coordination between NHQ, regions and the College would be required to ensure training needs are identified and courses are planned and scheduled to align with crew schedules.

OTHER POSSIBLE SOLUTIONS TO ADDRESS GAPS IN CERTIFICATES OF COMPETENCY

In addition to expanding the role of the College, CCG personnel made suggestions for other ways in which some of the challenges related to certification gaps could be addressed.

Improve Administrative Processes

• Improve the planning and coordination between crewing staff and training staff to better align the training opportunities with the crewing schedules.
• Include training dates in the leave plans of vessels (for both on- and off-cycle training) so that training and leave are better planned.

Increase Support Available to Personnel

• Increase the funding and support that is available to personnel, including costs for training and related travel and time to take the training.

Implement Changes to the Crewing System

• Better align crewing cycles to facilitate rotational assignments and offer developmental opportunities.

3.3 Personnel Development Governance

3.3.1 Definition

CONTEXT FOR THE MAPPING EXERCISE

To support the work being conducted as part of the Force Generation Project, the evaluation included a mapping exercise, which was designed to understand what activities are undertaken within the organization related to the development of both seagoing and shore-based CCG personnel. The activities included in the mapping were organized into four main pillars, each with its own sub-activities.

Recruitment and Staffing

The process of bringing new people into the CCG. Included the sub-activities of: development of recruitment strategies, outreach, communications, analysis of recruitment efforts, staffing process, and selection of individuals.

Training

Included both operational and non-operational training. Included the sub-activities of: development of policies and standards, identifying training needs, coordinating training, developing curriculum, and delivering training.

Career Management

The process that enables an employee to move toward career goals. Included sub-activities of: development of policies and standards, tools for career advancement (e.g., mentoring, coaching, career counselling, rotational assignments), and forward-thinking planning (e.g., succession planning).

Wellness

The state that is achieved when employees’ personal and organizational needs are met. Included the sub-activities of: promoting wellness, strategies and initiatives for diversity and inclusion, and strategies and initiatives for retention.

3.3.2 Responsibility for Activities

This table provides an overview of where key lead responsibilities for personnel development activities lie. There are some exceptions, which are reflected in the notes or in the following slides. This table does not reflect where some directorates do not have the lead, but may be involved in these activities.

The evaluation did not assess what activities were being undertaken in these areas or their effectiveness.

RECRUITMENT: SUMMARY OF RESPONSIBILITIES

Recruitment strategies, outreach, communications, data and reporting on recruitment

Key Observations for Recruitment

• There is little knowledge of recruitment activities and strategic directions amongst CCG personnel in NHQ and regions.
• IBMS relies on regional program personnel to attend outreach events, but directorates have limited capacity to participate
• A lack of both resources and a coordinated approach for recruitment activities have an impact on the CCG’s ability to address personnel shortages.

STAFFING: SUMMARY OF RESPONSIBILITIES

Key Observations for Staffing

• There are no nationally consistent statements of merit criteria available for hiring managers.
• Not all directorates use IBMS support; some work directly with DFO Human Resources to complete their staffing processes.
• Regional directorates reported that staffing activities are not timely enough to respond to operational needs.

OPERATIONAL TRAINING: SUMMARY OF RESPONSIBILITIES

Policies, and standards, curriculum development, delivery, identifying training needs, coordination

NON-OPERATIONAL TRAINING: SUMMARY OF RESPONSIBILITIES
Policies and standards, curriculum development, delivery, identifying training needs, coordination

CAREER MANAGEMENT: SUMMARY OF RESPONSIBILITIES
Policy, career advancement, forward-thinking planning

Key Observations for Career Management

• Coordination for career management occurs ad hoc, informally, and for the most part, at the regional level.
• The performance management process and individual learning plans seem to be the primary mechanism to manage career progression for an individual employee.
• Through the Force Generation Project, competency profiles are currently being developed for all operational positions within the CCG.

WELLNESS: SUMMARY OF RESPONSIBILITIES

Key Observations for Wellness

• There are a number of initiatives related to wellness that are organized at different levels (e.g., federally, departmentally (DFO), organizationally (CCG), and regionally).
• Directorates promote wellness activities to their own staff, but often lack the time to participate in wellness-related activities.
• Responsibilities related to retention are limited within a specific directorate or section.

3.3.3 Mechanisms for Coordination and Information Sharing

MECHANISMS IN PLACE TO COORDINATE PERSONNEL DEVELOPMENT ACTIVITIE

There are a number of committees within the existing CCG governance structure (e.g., National Program Management Committee, Regional Management Boards, Operations Executive Board) that provide an opportunity for coordination and information sharing with respect to personnel development. Interviewees did not identify any of these committees as having the overall responsibility for any aspects of, or decisions with respect to, personnel development. In addition, interviewees indicated that the roles and responsibilities for personnel development activities across the organization are not clear, including in particular, those of IBMS and the Force Generation Project.

As a result, there is little knowledge across the organization on what initiatives, activities, or tools are being developed and implemented to support personnel development.

4.0 Conclusions and Recommendations

CREWING FACTOR METHODOLOGY

The findings from the Evaluation to Support the CCG’s Seafarers Establishment indicated that the CCG’s current crewing factor does not reflect all of the elements that are considered to be important and it was recommended that the CCG review and modify its crewing factor. The calculation of a new crewing factor for the CCG will require a number of steps. One of the objectives of this evaluation was to provide information to assist the CCG with the first step in advancing the development of a new crewing factor for seagoing personnel by developing an updated crewing factor methodology.

Recommendation: The CCG should develop an updated crewing factor using the methodology established through the evaluation.

CERTIFICATION AND TRAINING DATA

The availability of high quality, reliable data is important to ensure that training needs are identified and well-coordinated. The training and certification modules in MariTime are not used by regional personnel, as the data are inaccurate and do not provide the information needed to determine certification and training needs. CCG regions are using their own methods to track and analyze training and certification information. Thus, nationally consistent data are not available, leaving a critical gap in information with respect to training and certification needs.

Recommendation: The CCG should improve the quality and reliability of certification and training data to ensure that nationally consistent data are available for decision-making.

MODERNIZING / ENHANCING TRAINING APPROACHES

The findings from the Evaluation of the Canadian Coast Guard College showed the importance of strengthening the role of the College as the Centre of Expertise for marine training. To date, some progress has been made to address the recommendations. Findings from the current evaluation indicate that there is an immediate need to increase the scope and the availability of training. Furthermore, factors such as the physical capacity and geographic location of the College, and the nature of operational and crewing schedules, suggest there is a need to explore new training approaches.

Recommendation: The CCG should modernize its training approaches (e.g., online or distance courses, delivery of courses by instructors in the regions, formal partnerships with marine institutions across the country) to expand access to training opportunities for CCG personnel.

PERSONNEL DEVELOPMENT GOVERNANCE

The evidence presented in the Evaluation to Support the CCG’s Seafarers Establishment supported the conclusion that the CCG is experiencing personnel shortages and lacked a formal career development program. That evaluation included a recommendation that the CCG develop and implement a national recruitment strategy and a formal career development program.

Findings from the current evaluation indicate that personnel shortages, lack of resources and coordination for recruitment, and the lack of support for career development are challenges that still persist. Furthermore, there is a need to improve national coordination and information sharing and to clarify roles and responsibilities for personnel development across CCG , particularly in the areas of recruitment, operational training, and career management. Challenges with respect to coordination and the understanding of roles and responsibilities has had an impact on the CCG’s ability to make progress in implementing the management action plan from the previous evaluation.

Recommendation: The CCG should clarify the governance structure for personnel development activities, particularly with respect to recruitment, operational training, and career development; and ensure that the national and regional accountabilities, mandates, and roles and responsibilities for these activities are clearly defined and communicated.

Appendix A: Evaluation Questions

The evaluation explored three evaluation issues and five evaluation questions. The scope for the crewing factor methodology and the certification requirements was restricted to the seagoing and shore-based personnel in fleet. These two issues cannot be expanded to include the shore-based personnel in other CCG areas until the work to address three of the recommendation of the 2017-2018 evaluation in support of CCG seafarers establishment is finalized^{Footnote 8}. The scope of personnel development governance was explored for both fleet and seagoing personnel.

Appendix B: Management Action Plan