Evaluation of Grants and Contributions Programming in the Ecosystems and Oceans Science Sector

1.0 Evaluation purpose, context and scope

Purpose of the evaluation

This evaluation assesses the , performance, and design and delivery of Fisheries and Oceans Canada's (DFO) Ocean and Freshwater Science Contribution Program (OFSCP), which includes the Partnership Fund. It also examines components of eleven other grant and contribution (G&C) programs within the Ecosystems and Oceans Science (EOS) sector (see Annex A). Finally it covers the activities of the secretariat within the EOS sector that supports all G&C programs delivered under the Ecosystems and Oceans Science Contribution Framework (EOSCF) terms and conditions^{Footnote 1}.

Evaluation Context

In order to inform Senior Management, this evaluation was undertaken early and was designed as an implementation evaluation to help determine whether activities are being implemented as intended and to provide evidence on what is working well and if any adjustments are required. The Partnership Fund was the primary focus of the evaluation, but challenges are also highlighted for the other G&C programs listed in Annex A.

Evaluation Scope

The evaluation was conducted by DFO's Evaluation Division between April 2019 and June 2020. It covered the years 2016-17 to 2019-20 and included all DFO regions: Newfoundland and Labrador, Maritimes, Gulf, Quebec, Ontario and Prairie, Pacific, Arctic^{Footnote 2}, and National Headquarters.

The evaluation was conducted in accordance with the Policy on Results and Financial Administration Act, which require departments to measure and evaluate performance and use the resulting information to manage and improve programs, policies and services.

Information gathered from multiple qualitative and quantitative lines of evidence (interviews, document and websites review, literature review, file review, a survey to recipients, a survey to DFO staff and a site visit) was triangulated to respond to the evaluation questions. Details on the evaluation methodology are found in Annex B.

2.0 Program profile

The Ocean and Freshwater Science Contribution Program (OFSCP) was created in 2016 with $4,470,000 in the first year and $5,170,000 in 2017-18 in ongoing funds. The objective of this contribution program is to increase the pool of scientific knowledge aligned with DFO science priorities.

The Partnership Fund component of the OFSCP, was allocated $3,500,000 in 2016-17 and $2,500,000 in ongoing funds to promote and facilitate the development of scientific knowledge and related science activities by external organizations in areas aligned with DFO priorities. As of December 2019, there were approximately 118 Partnership Fund projects totalling just under $18,000,000.

Over the period of the evaluation, many new G&C programs within the EOS sector have brought the total sector contribution funding to approximately $25,000,000 as of 2018-19. More funding has since been added bringing the total value of sector G&Cs to around $35,000,000. Due to the high increase, a G&C secretariat was created within the sector to help support the exponential increase in contribution funding. It is staffed with approximately seven full-time equivalents (FTEs).

All the contributions are delivered under the Ecosystems and Oceans Science Contribution Framework (EOSCF) terms and conditions, formerly known as the Ocean and Freshwater Science Contribution Program terms and conditions before being revised in 2019.

Figure 1: Publicly reported spending under the OFSCP reflects the exponential growth in G&C funding in the EOS sector.

Description

This bar graph represents publically reported amounts spent under the Ocean and Freshwater Science Contribution Program and highlights the exponential growth in Grant and Contribution funding in the Ecosystem and Ocean Science sector from 2016-17 to 2018-19.

The amounts spent are: $4,273,033 in 2016-17, $15,432,471 in 2017-18, and $25,137,184 in 2018-19

Figure 2: Actual financial resources for the Partnership Fund for the years 2016-17 through to 2018-19.

Description

This bar graph represents actual financial resources for the Partnership Fund for the years 2016-17 to 2018-19. The bars include resources allocated to salaries, operating and maintenance, grants and contributions, and total financial resources.

In 2016-17 $39,732 was allocated to salaries, $1,274,667 was allocated to operating and maintenance, $3,542,587 was allocated to grants and contributions, and the total financial resources equalled $4,856,986.

In 2017-18 $284,163 was allocated to salaries, $402,652 was allocated to operating and maintenance, $8,423,111 was allocated to grants and contributions, and the total financial resources equalled $9,108,926.

In 2018-19 $424,618Kwas allocated to salaries, $354,420 was allocated to operating and maintenance, $2,526,852 was allocated to grants and contributions, and the total financial resources equalled $6,605,900.

3.0 Evaluation findings

3.1 Funded projects are addressing gaps in the ocean and freshwater science community

The ocean is of key importance to Canada's culture, environment, public health, and economy. It is also integral to the culture of many Indigenous and coastal communities. Although research in ocean science has made steady progress, certain areas remain poorly understood. A 2012 report^{Footnote 3} highlighted 40 priority research questions that, if answered, would have the greatest impact on addressing future opportunities and challenges relating to ocean science in Canada. These research questions can be grouped under the four areas indicated below.

Although most projects align with fundamental scientific understanding, respondents to both surveys perceive that the greatest gaps are related to the management and access to scientific data as well as understanding the impact of human activities.

When recipients were asked if their organization faced gaps for conducting activities and/or research, most who answered yes (58%) elaborated that lack of funding was a gap.

Interviews and survey results indicate most projects would not have taken place or would have taken place on a smaller scale (e.g., lower spatial coverage, longer timelines, less frequent sampling) in the absence of contribution program funding.

Further, just over half the respondents to the recipient's survey feel that there are no alternative sources of funding for this type of work; 30% said there were some, including provinces, industry and possibly the Natural Sciences and Engineering Research Council of Canada, but not all would be a good fit.

3.2 Partnership Fund projects align with the core responsibilities of the department

Proposals submitted to the Partnership Fund are evaluated by the Science Executive Committee on the extent to which they contribute to knowledge in key program areas of the department. The file review found that 98% of projects clearly align to program areas of the department, which in turn align to the DFO core responsibilities of Fisheries and Aquatic Ecosystems. Forty-three percent of the projects aligned with multiple program areas. These findings were supported by interviews.

Projects funded under the Partnership Fund generate more scientific data and results in the public domain which can be accessed by DFO staff for management decisions. Most EOS staff indicate they regularly access science research and data from external sources, including to support decision-making.

Figure 3: DFO staff utilize external science research and data to support their work.

Description

This bar graph shows the responses from Fisheries and Oceans Canada staff to a series of survey questions regarding the extent to which they use external data to support their work:
80% of staff use external science research to supplement Fisheries and Oceans Canada's data, 78% access data that Fisheries and Oceans Canada does not have, 75% use data to supplement a lack of Fisheries and Oceans Canada's capacity, 68% use data for decision making, 65% use data to inform scientific projects, and 60% use data to keep current in the field.

3.3 Partnership Fund projects are supporting intended results

The Partnership Fund projects, as of December 2019^{Footnote 4}, supported one or more intended results.

3.3.1 There is an increase in scientific research and related scientific activities conducted by external organizations (immediate outcome).

• Contribution agreements support an increase in scientific research that aligns with DFO science priorities.
• 61% of projects funded under the Partnership Fund were related to research and data transfer.
• 63% of all survey respondents said that the Partnership Fund supported this result from a considerable to a great extent.

A group of projects funded in DFO's Central and Arctic region (renamed the Ontario and Prairie region June 1, 2020) supported research related to the recovery of Species at Risk (SAR) in freshwater habitats. A network of scientists was established with Partnership funding in 2016. The network conducted a scoping exercise which identified there was a lot of research available on Species at Risk population ecology and habitat but not much on threats to species and reintroductions (i.e. recovery). Projects funded in years two and three of the Partnership Fund were informed by this gap analysis.

External partners with expertise and access to special laboratory environments, not readily available to DFO, were able to produce results aligning with the responsibilities of DFO under the Species at Risk Act.

3.3.2 Canadians have increased access to enhanced scientific results and data from funded scientific activities (immediate outcome).

• 96% of respondents to the recipient survey said that they had plans to share, or results were already being shared with the scientific community through various means.
• Respondents (81%) are also planning to disseminate their research more broadly with the Canadian public.
• Researchers engage in a variety of knowledge mobilization strategies. The most common form of knowledge dissemination are workshops or symposia.
• Research teams are also utilizing publications and social media to share project findings.

Figure 4: Most recipients use conferences, symposia, meetings, and workshops to disseminate information to Canadians

Description

This bar graph shows the survey responses from recipients regarding the methods being used or planned to share project results with Canadian public:
79% of recipients use conferences, symposia, meetings, and workshops, 50% use formal reports, 43% use websites, 36% use open data, 29% use press releases, and 7% use videos.

3.3.3 The Canadian ocean and freshwater science community has increased capacity and resources to conduct ocean and freshwater science (intermediate outcome).

• All contribution programs are supporting capacity building to some extent.
• Highly qualified personnel^{Footnote 5} are being trained from each of the following groups:
  • Post-doctoral researchers
  • Doctoral and Masters students
  • Undergraduate students
  • Community researchers
  • Indigenous community members
  • Technicians

Figure 5: Survey respondents, felt that yes, funded projects are contributing to the development of highly qualified personnel

Description

This donut charts represents the extent to which recipients and Fisheries and Oceans Canada's staff feel that funded projects are contributing to the development of highly qualified personnel.

93% of recipients agree that projects are contributing to the development of highly qualified personnel.

86% of Fisheries and Oceans Canada's staff agree that projects are contributing to the development of highly qualified personnel.

Moreover, recipients said that these projects helped to train the next generation of researchers (i.e., highly qualified personnel).

Recipients are benefitting in many ways from funded activities

• Survey evidence shows recipients have strengthened their networks through collaboration
• 78% of recipients felt that they had expanded their professional network through their funded work
• Survey evidence revealed that recipients expanded their knowledge (73%), skillset (70%), and developed new skills (42%)

Some projects delivered results for all three outcomes!

Highly qualified personnel were tracked for a group of 14 research projects funded in
DFO's Central and Arctic region. From 2017 to 2019, 39 highly qualified personnel were
trained through their involvements.

Outputs were also tracked for the same time period. For the same 14 projects, about 100 products were created or were in progress as of November 2019 and numerous public outreach activities took place.

3.4 Recipients provided positive feedback about the support and experiences they have had

Most recipients of the contribution programs are positive about the support they have received during different phases of the funding cycle.

• Guidance received during the preparation of proposals was easy to understand
• They knew what activities and expenses were eligible
• Templates were easy to use
• 67% of survey respondents felt that reporting requirements were reasonable to a considerable or great extent
• Partnership Fund recipients noted reporting is not onerous and is well-aligned with the amounts being funded

There is no evidence to suggest widespread delays

• Based on a review of 118 Partnership Fund projects, the average number of days to sign contribution agreements was 16 days (the range was 0 to 77).
• Some interviewees from academic institutions mentioned that administrative processes can be cumbersome at their end.

About half of the academic recipients interviewed indicated there is a mismatch between the DFO and the academic calendar years which may be why some recipients reported that funds are not transferred in a timely manner.

30% of recipients made suggestions about where the financial aspects of the contribution programs could be improved:

• There is a mismatch with the academic year impacting the ability to carry funds over between years which, in turn, makes it harder to begin projects or complete objectives as planned
• Too much detail is required in budgets (e.g. related to travel)
• There is a lack of flexibility in funding categories

Overall, recipients are satisfied with the level of communication they have with both national and regional DFO staff throughout the funding process. Communication was effective at:

• the drafting of contribution agreements;
• the notification of approval;
• the transfer of funds; and
• during reporting.

"The people that we worked with at DFO to help alleviate the issues with approval and funding transfers were incredible. They were accommodating and I have found them to be incredible contacts"

Some Partnership Fund recipients said communication challenges occurred when contribution agreements were being drafted by staff in the G&C secretariat. This might reflect a break in their communication with regional staff that were no longer involved in this stage.

3.5 Recipients complained that it is hard to find information on the EOS G&C programs

Recipients mentioned they do not know about the range of funding opportunities available in the Ecosystems and Oceans Science sector and that it is hard to find information about them. With only a couple of exceptions, most Partnership Fund recipients interviewed had not heard about any of the other contribution programs.

Figure 6: Recipients found it difficult to find information on the contribution programs.

Description

This bar graph shows the responses from recipients to the survey question: To what extent was it easy to find information on the contribution programs?

8% of recipients responded not at all, 28% to a limited extent, 30% to a moderate extent, 18% to a considerable extent, and 7% to a great extent.

One issue is a lack of online presence. It is cumbersome to publicly locate information on funding opportunities within the EOS sector. A review of websites for 12 other federal government departments found that G&C information was typically easy to find.

Most departments reviewed had a "button" on their home page to direct people to funding opportunities.  It then took two to three clicks to access G&C program pages where there was a wide range of information available. The most common information items provided were:

• frequently asked questions
• glossary
• eligibility criteria
• eligible projects
• activities and expenditures
• application toolkits
• contact information
• program description
• deadlines
• expected results

Having an online presence is one way to engage with applicants and recipients.

In 2019, DFO did a review of its grant and contribution programs and found there is limited G&C information available to potential recipients globally at DFO. The review recommended the department develop an online tool to search for information such as: contribution program descriptions, eligibility criteria, targeted recipient groups, application processes, and intake models. The Chief Financial Officer sector, where the G&C Centre of Expertise resides, supports this recommendation; like the EOS sector, the department has had exponential growth in G&C budgets over the last few years.

Web renewal is coming!

In 2019, the outreach group in the EOS sector initiated a process with corporate communications to renew their webpages. There is a plan to centralize information on the G&C programs so that interested applicants can access them.

3.6 Most recipients heard about the programs through informal networks

Under the Directive on Transfer Payments, departmental managers who have been assigned responsibilities for the management of transfer payment programs are responsible for "ensuring that potential recipients have ready access to information about transfer payment programs and that a description of each program is made public, including application and eligibility requirements and the criteria against which applications will be assessed."

The evaluation did not find evidence of standardized tools being used to communicate with recipients at different touchpoints in the funding cycle. When recipients were asked how they heard about the contribution programs, many replied they had heard through word of mouth.

Internal staff identified that most eligible recipients were made aware of contribution programs through direct/targeted calls to specific eligible recipient groups or individuals, and through word of mouth.

Figure 7: Recipient responses to: "How did you learn about the funding opportunities?"

Description

The bar graph represents how funded recipients responded to the question "How did you learn about the funding opportunities?" 59% of recipients heard through word-of-mouth from a Fisheries and Oceans Canada employee, 23% heard through word-of-mouth from their own networks, and 35% of recipients heard through an official notification from Fisheries and Oceans Canada.

Practices that support transparency and equitable access

• Conduct open, unrestricted calls at least occasionally. Response rates can provide information about contribution program issues, or alternatively, greater need.
• Provide robust online information about contribution programs.
• Identify funding available.
• Conduct webinars with eligible recipient groups to describe funding programs and opportunities.

A commonly voiced concern about open calls is they can place a burden on staff to review a high volume of proposals. One government department suggested this challenge can be managed by using expressions of interest as a filtering tool. Applicants can submit short letters of interest which can be reviewed quickly and used to identify projects to be developed into full proposals.

3.7 Service standards are being met

The Treasury Board Secretariat's Guideline on Service Standards specifies service standards for three aspects in the funding process:

1. receipt of application
2. notification of funding decision
3. issuance of payment

As a good practice, some other government departments have additional service standards, for example one business day to respond to general inquiries.

The Treasury Board Secretariat's Directive on Transfer Payments requires managers to ensure "transfer payments are paid to recipients in a timely, prudent and efficient manner."

Service standards were reviewed for 13 other government departments. None of the departments, including DFO, had a service standard notifying G&C recipients of approval of deliverables, i.e., for reviewing and approving deliverables on a timely basis.

Within the EOS sector, timely approval appears to happen in practice, but it is not an integral part of a formal service standard, against which performance is reported.

Figure 8: Service standard targets were met or exceeded in 2017-18 and 2018-19

Description

This figure shows the extent to which the Ocean and Freshwater Science Contribution Program is meeting service standards for the acknowledgement of the receipt of an application, the notification of the funding decision, and the requisition for payments.

The target for the acknowledgement of the recipient of an application is 5 days. The Ocean and Freshwater Science Contribution Program attempts to meet this standard for 90% of applications. In 2017-18 the Ocean and Freshwater Science Contribution Program met this target for 99% of applications and in 2018-19 the OFSCP met this target in 100% of applications.

The target for the notification of funding decisions is 45 days. The Ocean and Freshwater Science Contribution Program attempts to meet this standard for 90% of applications. In 2017-18 the Ocean and Freshwater Science Contribution Program met this standard for 95% of applications and in 2018-19 the Ocean and Freshwater Science Contribution Program met this standard in 100% of applications.

The target for the requisition of payments is 15 days. The Ocean and Freshwater Science Contribution Program attempts to meet this standard for 90% of applications. In 2017-18 the Ocean and Freshwater Science Contribution Program met this standard for 90% of applications and in 2018-19 the OFSCP met this standard for 100% of applications.

Source: Fisheries and Oceans Canada's service performance reporting for grants and contributions

3.8 Better communication and guidance are needed to support the delivery of EOS contribution programs

Staff in the EOS sector generally feel that roles and responsibilities are clear regarding the delivery of contribution programs. The secretariat is responsive in answering questions and is making continuous improvements. A few key informants noted that more staff are likely needed to handle the workload.

In terms of challenges, there are internal communication breakdowns both between staff associated with different contribution programs and also between National Headquarters and the regions. Some staff find it hard to get information outside of their contribution program. There is inconsistent messaging, and a lack of cohesion amongst the different programs in both G&C administration and literacy. The different contribution programs operate in silos which leads to duplications in the creation of tools (e.g. templates) or in processes such as financial tracking. Many employees feel they do not have access to the guidance they need to work efficiently.

Figure 9: Extent to which DFO staff agree it was easy to find information on contribution programs, the process to deliver contribution programs was well-documented, and instructions to deliver contribution programs were clear.

Description

This bar graphs represents responses of Fisheries and Oceans Canada staff to the following survey questions:

When asked to what extent they agreed it was easy to find information on the program 19% of staff responded not at all, 44% to a limited extent, 19% to a moderate extent 10% to a considerable extent, and 3% to a great extent.

When asked to what extent they agreed that the process to deliver contribution programs was well-documented 13% of Fisheries and Oceans Canada staff who responded to the survey responded not at all, 44% responded to a limited extent, 19% responded to a moderate extent, 9% responded to a considerable extent, and 6% responded to a great extent.

When asked to what extent they agreed that instructions to deliver contribution programs were clear 13% of Fisheries and Oceans Canada staff responded not at all, 56% to a limited extent, 22% to a moderate extent, 3% to a considerable extent, and 3% to a great extent.

According to responses provided in the DFO staff survey, two of the changes that would most help improve the delivery of the contribution programs are clear, step-by-step guidance documents, and standardized templates for use across all the EOS contribution programs.

• 88% of staff surveyed said that step-by-step guidance would help to improve the delivery of contribution programs.
• 84% of staff surveyed said that a webpage with information on EOS G&C programs would help to improve the delivery of contribution programs.
• 81% of staff surveyed said that standardized and centralized templates would help to improve the delivery of contribution programs.

Note: Above percentages combine respondents that agreed to a considerable and great extent.

3.9 Recent initiatives support improvements

Over the period of the evaluation, the G&C secretariat within the EOS sector has conducted reviews to highlight results and accomplishments, and to identify challenges, lessons learned and recommendations to address issues. Two recent initiatives that could address some of the current challenges are:

National Coordinators Committee

The first meeting has taken place with membership from regions and contribution programs in the EOS sector using the EOSCF terms and conditions. The committee is currently drafting terms of reference. The objectives are broad in scope, but are oriented towards improving:

• communications;
• overall G&C literacy; and
• standardization of guidance, practices and tools.

The delivery of training

The G&C secretariat within the EOS sector delivered their first training early in 2020. It was attended by regional representatives and contribution program leads. The focus was to clarify the application process and provide consistent, standardized messages to improve proposals' compliance with the terms and conditions. The secretariat reports, anecdotally, that the training helped increase the compliance rate of the most recent batch of proposals. The next training will be related to the SharePoint database that is being used as a central repository for projects funded under the EOS sector contribution programs.

Another significant change was a revision of the terms and conditions in July 2019. Among other changes, the name was updated to better reflect the range of  programs using them and to avoid confusion with the similarly-named contribution program (i.e. Ocean and Freshwater Science Contribution Program).

While the number of contribution programs within the EOS sector have greatly increased, the capacity of the G&C secretariat within the sector has not. There are seven full-time equivalents on the current organizational chart within the secretariat, however positions have not been fully filled since it was created. This presents challenges, especially at particularly busy times in the funding cycle. Insufficient human resources could limit the secretariat's ability to make changes that are needed, for example to improve services to all contribution programs and/or to recipients through outreach and communications.

3.10 The database in place to support the contribution programs needs improvement

Currently, the secretariat collects, and stores information related to funded projects in a database housed in SharePoint. The database was created as an interim system to support staff managing and administering contribution programs. While it provides a single storage space for data enabling access to information (e.g. project descriptions, contact information, project value, leveraging amounts, and contribution agreements), the following issues were identified:

1. A limited number of staff have access to the database.
2. Some information is stored outside of SharePoint (e.g. financial information, dates to support service standard reporting).
3. There are inconsistencies, particularly with financial information: contribution programs tracked their own information and it did not always match the information being tracked by the secretariat.
4. Some information in SharePoint is incomplete (i.e. missing documents, empty fields) reducing reliability.

These issues lead to overall inefficiencies. Staff are not able to quickly or confidently access information they require, leading to some duplication in processes and excess time investments as staff need to access more than one source for their needs.

The SharePoint database lacks data fields for performance-related data, making it difficult to assess the results of the projects. Performance information is available in some documents (e.g. final reports) however, based on the file review, only 69% of final reports and 56% of the evaluation reports (prepared by DFO scientists who review project final reports) were available in SharePoint for completed projects.

The secretariat recognizes some of the issues with the SharePoint database, and as a result has asked each contribution program to identify one person who will have "read and write" access. They will also expand the number of staff who have "read access" and the next training session will focus on the use of SharePoint.

As a possible long-term database solution, the G&C secretariat is examining the system used by the Atlantic Salmon Fund program. The information management system, called CFISH, is endorsed by the Chief Financial Officer's Centre of Expertise for grants and contributions; they may eventually consider establishing this enterprise-wide system to support the management and administration of transfer payment programs across the department.

Benefits of a department-wide G&C database

• Having all G&C recipient information in one place supports risk assessment (i.e. do they have an excellent, good or poor track record?) It also helps to ensure stacking limits aren't exceeded.
• Access to results information streamlines program reporting.
• Standardizes internal requirements for data entry and management.
• Streamlines the need for requiring repetitive information from recipients ("tell us once").
• Could allow for ability to generate contribution agreements with standard language thus providing a consistent recipient experience.
• Improves overall program efficiencies.

3.11 O&M funding available through the Partnership Fund is benefitting G&C programming in the EOS sector

The Partnership Fund was created with both a contribution component and an operations and maintenance (O&M) component. This model allocates funding to DFO researchers to facilitate projects, as well as provides funding to external recipients. Evidence shows that O&M funds were transferred to support 48 projects during the first three years of the Partnership Fund. Only 24 O&M reports were available in SharePoint for completed projects.

DFO staff could not think of any drawbacks of the O&M funds; they reported the following benefits:

1. Helped align projects with contribution program objectives.
2. Facilitated communication with recipients.
3. Allowed DFO staff to provide an advisory role in projects.
4. Enhanced their ability to understand project results.

Staff reported they used O&M funds to:

• Support recipients (e.g. providing samples for their analysis)
• travel to project-related workshops
• fund project management and administration

The analysis of O&M spending associated with the Partnership Fund shows that its use decreased over the last three years. Financial information obtained from the Chief Financial Officer sector indicates that actual amounts were less than planned expenditures.

Only 30% of those involved in the Partnership Fund who responded to the staff survey used O&M funding. Based on evidence collected, some internal staff facilitating Partnership Fund projects were not aware of the O&M component.

Surpluses of O&M funds were used to support the functioning of the G&C secretariat to help manage the exponential growth of G&C programming in the Ecosystems and Oceans Science sector and related Partnership Fund expenses.

3.12 Contributions reach all recipient groups through collaboration with others

Since 2016, about 80% of the projects and 84% of the funding for the Partnership Fund have gone to three of the 10 eligible recipient groups (see Annex C): academic organizations, followed by not-for profit groups and Canadian research organizations. As shown in Figure 10, all groups, except Port Authorities and community groups, have had some projects funded. It is interesting to note however, that in the recipient survey there were no respondents from Port Authorities or community groups, yet 17% and 19% of respondents said that Port Authorities and community groups, respectively, were involved as partners in their projects.

The contributions have greater reach than to funded recipients. Figure 11 shows that the contribution programs, including the Partnership Fund, reach all recipient  groups to some extent through collaborations with others.

Figure 10: Three eligible recipient groups received most of the Partnership Funding

Description

This bar graph represents the proportion of total funded projects associated with each of the eligible recipient groups.

Port authorities did not have any funded projects, foreign academic institutions made up 3% of funded projects, foreign government and non-governmental organizations made up 2% of funded projects, Canadian not-for-profit organizations made up 17% of funded projects, community groups did not have any funded projects, business and industry made up 3% of funded projects, Canadian governments made up 3% of funded projects, Indigenous organizations or groups made up 3% of the project, Canadian non-governmental organizations made up 14% of funded projects, and Canadian post-secondary institutions made up 48% of funded projects.

Figure 11: All eligible recipient groups were reached through collaboration with funded recipients.

Description

This bar graph represents the proportion of total projects in which eligible recipient groups were collaborators.

Out of the total number of projects Port Authorities collaborated in 17% of them, Foreign Academic Institutions collaborated in 19% of them, Foreign governmental and non-governmental organizations collaborated in 17% of them, Canadian not-for-profit organizations collaborated in 31% of them, Community groups collaborated in 19% of them, Business and Industry collaborated in 40% of them, Canadian Governments collaborated in 33% of them, Indigenous organizations or groups collaborated in 39% of them, Canadian non-governmental organizations collaborated in 35% of them and Canadian post-secondary institutions collaborated in 79% of them.

For the Partnership Fund, 185 unique collaborators were identified. Engagement with other stakeholders or partners was evident in 62 (53%) of the 118 projects reviewed. About half of the 62 projects had one partner and about half had 2 to 5 partners.

Canadian not-for-profit recipients had the widest range of partners.

Canadian governments, businesses, and Indigenous partners were more likely to be involved as partners rather than recipients:

• Three contribution agreements were signed with Indigenous organizations, and twelve Indigenous partners were identified with other recipients.
• There were two unique business recipients, and 21 business organizations represented were identified as partners in other projects.

3.13 Some eligible recipient groups could face barriers

Interviewees suggested that academic organizations have greater capacity to apply to G&C programs (e.g. past-experience, infrastructure) which improves their success rate.

Recipients felt that Port Authorities, international and Indigenous organizations might experience barriers to accessing contribution programs funding due to a lack of capacity (staff and time) and/or lack of networks to make them aware of funding opportunities.

The survey of recipients shows there are perceived barriers for all the contribution programs examined in the evaluation based on geographic location of work and size of organization.

Recent revisions to the terms and conditions added criteria to support 100% funding to Indigenous groups, small NGOs and community groups. These changes could address some barriers by improving these groups' willingness to apply as it removes the necessity and effort to find funding from other sources.

"Smaller recipient organizations have less staff and therefore less capacity to complete applications and reporting requirements."

Using direct calls for proposals and word of mouth methods to disseminate information could create unintended obstacles for some eligible recipients. For example, when recipients were asked if there were any barriers based on career status, 18% answered yes and elaborated that early career researchers would not have well developed networks affecting their ability to find out about or be approached about funding opportunities.

"The funding would not be accessible without established contacts within DFO who can notify of funding available, and help navigate the application process"

"As these networks tend to develop over a career, it is likely that early career individuals may not have the same opportunities. Enhanced advertising and notifications would address this issue."

Early career female researchers have been found to be less successful in research funding competitions than male applicants and seasoned researchers respectively. A review of federal science research programs in Canada completed in 2017 found that the community of early career researchers in Canada tends to be quite diverse, including more underrepresented populations such as immigrants, Indigenous people, and those with disabilities.^{Footnote 6}

3.14 Some Partnership Fund projects consider Gender-Based Analysis Plus factors

A review of all Partnership Fund projects (as of December 2019) showed that the contribution program approved a "handful" of projects where applicants considered GBA+ factors in their project design or objectives. One project was oriented towards empowering women in marine management of small island states; but most of these projects were focussed on Indigenous groups:

• Three projects were focussed on building capacity in Northern, remote Indigenous communities.
• One project covered the cost of travel from remote communities to allow 16 Indigenous persons to participate in a symposium focussed on rebuilding abundant fisheries.
• One project funded expensive analysis in remote communities where there would be direct benefits to Indigenous communities.
• One project was oriented to increasing First Nations Engagement in Fisheries Science and improving the integration of Indigenous knowledge.
• One project proposed the translation of deliverables into the relevant Indigenous language

In the context of grant and contribution programs, gender-based analysis plus (GBA+) seeks to understand if diverse groups of individuals in target populations (i.e. recipient groups) experience government programs equitably. It is difficult to answer this question without reliable data.

In 2018, the federal Tri-agencies^{Footnote 7} and the Canadian Food Inspection Agency began collecting self-identification data through the application process to their funding programs. A total of 39,326 forms were filled out. Responses were provided for an average of 94% of the applications (6% preferred not to answer). Data was collected on gender, Indigenous status, visible minority, and disability. This information will be used to [understand the population using and accessing their programs] and to inform decision-making for their programs to increase equitable participation in the research system.^{Footnote 8}

3.15 Good practices for the delivery of Grant and Contribution programs

To understand good practices in the delivery of G&C programs, the evaluation assessed G&C websites and reviewed service standards for 12 other government departments (OGDs); reviewed federal government materials related to G&Cs and documents from Natural Resources Canada's G&Cs Centre of Expertise; reviewed the findings of 12 evaluations of OGD G&C programs; and interviewed five program managers from G&C programs.

Good practices in delivering G&C programs fall into five categories:

• Pursue excellence in service delivery and client-focused G&C programs.
• Provide internal support at G&C program level.
• Develop streamlined, consistent and standardized processes and tools.
• Establish a process for risk-based assessment of projects.
• Support performance management and reporting.

Annex D details the good practices in delivering G&C programs.

4.0 Conclusions

As an implementation evaluation, evidence indicates that Partnership Fund activities are being implemented as intended. The Gs&Cs within the Ecosystems and Oceans Science sector are addressing gaps in the Canadian oceans and freshwater science community. Projects align well with DFO core responsibilities and there are early signs that immediate and intermediate outcomes are being achieved. As a result of funded projects, there is an increase in scientific research and related scientific activities conducted by external organizations; Canadians have some increased access to enhanced scientific results and data, and the Canadian ocean and freshwater science community is developing some increased capacity and resources to conduct relevant science.

Recipients are generally satisfied with their experiences with the contribution programs. Most indicated that they are not aware of other comparable funding programs for the types of projects they have undertaken. The main challenge is that eligible recipient groups are not aware of the funding opportunities in the Ecosystems and Oceans Science sector. There is a lack of publicly available, online information. Further, the methods used to inform potential recipients of funding opportunities, mostly by word of mouth, impact eligible recipients not connected to existing networks.

There are some internal challenges associated with the delivery of the contribution programs within the Ecosystems and Oceans Science sector. Staff associated with different ones feel they operate in silos. They indicate they would benefit from step-by-step guidance documents, standardized materials and templates to help them with many aspects of G&C program delivery. Further, staff experience inconsistent messaging and difficulty finding the information they need. For example, the database currently being used to centralize contribution program information is not fully available to all those who need access. This leads to duplication in some administrative processes.

The G&C secretariat within the EOS sector is investigating a more permanent database solution to support its contribution programs. The database being explored has the capacity to not only tackle the issues described above, but it would also support multiple good practices:

• wide access to more consistent, reliable information reducing duplication of internal processes;
• more consistent experiences for applicants (e.g. "tell us once") and recipients of DFO funding (e.g. ability to generate standardized contribution agreements); and
• more centralized and comprehensive information to support risk management and results-reporting.

A current initiative that will help address the difficulty eligible recipients have in finding information on EOS funding programs, is the collaboration between the EOS sector outreach team and DFO communications to renew public webpages, including information on G&C programs. Recipients will be more likely to find online information when they conduct web searches for funding opportunities. DFO staff will be able to provide better client service as they will be able to orient potential recipients to online information on EOS contribution programs.

Finally, two recent initiatives, the introduction of a G&C coordinators committee and training sessions will provide internal support to contribution program staff by providing consistent guidance and improving communication and G&C literacy.

Overall, the G&C secretariat within the Ecosystems and Oceans Science sector is responding well to the increase in transfer payment programs that have been added to the sector since 2016. It is demonstrating continuous improvement by identifying lessons learned and initiating changes to address internal issues, most of which align with challenges uncovered by the evaluation. Continuing to advance these initiatives will improve the delivery of grant and contribution programs across the EOS sector.

5.0 Annexes

Annex A: Contribution programs^{Footnote 9} examined in the evaluation

1. Ocean and Freshwater Science Contribution Program – including the Partnership Fund
2. Freshwater Research Contribution Program
3. National Contaminants Advisory Group Contribution Program
4. Whale Protection and Recovery Initiative Contribution Program
5. Coastal Environmental Baseline Program Contribution Program
6. Reducing the Threat of Vessel Traffic on Marine Mammals Contribution Program
7. Marine Environmental Quality Regulatory/Non-Regulatory Measures Contribution Program
8. Multi-partner Oil Spill Response Research Contribution Program
9. Ocean and Climate Change Science Contribution Program
10. Marine Spatial Planning Contribution Program
11. Freshwater Habitat Science Contribution Program
12. Terrestrial Cumulative Effects Initiative Contribution Program

Annex B: Evaluation methodology

Annex C: List of eligible recipients

The following groups have been identified as eligible recipients for the Ecosystems and Oceans Science Contribution Framework Terms and Conditions^{Footnote 10}:

• Canadian post-secondary academic institutions;
• Canadian non-governmental organizations;
• Indigenous organizations or groups;
• Canadian provincial, territorial, and municipal governments;
• Businesses and industry, including business and industry associations;
• Community groups;
• Canadian not-for-profit organizations;
• Foreign governments and international non-governmental organizations;
• Foreign Academic Institutions; and
• Port Authorities.

Annex D: Good practices for the delivery of Grant and Contribution programs

Good practices in delivering G&C programs can be summarized in five categories:

1. Pursue excellence in service delivery and client-focused G&C programs:

• Develop robust, client-oriented information and provide it on web-based platform.
• Provide online access for eligible recipient engagement processes (e.g. for applications).
• Minimize how often recipients are asked to provide the same information (i.e. support a "tell us once" experience) by collecting basic applicant/recipient information for access later. Prefill up-front sections of application forms.
• Offer support services to applicants during application process (e.g. guidance on what constitutes a strong application). Allow applicants to address questions raised by reviewers and support those who are not successful. Provide constructive feedback on applications offering areas for improvement.
• Develop thorough understanding of target groups i.e., potential applicants, including sub-groups (e.g. early career researchers who are women). Develop profiles to support targeted outreach strategies. Targeted outreach strategies may help small and medium-size enterprises navigate funding environment as they have limited capacity, time and experience in applying for funding.
• Promote funding programs and support services.
• Develop knowledge about other contribution programs, both internal and external to department. It may be beneficial to know about provincial/territory funding as well. This information can be provided to target groups to help them navigate similar federal science and research funding to achieve a good fit between project objectives and federal funding programs.
• Consider addition of services standards for timely response to questions and internal timely review of deliverables.

2. Provide internal support at G&C program level

• Ensure well-defined roles and responsibilities for all groups involved in the delivery of G&C programs.
• Develop and provide:
  • various guidance documents (e.g. application guides, monitoring projects and reviewing project reports, interpretation of Transfer Payment Policy, when to use Grants versus contribution versus contract; stakeholder engagement)
  • training (e.g. for use of standardized templates and tools)
  • a community of practice to support consistent messaging and facilitate information sharing.

3. Develop streamlined, consistent and standardized processes and tools to:

• Support internal and external efficiencies, and a common recipient experience.
• Reduce administrative and reporting burden on recipients.
• Provide training on how to use them.
• Standardize the process for generating contribution agreements

4. Establish a process for risk-based assessment of projects so:

• High-risk projects are identified and managed appropriately;
• An appropriate balance between flexibility and control is supported; and
• Cost-effective oversight, innovation and sensible risk-taking are supported.

• Establish risk management across all contribution programs based on criteria related to recipients (e.g. past experiences with them) and proposals (e.g. compliance with program objectives).
• Create standard templates to summarize risk information.
• Use results to support planning as well as flexibilities (e.g. reporting requirements, level of approval).
• Share risk information across programs.

5. Support performance management and reporting

• Use a systematic approach to documenting long-term results and close-ended questions in reporting templates to help increase the clarity and availability of performance information.
• Ensure performance data availability, validity and accessibility.
• Standardize the amount of information required and format in which it is to be presented.
• Establish performance management across all programs and use results to support planning and reporting.