EVALUATION OF THE ACADEMIC RESEARCH CONTRIBUTION PROGRAM

FINAL REPORT
JANUARY 23, 2015

EVALUATION DIRECTORATE


Table of Contents

ACKNOWLEDGEMENTS AND ACRONYMS


ACKNOWLEDGEMENTS

The Evaluation Directorate would like to thank all individuals who provided input to this study of the Academic Research Contribution Program. In particular, the Directorate acknowledges with gratitude the time and effort of key informants who shared information and opinions during interviews.

ACRONYMS


List of acronyms
ARCP Academic Research Contribution Program
CSAS Canadian Science Advisory Secretariat
DFO Fisheries and Oceans Canada
G&C Grants and Contributions
NSDC National Science Directors Committee
O&M Operation and Maintenance
RDS Regional Director of Science
T&C Terms and Conditions

EXECUTIVE SUMMARY


INTRODUCTION

This report presents the results of an evaluation of the Academic Research Contribution Program (ARCP) led by the Evaluation Directorate in Fisheries and Oceans Canada (DFO) and covering the operation of the program during the five-year period from 2009-10 to 2013-14. The evaluation assessed the core issues of relevance and performance, including effectiveness, efficiency and economy.

The ARCP is a generic funding mechanism that DFO science programs may use, providing that they identify funds within their own budgets, to support academic research of relevance to DFO research priorities. While the ultimate accountability for this program lies with the Assistant Deputy Minister (ADM), Ecosystems and Oceans Science, specific accountabilities are split between the ARCP program management office, which is responsible for providing oversight of the financial aspects of the ARCP contribution agreements and for submitting eligible proposals to the program governance committee for their approval, and the Regional Directors of Science (RDS), who initiate,  fund and manage specific ARCP contribution agreements.  

A small-scale evaluation approach was chosen for this evaluation in light of the program’s low level of risk and materiality (i.e., amounts of funding disbursed over the past five years) and the facts that the program has previously been evaluated, has collected a certain amount of performance data, and has a simple delivery model. The methodology included a review of program documents, ten key informant interviews, analysis of program data, and a review of final reports for a sample of ten completed projects.

Key Findings and Conclusions

The evaluation found that the program remains relevant, as there is evidence of a continuing need for a centrally governed mechanism that enables DFO science programs to draw on academic research on topics of relevance to the department’s mandate. The program also aligns with government priorities for academic research, responsible resource management and conservation of fisheries, and contributes to the expansion of the knowledge base required for the achievement of DFO’s strategic outcomes. In addition, the ARCP aligns with responsibilities of the federal government, as it is designed to enable research in support of the entire range of departmental outcomes and ministerial responsibilities, as outlined in DFO’s enabling legislations.

Evidence also suggests that the program is successful in achieving its intended outcomes, particularly in enabling high quality academic research that meets the needs of DFO science programs and enhances the quality of science advice for DFO decision-makers.

The program was found to be generally well administered and interviewed DFO scientists are satisfied with the knowledge generated by ARCP projects.

Program implementation is overall compliant with established Terms and Conditions (T&C) except that the existence of the ARCP is not systematically and widely communicated to potential funding recipients. It is however unclear whether this activity is advisable given that there are no pre-determined research topics or budget attached to the ARCP.  The program would however need to explore ways to ensure that the selection of funding recipients is not unduly limited to a too narrow pool of known academic researchers and to reduce risks that there could be a perception of favoritism in funding allocation decisions.

Evaluation findings also showed that the governance of the program is not functioning as might be expected. Meeting minutes revealed that ARCP funding proposals have not been the subject of discussions among National Science Directors Committee (NSDC) members and no evidence was found of their engagement in monitoring program performance.

Several improvements were made to the program’s administrative tools since the last evaluation, namely the development of an updated logic model and the implementation of a project summary report form. A few additional improvements could however be made in order to further strengthen the program’s management and implementation.

First, the evaluation found that there are no written operating procedures for administering the ARCP program, which puts the department at risk should its current vital corporate memory be lost due to retirements or staff turnover.

More generally, there is limited and unclear documentation of the program’s specific mandate, delivery mechanism, and roles and responsibilities of program administrators, senior management and DFO science program clients. The program’s particular role as a funding mechanism should be clarified to demonstrate its compliance with the Transfer Payment Policy and more clearly explained and communicated to program stakeholders.

Finally, while recognizing that several improvements have already been made in those respects, the program would benefit from further expanding its project reporting requirements to include additional financial and results information, and from ensuring more systematic central file keeping. This would better position the program to address potential information requests by management, to produce evidence for future audits and evaluations, and to enable access to research results for potential future use by other DFO science programs.

Overall, the program appears to be operated efficiently and economically, the funded projects resulting in good value for the department.

Recommendations

Evaluation findings and conclusions led to the following recommendations: 

Recommendation 1: It is recommended that the ADM, Ecosystems and Oceans Science, consult with NSDC members in order to ensure that appropriate means are in place for identifying potential project proponents, including a consideration of the pros and cons of formally and purposefully communicating the existence of the program to a wider community of academic researchers beyond those already known to DFO scientists.

Recommendation 2: It is recommended that the ADM, Ecosystems and Oceans Science, ensure that members of the NSDC engage in discussions for all ARCP proposals and conduct a review of the program’s performance on an annual basis.

Recommendation 3: It is recommended that the ADM, Ecosystems and Oceans Science, ensure that ARCP reporting guidelines be updated in order that all financial and results information essential for effective monitoring of program performance be captured, presented to the NSDC, and maintained in the program’s central filing system.

Recommendation 4: It is recommended that the ADM, Ecosystems and Oceans Science, ensure production of clear and complete documentation covering all essential aspects of ARCP governance and administration, including roles and responsibilities of program administrators, senior management, and DFO scientists, and standard operating procedures for the administration of the program. It is also recommended that the program T&C and logic model be revised accordingly and to demonstrate compliance with the Transfer Payment Policy.

1. INTRODUCTION


1.1 PURPOSE AND SCOPE OF THE EVALUATION

This report presents the results of an evaluation of the Academic Research Contribution Program (ARCP) led by the Evaluation Directorate in Fisheries and Oceans Canada between June and December 2014. In accordance with section 42.1 of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009), the evaluation assessed the core issues of relevance and performance, including effectiveness, efficiency and economy.

Launched in 2004-05, the ARCP was first evaluated in 2009. That evaluation was generally positive. The four recommendations that emerged from the evaluation and the ARCP management’s response to them are provided in Annex A.

The present evaluation covers the operation of the program during the five-year period from 2009-10 to 2013-14.

1.2 REPORT STRUCTURE

The Executive Summary provides an overview of the evaluation’s purpose, methodology, findings and recommendations. Following this section’s introduction to the evaluation report, Section 2 presents information on the ARCP and Section 3 describes the evaluation methodology. Evaluation findings are presented in Section 4; conclusions and recommendations are in Section 5. Annexes provide information to complement that presented in the body of the report.

2. PROGRAM PROFILE


2.1 MANDATE AND OBJECTIVES

The ARCP mandate stems from Terms and Conditions (T&C) that were approved by Treasury Board in 2004 and modified in 2009-10. The current T&C came into effect in April 2010.

The program’s fundamental objectives, as stated in the T&C, are:

  • to enhance DFO's capacity to deliver its mandate as well as its ability to meet future scientific challenges in areas relevant to DFO’s science program and priorities;
  • to leverage DFO's science capabilities through access to academic resources and knowledge;
  • to support DFO's research and development (R&D) programs, promoting innovation in marine, freshwater, fisheries, and aquaculture industry sectors; and
  • to allow DFO to work with academia towards realizing Canada’s full potential in marine, freshwater, fisheries, and aquaculture research by building linkages to support and enhance the complementary strengths of government, and universities in Canada’s innovation system.

2.2 ACTIVITIES

The core activities of the ARCP program are the review of proposals for DFO contributions to academic research projects, the providing of authority to enter into contribution agreements with participating universities, and the review of final reports on completed projects to determine if the objectives of the agreements have been achieved.

While the ultimate accountability for this program lies with the Assistant Deputy Minister (ADM), Ecosystems and Oceans Science, specific accountabilities are split between the ARCP program management office, which is responsible for providing oversight of the financial aspects of the ARCP contribution agreements and for submitting eligible proposals to the program governance committee for their approval, and the Regional Directors of Science (RDS), who initiate, fund and manage specific ARCP contribution agreements.1

Proposal preparation begins after a DFO science program has identified a need for academic research to meet current or future information needs in his or her area of responsibility and has located operation and maintenance funds to pay for the research. The DFO scientist or manager in charge of the science program in question makes contact with an academic colleague capable of conducting the required research. Together they prepare a proposal for consideration by members of the National Science Directors Committee (NSDC) in DFO. The proposal is submitted to the ARCP program management office for an eligibility screening and, if eligible, is sent secretarially (by email) to NSDC members for review.

Contribution agreements are prepared and negotiated once NSDC members have deemed the proposal acceptable. Once a draft agreement has been finalized to the satisfaction of all parties it is signed by the ADM, Ecosystems and Oceans Science and by appropriate persons at the researcher’s university.

The academic research then proceeds, usually monitored by the DFO scientist or manager responsible for the science program that funded the research. Payments are approved by the responsible RDS or Director General, subject to recipients delivering progress reports and statements that satisfy the terms of the ARCP agreement.  Upon completion of the research project, a final report is produced by the recipient and submitted to the responsible RDS or Director General for release of the final payment. Based on the final report submitted by the recipient, the DFO scientist or manager responsible for the science program prepares a summary report, using a standard form provided by the ARCP management office. The standard form requires, among other types of information, the purpose of the project, a brief description of the project, and its linkage to DFO research priorities. The form also contains an annex that seeks to assess the project’s performance according to a list of indicators identified for the program’s five immediate outcomes. The ARCP management office subsequently forwards these reports to the NSDC.


1 DFO has six regions from which an ARCP contribution agreement can be initiated: Pacific, Central & Arctic, Gulf, Maritimes, Quebec, and Newfoundland & Labrador. Contribution agreements can also be initiated by a Director General responsible for a science directorate in the National Capital Region (NCR).

2.3 LOGIC MODEL AND PERFORMANCE MEASUREMENT

The performance measurement framework that was provided in the 2010 T&C for ARCP included a logic model, which is attached as Annex B. The main (immediate) intended outcomes of the program are:

  • Academic programs relevant to DFO science priority areas
  • R&D activity in areas related to DFO science priorities areas
  • Leveraging of DFO science capabilities as a result of resources and knowledge contributed by academic institutions
  • Interaction and collaboration between DFO and the Canadian academic research community
  • Partnering with industry and stakeholders in support of DFO R&D programs2

2 This outcome was not examined as part of the current evaluation because initial consultations revealed that none of the program’s activities are clearly linked to the achievement of this result.

2.4 STAKEHOLDERS

The DFO scientists involved in preparing proposals and managing agreements are the primary clients of the program. They normally report to a DFO science manager, an RDS or DG who manages the budget from which contribution payments will be issued.  Other key stakeholders include the ARCP program management office which screens project proposals for eligibility and ensures that contribution agreements meet the requirements of the Transfer Payment Policy and respect the parameters laid out in the program T&C; the participating academic researchers and their institutions; the Chair and members of the NSDC, who are tasked with reviewing and recommending approval of academic proposals; and the ADM of Ecosystems and Oceans Science who holds ultimate accountability for the ARCP.

2.5 GOVERNANCE

The NSDC3, chaired by the ADM for Ecosystems and Oceans Science, is the principal governance mechanism for the ARCP. The ADM is ultimately responsible for ARCP governance.

The NSDC is comprised of the ADM, the Director General of Strategic and Regulatory Science, the Director General of Ecosystems Science, the Director General responsible for Canadian Hydrographic Service and Oceanographic Services, and RDS from the six DFO regions.

The Director General for Strategic and Regulatory Science serves as the senior manager for the ARCP management office and the Manager of Intellectual Property and Partnering serves as the ARCP program manager4.


3 In September 2014 the NSDC was renamed as the Science Executive Committee (SEC).

4 Given the small number of ARCP proposals submitted in a given year, the ARCP program management office has operated by using only a portion of the time of the Manager of Intellectual Property and Partnering.

2.6 RESOURCES

There is no budget attached to the ARCP. The funds that flow through the ARCP as contributions to academic research projects come from the budgets of science programs in DFO regions or central science directorates.

Resources for direct management of the ARCP, including ensuring compliance of academic proposals with program requirements and setting-up contribution agreements, have in recent years been relatively small as a result of low use of the ARCP by the science programs, and come from the salary budget for the Strategic and Regulatory Science Directorate. Resources needed for developing proposals with academia and managing existing agreements come from budgets in the region or directorate that is funding a given academic research project.

The T&C for ARCP allow no more than $1.6 million to flow through the program in a given year. Table 1 shows that over the past five years the flow of contributions has not come close to the allowable limit.

Table 1: ARCP Contribution “Budgets” and Expenditures 5

Table 1
  Thousands of Dollars (to nearest thousand) %
2009-10 2010-11 2011-12 2012-13 2013-14 5-year total
Maximum Allowable Contribution 1,600 1,600 1,600 1,600 1,600 8,000 100.0
Actual Contributions 797 596 584 275 170 2,422 30.3

5 Source: Program database maintained in the ARCP Managing Directorate.

3. EVALUATION METHODOLOGY


3.1 SCOPE

In determining the scope of the evaluation, the Evaluation Directorate took into account the relatively small amount of DFO funding that has flowed through the ARCP to university research over the last five years as well as the low level of risk.6 It also considered the facts that the program has previously been evaluated, has collected a certain amount of performance data, and has a simple delivery model. In light of these factors, a small-scale evaluation approach was deemed appropriate.

A senior evaluation manager engaged by the Evaluation Directorate conducted the evaluation. Directorate staff critically reviewed drafts of key evaluation documents. ARCP management provided essential assistance for the evaluation by identifying major documents, making themselves available for discussions, providing feedback on evaluation instruments, encouraging key informants to participate, and reviewing draft reports.


6 ARCP expenditures constitute a very small proportion of total DFO spending on science. The Expert Panel in Ocean Sciences has estimated that in 2010-11 DFO science spending was approximately $250 million. The expenditures through ARCP in that year, $596 thousand, would constitute 0.24% of the total DFO science expenditure, that is, about one quarter of one percent.

3.2 EVALUATION APPROACH AND DESIGN

The study set out to determine the evaluation issues, articulate the questions that needed to be answered, collect data to address the questions, analyse the results and develop recommendations.

First, a clear understanding of the ARCP was sought through background readings, discussions with managers and iterative examination of various models of the program theory. Then, starting from the basic evaluation issues specified in the Treasury Board (2009) Policy on Evaluation, questions were framed to address each issue within the context of the program. The data required to answer each question was identified along with the sources from which it could be obtained and the appropriate data collection methodology.

None of the questions required an experimental design. When possible, the validity and reliability of findings with respect to each question was strengthened by using data from multiple lines of evidence.

3.3 KEY ISSUES AND EVALUATION QUESTIONS

Annex C presents the core issues, the specific evaluation questions that were addressed, sources of evidence and data collection methods. The core issues were relevance and performance including effectiveness, efficiency and economy. These issues were addressed through 12 broad questions, each of which was examined through sub-questions that were articulated in the evaluation instruments.

3.4 DATA COLLECTION METHODS

The evaluation used the following methods to collect data: key informant interviews; analysis of administrative data; examination of final project reports; and, review of documents.

3.4.1  Key Informant Interviews

A total of ten (10) key informant interviews were conducted. Key informants were identified using purposive sampling, as follows:

  • Four (4) interviewees were DFO users of the program (i.e., DFO scientists),  selected to represent regions or directorates having made high, medium, low or no use of the ARCP;
  • Two (2) interviewees were from ARCP management, selected for their knowledge of the workings of the program; and
  • Four (4) interviewees were project funding recipients, selected to ensure representation from a large research network, a project with a social science component, and an applied laboratory project.

Two of the interviews were conducted in person at DFO headquarters in Ottawa. The remaining eight were by telephone. Most interviews involved 50 to 60 minutes of discussion in the official language of choice of the respondent.

3.4.2  Analysis of Program Data Files

A master spreadsheet maintained by the program provided data on project title, participating institution, DFO cash contribution by region and year, and future year commitments. Sub-sheets indicated project manager, date of signature of agreement, and milestone payments. Data was used for calculations to answer specific evaluation questions.

3.4.3  Examination of Reporting on Completed Projects

Ten of the 28 ARCP projects that were completed in the last five years were selected at random. For each of the 10 projects the centrally maintained folder of final project summary reports was searched. If a final summary report was located, it was examined for completeness.

3.4.4  Document Search and Review

Information was extracted from program documents and publications related to ARCP activities. The Web was searched as necessary to learn more about DFO science activities and institutions, DFO policy for scientific data, Treasury Board Transfer Payment Policy, profiles of academic research partners, and fisheries and oceans research projects.

3.5 ANALYTICAL METHODS

Analysis of qualitative data involved transcribing notes, organizing text and then extracting general themes and specific messages with respect to a given question or set of documents. Quantitative analysis involved crosschecking data, creating new variables, programming required calculations, graphing and calculating statistics.

3.6 LIMITATIONS AND MITIGATION STRATEGIES

The main limitation to this study stems from the limited number and range of key informants interviewed. Specifically, no representatives from the program’s senior management governance structure and few representatives from the academic institutions having received ARCP contribution funding were interviewed. This limitation was deemed acceptable in light of the low risk presented by the program, the availability of results from a previous evaluation, and the availability of program administrative data.

4. MAJOR FINDINGS


4.1 RELEVANCE

The evaluation examined whether the ARCP addresses an essential need and is aligned with federal and departmental priorities and responsibilities.

4.1.1  Need for the ARCP

Key Findings

There is a continuing need for a centrally governed mechanism that enables DFO science programs to draw on academic research on topics of relevance to the department’s mandate.

The need for DFO to have access to academic research is increasing as the department focuses on understanding complex ecosystems requiring a multi-disciplinary effort that involves a broader range of expertise that is not always available within the department. DFO needs to access expertise available in Canadian universities and continues to require a mechanism that will allow it to do so.

The need to access external research expertise is further heightened as DFO’s internal science capacity is diminished by the retirement of senior scientists and reductions in departmental budgets.

The central governance of the program provided by the NSDC is a needed mechanism to ensure oversight over the allocation of limited federal government resources to science research areas of greatest priority for the Ecosystems and Oceans Science and the department.

The role of the ARCP management enables consistent adherence to the program T&C and to Transfer Payment Policy requirements7, as well as consistent oversight over the financial aspects of ARCP proposals and contribution agreements.


7 A transfer payment is a monetary payment, or a transfer of goods, services or assets made, on the basis of an appropriation, to a third party, including a Crown corporation, that does not result in the acquisition by the Government of Canada of any goods, services or assets. Source: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=13525&section=text

4.1.2  Alignment with Federal Government and Departmental Priorities

Key Findings

The ARCP aligns with government’s priorities for academic research, responsible resource development and conservation of fisheries. The ARCP contributes to the expansion of the knowledge base required for the achievement of DFO’s strategic outcomes.

Responsible resource development is a stated priority of Canada’s Economic Action Plan. In its 2014 federal budget the government mentioned conservation of fisheries as a priority. Government boosted academic research by providing the federal granting councils with the largest increase in over a decade.8

ARCP enables academic research that supports DFO’s efforts to achieve economically prosperous maritime sectors and fisheries, sustainable aquatic ecosystems, and safe and secure waters, while responding to priorities of the Canadian government. Information obtained by DFO through ARCP projects contributes to the development of science advice, which in turn contributes to the evidence-based management of departmental activities through which DFO strives to attain its strategic outcomes.


8 Canada’s Economic Action Plan - Priorities, http://actionplan.gc.ca/en/page/priorities; Canada, Prime Minister of Canada Stephen Harper Priorities: http://www.pm.gc.ca/eng/priorities; Canada, Economic Action Plan 2014 – the Budget Speech: http://www.budget.gc.ca/2014/docs/speech-discours/2014-02-11-eng.html.

4.1.3  Alignment with the Responsibilities of Government

Key Findings

The ARCP is designed to enable research in support of all three departmental strategic outcomes. As such, it aligns with responsibilities of the federal government, and specifically the Minister of Fisheries and Oceans Canada, as laid out in the Department of Fisheries and Oceans Act, the Oceans Act and the Fisheries Development Act.

The program T&C state that the ultimate outcome of the ARCP is a body of scientific knowledge that will contribute to all three DFO strategic outcomes: Economically Prosperous Maritime Sectors and Fisheries; Sustainable Aquatic Ecosystems; and Safe and Secure Waters. The program draws its authorities from three enabling legislations: the Department of Fisheries and Oceans Act, the Oceans Act and the Fisheries Development Act.  These legislations specify the responsibilities of the Minister of Fisheries and Oceans Canada for activities related to the achievement of these strategic outcomes. For example, under Section 42(d) of the Oceans Act, the Minister may, among other responsibilities, “conduct basic and applied research related to hydrography, oceanography and other marine sciences, including the study of fish and their supporting habitat and ecosystems”.

4.2 PERFORMANCE


4.2.1    Effectiveness

The evaluation examined the extent to which the program has achieved its expected outcomes.

Key Findings

Evidence suggests that the results of ARCP projects are of high quality and are useful when preparing science advice within DFO, thereby increasing DFO science capacity.

Twenty-eight ARCP projects were completed in the last five years. All 28 projects were directly related to issues involving Canada’s fisheries and oceans.

Key informants described a variety of activities that help ensure the validity and reliability of ARCP research. The common thread is peer review, that is, critical examination of data and documentation by other experts. There will usually be ongoing communication between the academic researcher and the scientist in DFO who is managing the contribution agreement. After a project has been completed, academic researchers will often publish research results in peer-reviewed journals. In preparing science advice on specific DFO issues, the Canadian Science Advisory Secretariat (CSAS) uses peer review to assess relevant information that had been generated internally or obtained from external academic research.

Evaluation evidence indicates that knowledge obtained through ARCP projects may be for immediate use or may serve as a foundation for likely future research needed for scientific advice. The ARCP-funded network on integrated multi-trophic aquaculture provides an example of research that was primarily intended to broaden the DFO knowledge base and has since been used to address a specific need for advice. The network was launched in 2010. Two years later, in 2012, DFO scientists and academic participants in the network figured prominently in a regional peer review of integrated multi-trophic aquaculture that led to a science advisory report on the topic in 2013.9


9 The Terms of Reference for the peer review are available at the following link: http://www.dfo-mpo.gc.ca/csas-sccs/Schedule-Horraire/2012/10_03-05b-eng.html. The science advisory report is available at http://www.dfo-mpo.gc.ca/csas-sccs/Publications/SAR-AS/2013/2013_056-eng.html

4.2.2  Efficiency and Economy

The evaluation examined the extent to which the program was implemented efficiently and economically.

Adherence to Program Terms and Conditions

Key Findings

The program does not systematically and widely communicate its existence to potential funding recipients. It is however unclear whether this activity is advisable given that there are no pre-determined research topics or budget attached to the ARCP.

The program does not fully adhere to its T&C in one respect. In the logic model attached to the T&C, one program activity is said to consist of “Communicating availability of program”. The evaluation found that there are no mechanisms or systematic processes to communicate the existence of the program to potential funding recipients and it is unclear whether communicating program availability is a responsibility of program management or of regional science directors and director generals.  Current program practices limit the pool of potential contribution recipients to academic researchers already known to DFO scientists. The T&C do not require that a competitive process be followed for soliciting academic proposals.

The relevance of communicating the program’s availability is however unclear. On the one hand, it might be inappropriate to advertise the mechanism’s availability given that it has neither specific earmarked funds nor a set of pre-defined research topics for which academic researchers could submit proposals. Such advertisements might create expectations that the department might not subsequently be able to meet.

On the other hand, if the program is not communicated to the academic community at large, academic researchers may not all have the opportunity to make their research activities and areas of expertise known to DFO scientists, thereby preventing DFO scientists from drawing from a complete pool of potential researchers. The potential contribution funding applicants will remain limited to those academic researchers already known to DFO scientists.

Also, several key informants among academic researchers suggested that the program could appear to reinforce existing relationships to the detriment of potential new ones because DFO scientists seeking required expertise will naturally tend to make contact with academics that they know well. If the mechanism underlying the ARCP is not known and understood by all DFO scientists and by all academics, the flow of funds from DFO to any university could be attributed to a personal relationship between DFO scientists and academic researchers.

There are justifications for DFO not to rely on a competitive process when selecting academic researchers/institutions for given priorities (for instance, if the scientific field corresponding to a given priority is so narrow that there is only one academic research team qualified to conduct research in support of that priority) but there is no systematic process or requirement for documenting such justifications.

Program Governance and Management

Key Findings

While daily administration of the program appears appropriate, the governance body responsible for program oversight is less engaged than expected. The absence of written operating procedures for administering the ARCP program also poses a risk for program management.

According to program T&C, the program is to be managed (governed) by the NSDC and administered by a unit to be designated by the ADM, Ecosystems and Oceans Science. Key informant interviews revealed no indication of dissatisfaction with ARCP contribution agreements and all ten key informants had positive views about the program.

Evaluation findings showed, however, that the governance of the program is not functioning as might be expected. Meeting minutes revealed that ARCP funding proposals have not been the subject of discussions among NSDC members. The proposals submitted to them each year have been reviewed and approved independently by NSDC members via e-mail. Overall, the ARCP requires only a small portion of the time of members of the NSDC and occupies a negligible portion of the committee’s agenda. The ARCP program manager is rarely invited to sit in on NSDC meetings and the NSDC has not requested any reports on overall ARCP performance over the past five years.

This governance challenge was recognized when the ARCP was first evaluated in 2009. The evaluation report included a recommendation for the creation of an evaluation committee that would have focused attention on program performance at least once a year. Management did not take up the idea, partly because it was hesitant about creating another committee and partly because it believed that improved final reporting on projects would serve the same purpose. However, the present evaluation found that governance of the program by NSDC continues to be problematic.

Finally, the evaluation found that there are no written operating procedures for administering the ARCP program, which puts the department at risk should its current vital corporate memory be lost due to retirements or staff turnover.

Documentation of Processes and Reporting Requirements

Key Findings

The program has demonstrated a sound project management infrastructure and practices. There are however a few areas requiring further improvements: clarifying roles and responsibilities, formalising the use of proposal assessment criteria that link funded projects to DFO science research priories, and strengthening the program’s project reporting requirements and file keeping practices.

Unclear Roles and Responsibilities

The above-noted issues regarding governance of the program might be linked in part to the finding that roles and responsibilities of ARCP governors, managers and program participants are not well defined. The T&C for ARCP make a start at documenting the terms of reference for various program stakeholders by indicating responsibilities for data collection and reporting in the performance measurement strategy.  However, that information is not easily accessible and there is no documentation of roles, other than performance measurement, for NSDC members, ARCP management, or the DFO staff directly involved in the execution of contribution agreements.

Need for Formal Proposal Assessment Criteria

Given the oversight role of the NSDC, its members need evidence that a proposal has been duly considered within the region or central directorate which will be providing funding for the proposed research, and that the proposed research activities are feasible and related to DFO’s mandate and DFO science research needs. They also need reasonable evidence that the proposed academic team is appropriately qualified to perform the research.

The program does not have a documented set of criteria, or process to develop and update such criteria, by which NSDC members are to assess the merits of each proposal, including against DFO science research priorities.

The 2009 evaluation included a recommendation that management should “Develop and apply selection criteria for the selection/refusal of ARCP proposals”. In response, management devised a form by which the proposers of projects would indicate the potential impact of the research through a Yes or No with respect to each of 12 indicators. If at least six areas of potential impact had been checked with a Yes, the proposal would be submitted to the NSDC for consideration. However, the form is an administrative screening tool rather than a set of criteria to be considered by the NSDC when reviewing proposals.

Need for Strengthened Project Reporting and File Keeping

Current program guidelines require DFO science managers to submit final project summary reports to NSDC using a standard report form. This report form captures most of the essential information and represents a notable improvement since the previous evaluation. It is nevertheless missing a number of key elements of information that are essential for NSDC members to assess the extent to which the funded projects have contributed to the achievement of program intended outcomes and offered value-for-money to the department. Missing elements of information include:

  • the final budget received for the project, including the cash and in-kind contributions of DFO, the university and any other partner; and
  • a summary of the research findings or results (as opposed to solely a description of the project).

Also, the amount of information provided in the final reports examined varies widely and does not always appear commensurate with the size of the project.

In addition to supporting the NSDC’s program performance monitoring role, this information would assist the sponsoring science programs in demonstrating achievement of outputs and outcomes when the program will be evaluated, as well as enable access to research results for potential future use by other DFO science programs.

Also essential to a program’s effective performance reporting and knowledge dissemination is the storage of all information pertaining to a given funded project in one central location. This is particularly relevant for a program such as the ARCP, where most project information (e.g., financial summaries, interim and final project reports by funding recipients) is collected by different DFO scientists in the regions. Potential information seekers (e.g., senior management, auditors, evaluators, other DFO scientists) should be able to access all the information on one or several projects from one single source rather than having to contact each separate DFO scientist having managed an ARCP contribution agreement.

A review of project files kept centrally by the ARCP program management office showed that documentation on individual projects is not always complete. Five final summary reports and one penultimate progress summary report were located for the 10 projects sampled.10 The six available reports were informative but some were difficult to fully appreciate because appendices referred to in the text were not attached. The performance measurement annex was included with only two of the six reports.

Allocation of Resources

Key Findings

There are no resources directly attached to the ARCP. Resources for approved projects and for program administration flow from other DFO science budgets.

The funds for ARCP projects originate from the region or central directorate that has arranged the research contribution to the university where the lead academic investigator is based. Over the five-year period covered by the evaluation, an amount of $2,422,000 was approved for ARCP-related expenditure from regional and central budgets.11 Figure 1 illustrates the proportion of the total provided by each DFO region.12

The evaluation attempted to identify reasons for the wide variation in regional expenditures on academic research. The following key factors emerged: support from the regional director (or central director general), awareness of the ARCP mechanism, regional (or central) information needs and availability of O&M funds. Convergence of these factors appears to underlie high use of the ARCP mechanism.


10 Note that the search was limited to the folder of final reports provided by ARCP management. It is possible that the four missing reports are on file in the region or central directorate that managed the academic research contribution agreement.

11 Source: ARCP master spreadsheet. The amount is rounded to the nearest thousand dollars.

12 Source: ARCP master spreadsheet. Central science directorates are referred to as “National Capital” region in the figure. The Newfoundland & Labrador region is referred to as Newfoundland.

Figure 1: ARCP Funding by Region, 2009-10 to 2013-14

Figure 1

The 2010 T&C limit to $1.6 million per year the total contribution funding given for academic research through the ARCP. The maximum allowable flow of funds through ARCP over the past five years would thus have been $8.0 million. The actual flow of $2.4 million was well below the maximum permitted.

For a given research project, DFO may contribute no more than $250,000 per annum. The highest payment during the past five years was $169,000. The allowable limit was respected.

The study also examined the amounts expended on each of the 28 projects that were completed during the five-year period. The mean DFO expenditure on projects was $133,000. The median amount was $78,000. The smallest DFO contribution was $10,000 for a three-month project using high throughput genome sequencing to examine microorganisms in samples of seawater from various locations and depths in the Arctic Ocean. The largest contribution was $707,880 for a five-year, multi-site, research network focused on invasive aquatic species such as the Asian carp, lamprey eel and zebra mussel.

Resources for central administration flow from the budget for Strategic and Regulatory Science. They appear to be minimal. Management of the ARCP normally takes up only a very small portion of the time of the Director General and the Manager of Intellectual Property and Partnerships.

Value for Money Considerations

Key Findings

The design of the ARCP fosters economical use of funds for scientific research.

Various features of the ARCP help ensure that public funds are used economically.

First, many key informants from DFO made the point that because there is no budget attached to the ARCP, funds for the academic research projects must come from the limited budgets available for science programs. DFO staff is therefore highly motivated to seek lowest cost options when exploring academic research with university personnel.

Second, ARCP T&C require that the university or other partners provide at least 25% of the cost of the research project. DFO contributes no more than 75% of the total project cost.

Third, for most projects graduate students do some of the research work. The modest pay scales for graduate students take into account that they are also benefiting by gaining experience, and usually academic credit as well, from their work on research projects. Graduate student labour is thus an additional economy facilitated by ARCP as an enabler of academic research agreements.

Fourth, academic research provides indirect benefits to DFO in addition to the new knowledge that is generated. It helps train the next generation of fisheries and oceans scientists. It broadens the networks of DFO scientists, thus creating more opportunities for them to access information of use to the department. It stimulates further fisheries and oceans research in universities. Projects enabled by the ARCP therefore provide good value for money.

The comparative effectiveness and efficiency of the ARCP is difficult to establish given its amorphous nature as a generic funding mechanism. Comparison would need to be made against programs that have similar objectives but ARCP scientific objectives being so wide-ranging, no direct comparable program could be readily identified.

5. CONCLUSIONS AND RECOMMENDATIONS


5.1 CONCLUSIONS

The evaluation found that the program remains relevant. There is evidence of a continuing need for a centrally governed mechanism that enables DFO science programs to draw on academic research on topics of relevance to the department’s mandate. The program also aligns with government priorities and departmental strategic outcomes, as well as with federal roles and responsibilities.

Evidence also suggests that the program is successful in achieving its intended outcomes, particularly in enabling high quality academic research that meets the needs of DFO science programs and enhances the quality of science advice for DFO decision-makers.

The program appears to be generally well administered and interviewed DFO scientists are satisfied with the knowledge generated by ARCP projects.

Several improvements were made to the program’s administrative tools since the last evaluation, namely the development of an updated logic model and the implementation of a project summary report form.

A few additional improvements could however be made in order to further strengthen the program’s management and implementation. There is limited and unclear documentation of the program’s specific mandate, delivery mechanism, and roles and responsibilities of program administrators, senior management and DFO science program clients. The program’s particular role as a funding mechanism should be clarified to demonstrate its compliance with the Transfer Payment Policy and more clearly explained and communicated to program stakeholders. Similarly, the development of written operating procedures for administering the ARCP program would reduce business continuity risks should the program’s vital corporate memory leave due to retirements or staff turnover.

While the program’s logic model calls for active communication of the program to potential funding recipients, it remains unclear whether this is an appropriate activity in light of the generic and unfunded nature of the ARCP as a funding mechanism. The program would however need to explore ways to ensure that the selection of funding recipients is not unduly limited to a too narrow pool of known academic researchers and to reduce risks that there could be a perception of favoritism in funding allocation decisions.

Proper governance of the program would require that the program governance structure be more actively engaged in monitoring the program’s performance and in tying decisions on project funding allocation to a formal set of criteria related, among others, to the department’s overall science priorities.

Finally, while recognizing that several improvements have already been made in those respects, the program would benefit from further expanding its project reporting requirements to include additional financial and results information, and from ensuring more systematic central file keeping. This would better position the program to address potential information requests by management, to produce evidence for future audits and evaluations, and to enable access to research results for potential future use by other DFO science programs.

Overall, the program appears to be operated efficiently and economically, the funded projects resulting in good value for the department.

5.2 RECOMMENDATIONS

A first recommendation arises from the observation that the program does not fully adhere to its intended activity of communicating its existence to potential funding recipients, which exposes the department to the risk of not tapping into the most optimal pool of academic researchers and of being perceived to favor certain academic researchers based on personal relationships with DFO scientists. Recognizing that full advertisement of the program may not be appropriate given the generic and unfunded nature of the program, consideration should nevertheless be given to possible ways of mitigating these risks.

Recommendation 1
It is recommended that the ADM, Ecosystems and Oceans Science, consult with NSDC members in order to ensure that appropriate means are in place for identifying potential project proponents, including a consideration of the pros and cons of formally and purposefully communicating the existence of the program to a wider community of academic researchers beyond those already known to DFO scientists.

The next recommendation is intended to improve the governance of the program, in light of the current limited engagement of NSDC members in the review of project proposals and in the monitoring of program performance.

Recommendation 2
It is recommended that the ADM, Ecosystems and Oceans Science, ensure that members of the NSDC engage in discussions for all ARCP proposals and conduct a review of the program’s performance on an annual basis.

A third recommendation aims to encourage ARCP management to enhance the information captured on individual projects and kept in the program’s central filing system in order to enable faster response to enquiries, facilitate sharing of information within DFO, and support future evaluations and audits as well as potential future use of project results by other DFO science programs.

Recommendation 3
It is recommended that the ADM, Ecosystems and Oceans Science, ensure that ARCP reporting guidelines be updated in order that all financial and results information essential for effective monitoring of program performance be captured, presented to the NSDC, and maintained in the program’s central filing system.

The fourth and final recommendation calls for comprehensive and clear documentation on the program to improve clarity of roles and responsibilities, increase awareness and understanding of the program funding mechanism within DFO and academia, and more effectively support program governance.

Recommendation 4
It is recommended that the ADM, Ecosystems and Oceans Science, ensure production of clear and complete documentation covering all essential aspects of ARCP governance and administration, including roles and responsibilities of program administrators, senior management, and DFO scientists, and standard operating procedures for the administration of the program. It is also recommended that the program T&C and logic model be revised accordingly and to demonstrate compliance with the Transfer Payment Policy.

ANNEXES


Annex A:  Recommendations from the Last Evaluation of the ARCP

Annex A
Recommendation from the 2009 Evaluation of ARCP Management Response to the Recommendation
  • 1.   Set-up an Evaluation Advisory Committee for ensuring that performance monitoring data is collected, valid, and reliable.

Science has already implemented procedures to improve reporting under the ARCP, and will set up mechanisms to ensure compliance with these procedures.

  • 2.   Assess whether the ARCP should direct funding towards creating ‘new’ academic research chairs.

The question of whether or not establishing new academic research chairs should be an explicit objective of the ARCP was deferred to Science senior managers who recommended that establishing new academic chairs does not need to be an explicit objective of the ARCP or referenced therein. The reference to establishing new academic research chairs will be deleted from the program, upon program renewal, expected before the end of March 2010.

  • 3.   Review and clarify activities and outputs and rework/remove duplicative immediate outcome statements from the ARCP logic model.

Science has modified the logic model in consultation with Evaluation Directorate. The new model will be included in the program upon its renewal, expected before the end of March 2010.

  • 4.   Develop and apply selection criteria for the selection/refusal of ARCP proposals.

Science has developed selection criteria for approving or denying approval of ARCP proposals.

ANNEXE B: ARCP LOGIC MODEL AS OF APRIL 2010

Annex B

ANNEXE C: EVALUATION ISSUES, QUESTIONS, DATA SOURCES AND COLLECTION METHOD

Matrix

Annex C
Core Evaluation Issue Evaluation Question Data Source/ Collection Method
Relevance Need for the programming Is there a continuing need for the ARCP?
  • Interviews
  • Document review
Alignment with federal government priorities Does the ARCP continue to align with government priorities?
  • Interviews
  • Document review
Alignment with federal responsibilities Is the ARCP consistent with federal and departmental roles and responsibilities?
  • Interviews
  • Document review
Performance Achievement of expected outcomes To what extent has the program provided research findings to DFO?
  • Database review
To what extent has the quality of funded research been assessed?
  • Interviews
To what extent has the program increased DFO science capacity?
  • Database review
  • Document review
What is the evidence that expected outcomes are being achieved?
  • Interviews
  • Document review
Efficiency and economy To what extent has the program been operated as outlined in the Terms and Conditions?
  • Interviews
  • Document review
Is the ARCP governed and managed effectively?
  • Interviews
  • Document review
Is there clear documentation of processes and reporting requirements?
  • Interviews
  • Document review
  • File review (final reports)
To what extent was value-for-money considered in the negotiating of agreements?
  • Interviews
  • Document review
Are there ways in which the effectiveness or efficiency of the program could be improved?
  • Interviews
  • Document review

ANNEX D: MANAGEMENT ACTION PLAN

Annex D
Recommendation 1

Rationale: The program does not fully adhere to its intended activity of communicating its existence to potential funding recipients, which exposes the department to the risk of not tapping into the most optimal pool of academic researchers and of being perceived to favor certain academic researchers based on personal relationships with DFO scientists. Recognizing that full advertisement of the program may not be appropriate given the generic and unfunded nature of the program, consideration should nevertheless be given to possible ways of mitigating these risks.

Recommendation: It is recommended that the ADM, Ecosystems and Oceans Science, consult with National Science Directors Committee (NSDC) members in order to ensure that appropriate means are in place for identifying potential project proponents, including a consideration of the pros and cons of formally and purposefully communicating the existence of the program to a wider community of academic researchers beyond those already known by DFO scientists.

Strategy
ADM, Ecosystems and Oceans Science, will explore with the Science Executive Committee (SEC, formerly NSDC) the pros and cons of communicating the existence of the program to Canadian academic institutions.
Management Actions Due Date (by end of month) Status Update: Completed / On Target / Reason for Change in Due Date Output
Consult with SEC on the pros and cons of more broadly communicating the existence of the program to Canadian academic institutions. March 2015    
Recommendation 2

Rationale: Improvements to the program governance are needed in light of the limited engagement of NSDC members in the review of project proposals and in the monitoring of program performance.

Recommendation: It is recommended that the ADM, Ecosystems and Oceans Science, ensure that members of the NSDC engage in discussions for all ARCP proposals and conduct a review of the program’s performance on an annual basis.

Strategy

As a result of the ARCP evaluation, the program implemented an ARCP proposal evaluation process which includes SEC members discussing the proposals to determine if they should be approved.

The ADM, Ecosystems and Oceans Science, will ensure that program’s performance is reviewed on an annual basis.

Management Actions Due Date (by end of month) Status Update: Completed / On Target / Reason for Change in Due Date Output
The program implemented an ARCP proposal evaluation process which includes SEC members discussing the proposals, prior to deciding whether they should be approved.  Specifically, ARCP proposal review is a standing item on the agenda of SEC teleconferences. October 2014    
An annual program performance evaluation will be implemented. June 2015    
Recommendation 3

Rationale: Encourage ARCP management to enhance the information captured on individual projects and kept in the program’s central filing system in order to enable faster response to enquiries, facilitate sharing of information within DFO, and support future evaluations and audits as well as potential future use of project results by other DFO science programs.

Recommendation: It is recommended that the ADM, Ecosystems and Oceans Science, ensure that ARCP reporting guidelines be updated in order that all financial and results information essential for effective monitoring of program performance be captured, presented to the NSDC, and maintained in the program’s central filing system.

Strategy
The ARCP manager will ensure that a central filing system is created or made available for saving key information on activities performed under ARCP, and that ARCP guidelines require that all such information be captured in that central filing system.
Management Actions Due Date (by end of month) Status Update: Completed / On Target / Reason for Change in Due Date Output
A central filing system for saving key information on activities performed under ARCP will be made available or created for ARCP users. May 2015    
Include in ARCP guidelines a requirement that all ARCP users capture key information on activities performed under ARCP in a central filing system. September 2015    
Recommendation 4

Rationale: Need for comprehensive and clear documentation on the program to improve clarity of roles and responsibilities, increase awareness and understanding of the program funding mechanism within DFO and academia, and more effectively support program governance.

Recommendation: It is recommended that the ADM, Ecosystems and Oceans Science, ensure production of clear and complete documentation covering all essential aspects of ARCP governance and administration, including roles and responsibilities of program administrators, senior management, and DFO scientists, and standard operating procedures for the administration of the program. It is also recommended that the program Terms and Conditions and logic model be revised accordingly and to demonstrate compliance with the Transfer Payment Policy.

Strategy
The ARCP manager will document all activities and processes required to implement the program, including the process for soliciting proposals, evaluating proposals, setting-up contribution agreements and administration of contribution agreements, and reporting.  As well the ARCP manager will revise the program Terms and Conditions and logic model, as required.
Management Actions Due Date (by end of month) Status Update: Completed / On Target / Reason for Change in Due Date Output
The program Terms and Conditions will be revised to demonstrate compliance with the Transfer Payment Policy. February 2015    
The program logic model will be revised to ensure compliance with the program Terms and Conditions. April 2015    
Develop and implement a governance framework, guidelines, and operating procedures for management and administration of activities undertaken under the ARCP. July 2015    
Document the ARCP mechanism and use it to communicate the ARCP mechanism to academia, as required. August 2015