Archived – Summative Evaluation Of The Academic Research Contribution Program

Archived information

The Standard on Web Usability replaces this content. This content is archived because Common Look and Feel 2.0 Standards have been rescinded.

Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Project Number 6B102
Final Report
February 2009

Table of Contents

List of Acronyms
1.0 Executive Summary
1.1 Introduction
1.2 Findings
1.3 Lessons Learned
1.4 Recommendations
2.0 Introduction
2.1 Background
2.2 Evaluation Objectives
2.3 Methodology
2.3.1 Methodology
2.3.2 Limitations
3.0 Observations and Recommendations
3.1 Program Relevance
3.1.1 The program meets the needs of DFO.
3.1.2 The program meets the needs of the target clientele.
3.1.3 The program is appropriate for the federal government.
3.2 Program Performance
3.2.1 The ARCP was implemented as planned.
3.2.2 The ARCP met its expected outcomes.
3.2.3 The program has been implemented at the best possible cost.
3.3 Lessons Learned
3.4 Conclusions
3.5 Recommendations
4.0 Management Action Plan

List of Acronyms

ADM – Assistant Deputy Minister
AIS – Aquatic Invasive Species
ARCP – Academic Research Contribution Program
CAISN – Canadian Aquatic Invasive Species Network
CEARA – Centre of Expertise for Aquatic Risk Assessment
DFO – Department of Fisheries and Oceans
FPT – Federal/Provincial/Territorial
GEOIDE – GEOmatics for Informed DEcisions
NCR – National Capital Region
NGO – Non-governmental organizations
NSDC – National Science Directors Committee
NSERC – National Sciences and Engineering Research Council
RMAF – Results-based Management and Accountability Framework
RPP – Report on Plans and Priorities
TB – Treasury Board

Top of page

1.0 Executive Summary

1.1 Introduction

The Department of Fisheries and Oceans Canada (DFO)’s Academic Research Contribution Program (ARCP) was created to enable, stimulate and enhance academic research relevant to DFO priority areas and to increase science capacity in DFO.  The purpose of this summative evaluation is to determine the extent to which the ARCP has achieved its objectives.

1.2 Findings

After an evaluation of the program relevance and performance, the Audit and Evaluation Directorate, found that:

1. The ARCP meets the ongoing needs of DFO and the recipients:
  • Project goals and objectives funded by the ARCP aligned with current DFO program and management priorities and strategic outcomes Sustainable Fisheries and Aquaculture and Healthy and Productive Aquatic Ecosystems.
  • The ARCP responded to DFO’s need for current research, supported academic institutions and supported DFO decision making.
2. The ARCP is appropriate for the federal government:
  • DFO has the legislative authority to engage in scientific research in marine sciences and consulted/collaborated where appropriate with provinces and territories.
3. The ARCP was implemented as planned: Activities
  • The availability of the program was communicated.
  • The ARCP reviewed and selected applications/proposals.
Outputs
  • The ARCP generated contribution agreements with academic institutions and Networks of Centres of Excellence.
4. The ARCP achieved its expected outcomes: Intermediate Outcomes
  • The ARCP generated knowledge related to DFO strategic priority areas.
  • The ARCP impacted favourably on DFO’s credibility within the academic and other stakeholder communities.
  • The knowledge generated by the ARCP research has benefited DFO and the evidence suggests that it has impacted favourably on decision making at DFO.
5. The ARCP was implemented at the best possible cost:
  • The ARCP was implemented as planned, and leveraged substantial financial and in-kind contributions.

However, there were several minor issues that we felt should be addressed.  First, the Performance Measurement Strategy was not implemented as planned although steps have already been taken to address this issue; however, additional action could be taken to improve a future evaluation of the ARCP. 

[Cabinet Confidence] This raised the question that in spite of the apparent success of the ARCP, perhaps program planners should direct some of its future investments towards the creation of new academic research chairs and reap the long term benefits of doing so.

Third, the ARCP logic model does not fully represent the unique aspects of the ARCP, and this should be reflected in the logic model.  There were also several outcome statements that should be re-examined due to their duplicative nature.

Finally, uniformly applied selection criteria are required for approving or refusing proposals.

We concluded that the program should be continued, and have made a few recommendations to improve accountability and its performance.

1.3 Lessons Learned

The ARCP successfully influenced research activities in recognized post-secondary academic institutions relevant to DFO strategic priority areas and leveraged financial and human resources heavily in its favour.  It involved relevant partner organizations and understanding the importance of mutual interest in research goals and objectives. Finally, close interaction and collaboration between DFO scientists and the research/academic community impacted positively on DFO credibility and assured knowledge transfers into DFO.

1.4 Recommendations

It is recommended that the ADM Science:

Recommendation 1: Set-up an Evaluation Advisory Committee for ensuring that performance monitoring data is collected, valid, and reliable.

Recommendation 2: Assess whether the ARCP should direct funding towards creating ‘new’ academic research chairs.

Recommendation 3: Review and clarify activities and outputs and rework/remove duplicative immediate outcome statements from the ARCP logic model.

Recommendation 4: Develop and apply selection criteria for the selection/refusal of proposals.

Top of page

2.0 Introduction

2.1 Background

The Academic Contribution Program (ARCP) was approved in 2004.  Funding in the amount of $600K per year over five years for the ARCP was secured through an internal transfer of Fisheries and Oceans Vote 1 (Operating) to Fisheries and Oceans Vote 10 (Grants and Contributions) and to convert up to an additional $1 million, as required.  ARCP is managed by the Integrated Business Management Directorate (previously Policy, Planning and Coordination Directorate) under the direction of the National Science Directors Committee (NSDC), Science Sector. According to the ARCP Result-based Management and Accountability Framework (RMAF), a summative evaluation was scheduled for 2008-09, the program's fifth and final year.   

The purpose of the summative evaluation is to determine the extent to which ARCP has or is positioned to achieve its overall objectives, that is, enhancing academic research relevant to DFO priorities and increase science capacity in DFO and in Canada. The time frame covered in this evaluation is from June 2004 to January 2009.

The overall objectives of the ARCP are to:

  • enable, stimulate and enhance academic research relevant to DFO priority areas; and
  • increase science capacity in DFO and in Canada in support of DFO science priority areas relating to marine, freshwater, fisheries, and aquaculture.

The ARCP consists of two components:

  • Contribution funding to academic research that will support DFO science priorities, such as research performed under academic research chairs; and
  • Contribution funding to academic institutions in support of partnering and specific collaborative research projects of mutual interest that:
    • are related to DFO science research requirements; or
    • support DFO Science partnering with Canadian industry in research areas where DFO may lack capacity or where DFO’s Science research capacity may be enhanced by the collaboration.

The target clientele for the ARCP includes:

  • The direct recipients of the Research Contribution program funding, which includes Canadian academic institutions and Networks of Centres of Excellence (as the beneficiary of additional research dollars (or in-kind contributions) provided by DFO in support of their research activity;
  • DFO Science (as a beneficiary of the increased interaction, collaboration and information sharing between DFO Science and the public and private research community and increased access to and involvement in the most current research activities);
  • DFO and other policy makers (as a beneficiary of the increased scientific knowledge);
  • The marine and other industries (as beneficiaries of the research findings); and
  • Other federal and other levels of governments (as beneficiaries of the research findings).

2.2 Evaluation Objectives

The main objectives of the evaluation are to assess the relevance and performance of the ARCP.  The evaluation questions focused the Treasury Boards Policy on Evaluation’s “Core Issues in Assessing Value for Money in Evaluations”.

Academic Research Contribution Program (ARCP)
Issues Evaluation Questions
Relevance Does the program meet the needs of DFO (mandate and strategic outcomes) and the target clientele?
Is the program appropriate to the federal government?
Performance Efficiency: Has the program been implemented as planned?
Effectiveness: Has the program met its expected outcomes?
Economy: Has the program been implemented at the best possible cost?

The evaluation has also determined whether a performance measurement strategy has served to gather valid and reliable data; draw lessons from this experience, i.e., distinguish what does and does not function well; and propose recommendations to help improve program delivery.  The evaluation was carried out in the National Capital Region (NCR).  

2.3 Methodology

2.3.1 Methodology

This evaluation employed a program theory-driven evaluation science approach.  The ARCP logic model served as the instrument describing the ARCP’s program theory.  The program theory or logic model describes the cause and effect sequences that link program activities to outcomes, and illustrates how a program is expected to result in its expected outcomes. Consistent with this approach, the effectiveness evaluation questions focused on key short term and all intermediate outcomes of the ARCP logic model.  We further attempted to describe how the program worked, and why it worked.

Three methods of quantitative and qualitative enquiry were implemented: (1) a document and file review, which involved the review and analysis of policy and planning documents such as Treasury Board submissions, contribution agreements, quantitative data from the ARCP Performance Measurement Strategy, program documentation, client reports, and management and evaluation reports; (2) research of the Internet (CAISN and GEOIDE) and; (3) interviews.  Seven (7) interviews, involving six (6) DFO officials in Ottawa and BC and one (1) funding recipient in BC were conducted between December 11th 2008 and December 22nd, 2008.

Analysis included comparing actual versus planned activities and outputs regarding the implementation of the ARCP and drawing common themes from interview notes in assessing the extent to which program outcomes were achieved.  Where possible, a system of triangulation (using multiple sources of data/evidence to support a finding/conclusion) was used to corroborate findings.   Multiple sources included documents, websites and interview notes. 

We focused our efforts on the internal validity of the ARCP and verified whether the observed changes in causal relationships described in the logic model can be attributed to the ARCP.  Particular attention was paid to the immediate outcome of leveraging DFO science capabilities and resources, and all the intermediate outcomes.

2.3.2 Limitations

Several factors affecting the rigour of this evaluation limited our ability to draw definitive conclusions and should be considered when assessing the findings of this evaluation:

  • The Audit and Evaluation Directorate conducted a small-scale evaluation of the ARCP which was determined to be of low risk. The scope of the evaluation was reduced to fulfill the evaluation requirements in a short period of time;
  • The number of persons interviewed was restricted to six DFO officials and one funding recipient directly involved in the program; 
  • The Performance Measurement Strategy described in the ARCP Results Based Accountability Framework (RMAF) was not fully implemented as planned thus reducing the availability of performance data in support of the outcomes;  So for example, the lack of baseline data in the knowledge areas targeted by the ARCP either prior to or following the full implementation of the ARCP restricted our ability to quantify the increase of knowledge generated as a result of the ARCP; 
  • Furthermore, the scope of the evaluation did not allow sufficient time and resources to compare the knowledge generated by the ARCP to other DFO programs or activities for generating knowledge.

    Top of page

3.0 Observations and Recommendations

3.1 Program Relevance

3.1.1 The program meets the needs of DFO.

We drew a sample of project contribution agreements (76% of all contribution agreements valued at 74% of all funds disbursed) and compared each project’s goals and objectives to the Scientific Strategic Plan for Fisheries and Oceans Canada (1999-2000) and the Departmental Plans and Priorities for 2008-2011 as stated in the 2008-2009 Report on Plans and Priorities (RPP). 

The ARCP’s approved projects demonstrated alignment with the Scientific Strategic Plan, which identified well developed networks, partnering with the private sector and universities, participation in multidisciplinary groups and acquiring knowledge through collaborative research projects as key strategic initiatives.  We found that ARCP approved projects aligned with a number of the RPP’s program and management priorities as a combination of Fisheries Renewal, Health of the Oceans, Science Renewal, Species at Risk Management and Human Resources Modernization. 

We were able to establish direct links between each project and at least one of the strategic outcomes Sustainable Fisheries and Aquaculture, Healthy and Productive Aquatic Ecosystems and Safe and Accessible Waterways.  Finally, interview respondents confirmed that the ARCP had the appropriate proposal review structure to ensure that research activities met with DFO priorities.  We concluded that the ARCP met the needs of DFO with respect to its strategic outcomes.

3.1.2 The program meets the needs of the target clientele.

At the turn of the century, DFO faced challenges to sustain priority science programs at their existing levels due to a lack of capacity (financial resources and expertise). The Scientific Strategic Plan for Fisheries and Oceans Canada identified that solutions to scientific problems for which it is responsible will increasingly depend on collaboration with academic institutions, industry, and other science-based departments and agencies. 

Academic institutions conduct research in marine, freshwater, fisheries and aquaculture by experts who are often at the leading edge in these fields.   The ARCP provided access to and involvement in the most current research activities in support of DFO priorities.  For example, each ARCP contribution influenced the research, and had a DFO Science employee(s) working hand in glove with the academic research teams, thus providing access to valuable research data, state of the art analysis techniques and privileged access to databases. 

The ARCP funding put additional research dollars at the disposal of Canada’s leading researchers and networks, expanding the research community’s reach, as well as Canada’s future research capacity in these fields by supporting masters and doctoral students. 

There was evidence that the knowledge generated supports DFO’s decision making and regulatory roles.  For example, the ARCP funded a University of New Brunswick ocean mapping project.  The data, which was considerable, informs the design of Arctic navigational charts, defines shipping lanes, and improves safety thus reducing the threat of marine ecological disasters.  The data will also be used for defining Canada’s boundaries and build a well supported claim with the United Nations Convention on the Law of the Seas.  It will also define the limits of our environmental control, economic and commercial activities in the Arctic.  In short, it will impact on Canadian sovereignty. 

These examples demonstrate that DFO science is engaged in valid interactions, collaborations and information sharing with the academic research community. 

We concluded that the ARCP meets the needs of DFO and of its targeted clientele.

3.1.3 The program is appropriate for the federal government.

Our review of the relevant literature revealed that a DFO Minister has the legislative authority to engage in activities that include scientific research related to hydrographic and marine sciences, and fisheries exploitations.  For example, under section 4 of the Department of Fisheries and Oceans Act, the powers, duties and functions of a Minister extend to hydrography and marine sciences.  This is further affirmed in the Oceans Act, under Sections 42 and 43, which specifies the activities and actions, (including scientific research in cooperation with others), a Minister may undertake in hydrography and marine science.  The Fisheries Development Act, under section 3, enables a Minister to undertake projects for the more efficient exploitation of fishery resources and for the exploration for and development of new fishery resources and fisheries.  Finally, our document review uncovered an example where the provinces and territories entered into agreements with the federal government, as is the case with the “Invasive Alien Species Strategy for Canada”, where DFO would be the lead for aquatic invasive species. 

We concluded that the ARCP is within the federal government’s jurisdiction, is linked to federal government priorities, that provinces were consulted/considered and is therefore appropriate for the federal government.

3.2 Program Performance

3.2.1 The ARCP was implemented as planned.

Activity - Communicating the availability of the program

DFO communicated the availability of ARCP funding to targeted academic institutions and researchers through National Science Director Committee (NSDC) members, and/or regional staff members.   There was no public call for proposals for the ARCP, however in some cases, unsolicited academic institutions did approach DFO. It is possible that some academic institutions may not have been reached by the ARCP but we estimate this to be minimal.  Discussions and negotiations were entered into with applicants to explain DFO’s priority areas.   We concluded that this activity has been implemented as planned.

Activity - Reviewing and Selecting Applications / Proposals

Proposals were submitted by academic institutions to the NSDC member in their region.  Proposals were reviewed by individual NSDC members, then by the NSDC as a whole.  Applicants described the project’s objectives and relevance against DFO priorities.  However, the April 2008 Management Review found that the processes/activities followed by NSDC indicates a lack of documented, detailed selection criteria for NSDC to approve academic proposals. Projects that supported current strategic science priorities and regional science priorities were selected for funding. Contribution agreements were ultimately approved at the Assistant Deputy Minister level.  We concluded that this activity has been implemented as planned.

Outputs – Contribution agreements with academic institutions and Networks of Centres of Excellence

DFO yielded contribution agreements with academic institutions that described specific deliverables.  This manifested two types of models:  The first model was the funding of academic research chairs and research conducted under Networks of Centres of Excellence. These agreements focused on a finite number of research projects for the duration of the agreement.  The second model was the funding of strategic academic networks.  Generally, seed money is provided to an academic institution over a two year period to lay the ground work of the strategic academic network, which includes (1) recruiting partner organizations such as other academic institutions, government departments/agencies and non-governmental organizations; (2) defining the research agenda for the network; and (3) preparing a proposal for submission to DFO to consider funding the network over a five year period. Once created, a network laid the foundation for DFO researchers to collaborate on research and development of mutual interest with academic institutions. Both models focused on DFO’s science priorities.  We concluded that this activity has been implemented as planned; however we noted a few discrepancies which are discussed below.

  • The ARCP invested almost 78% of its total planned contributions of $3 million over the five years of the program.  Pressure to restrict spending was cited as a reason for not allocating 100% of the $3 million.  It should be noted that the ARCP was delayed in its implementation while awaiting final Treasury Board approval.  The program officially started in October 2004.  Table 1 below illustrates the breakdown of annual allocation of contributions and the actual contributions

Table 1 – Budgeted versus actual contributions (thousands of dollars)

2004/05 2005/06 2006/07 2007/08 2008/09 Total %
Planned Contributions 600K 600K 600K 600K 600K 3,000K 100%
Actual Contributions 275K 325K 530K 337K 861K 2,328K 77.6%
  • [Cabinet Confidence] The cost of creating a new academic research chair is normally in excess of $500K over a five year period. Once established, the chair becomes permanent, and requires no further assistance to sustain over the long term. The ARCP contribution agreements were with already ‘established’ academic research chairs, an activity which is not intended to lead to the establishment of a ‘new’ chair. There were no apparent efforts by the ARCP to establish ‘new’ academic research chairs. The ARCP should evaluate whether it should direct some of its future investments towards the creation of new academic research chairs and reap the long term benefits of doing so.

  • [Cabinet Confidence] There lacked consistency in gathering data from individual contribution agreements, and the lack of a centralized function to monitor the ARCP as a whole. A Management Review picked up on this issue in April 2008, and steps have since been taken to develop a progress report template and include a section for reporting on relevant performance measurement indicators.

  • The uniqueness of the ARCP is not adequately reflected in its logic model. In its current form, the activities and outputs could apply to most government Grants and Contributions programs. The ARCP logic model should reflect the research and knowledge that is generated, the link to the sponsoring organization’s priorities, the mutual interests of the partners involved, the outreach, discussions and networking involved and the close interactions between the sponsoring organization and the research/academic community. There are also immediate outcome statements that appear to be similar, and could be streamlined.

3.2.2 The ARCP met its expected outcomes.

Increased scientific knowledge related to DFO strategic Priority areas

All of the respondents interviewed agreed that DFO benefited from an increase in scientific knowledge directly related to DFO strategic priority areas.  Our review of the Canadian Aquatic Invasive Species Network (CAISN) supported this claim.  For example, after its second year of operation, CAISN published 34 papers in peer reviewed journals, presented at 147 conferences, produced seven thesis/dissertations and four technical reports. The GEOIDE network is another case in point where 15 research topics were pursued which has resulted in publishing approximately 234 peer reviewed publications. The knowledge gained from contributions towards academic chairs equally produced knowledge for DFO.  For example, from 2006 to 2008, Simon Fraser University (SFU) published 16 papers in peer reviewed journals and presented five papers in refereed conference proceedings.  Other contribution agreement projects produced DNA data for the Atlantic Cod fish stocks, fish stock assessment models, feeding ecology of Atlantic salmon stocks, fisheries resource conservation genetics and biotechnology knowledge, all topics that align with DFO strategic priority areas.  One of the challenges of assessing the full value of the ARCP is our inability to predict what tomorrow’s crisis will be.  By investing in tomorrow’s potential issues today, the ARCP can bring benefits to DFO in the longer term.   We concluded that scientific knowledge related to DFO strategic priority areas was produced as a result of the ARCP.

Increased credibility of DFO Science within the academic and other stakeholder communities

All respondents agreed that DFO`s visibility and credibility increased since the ARCP. Interactions with academic institutions were cited as one of the main reasons for this increase. For example, DFO is collaborating with researchers from post secondary academic institutions on all of its projects, holding offices in some cases directly on site.  This has led to DFO scientists supervising graduate student dissertations; presenting research findings at conferences; seeing an increase in the number of visits by academics to regional offices; being invited to present lectures at universities; and co-authoring research papers in peer reviewed journals.  Some of these findings were supported by our document review as well.  

Since the inception of the ARCP, several graduate students have opted for employment with DFO, and  DFO visibility with industry (i.e. commercial fishing, shipping industry, etc.) and provincial governments has increased thanks to DFO`s participation in the networks. The Ontario government for example received key advice regarding an invasive species, the Asian Carp, and used research findings to justify increased investments into mitigation strategies. This has positioned DFO favourably with the clients it normally provides advice.   Since its involvement with CAISN, DFO`s reputation has also increased with the Natural Sciences and Engineering Research Council (NSERC), where the latter now welcomes letters of intention for new networks, several of which are in development at the time of writing this report.  We concluded that there is evidence that the interactions with the academic community have impacted positively on DFO’s credibility within the academic and other stakeholder communities.

Improved DFO Science decision making

Interview respondents stated there has been an uptake of knowledge and believe the ARCP is generating knowledge and expertise on subject matter that contributes towards credible, timely and scientific-based decision making on emerging issues. For instance, in 2008-09, DFO Policy had plans to map a regulatory framework and identify the best tools for specific Aquatic Invasive Species (AIS) pathways and regions in Canada.  In another example, DFO obtained and analysed mapping data thanks to an ARCP agreement with the University of New Brunswick and as a result, improved its Arctic Ocean navigational charts, consequently improving safety in Canadian waters.

The assessment of the uptake of knowledge from ARCP research has its limitations. Linking specific research findings to the decision making process is challenging, but overall the knowledge generated by ARCP research has benefited DFO and the evidence suggests that it has impacted on decision making at DFO.

3.2.3 The program has been implemented at the best possible cost.

The ARCP increase in scientific knowledge was attributable to DFO’s ability to leverage science capabilities and resources of partner organizations.  The Canadian Aquatic Invasive Species Network brought together 15 universities from eight provinces, over 30 researchers and 42 graduate and postdoctoral students with specialization in invasive species, along with two Ontario provincial ministries and two non governmental organizations. Just one of the GEOIDE projects, Mapping the Ocean Surface with Geodetic and Oceanographic Tools, leveraged 45 investigators and undergraduate students. Similarly, contributions towards academic chairs leveraged access to important databases that were not created directly by ARCP funding, accessed the talents of undergraduate students and produced numerous published papers, articles and conference presentations.  

DFO investments in Networks amounted to $687K and leveraged over $8 million in monetary contributions or almost 12 times the investment, while DFO contributions to academic chairs amounted to $736K and yielded $3.4 million in monetary contributions or almost five times DFO’s contribution.  For example, one academic chair funded by the ARCP at $125K yielded two and half post-doctoral studies, and six master’s degree students working alongside one DFO scientist. One of our respondents concluded that DFO ends up with more people working on its behalf, and gets voluntary advice from faculty members working with leading edge technologies. 

Finally, the ARCP was implemented as planned, and efficiently with no apparent need to expend any supplemental resources to reach its stated objectives.  We concluded that the program was implemented at the best possible cost, and furthermore, with the benefits that accrued through leveraging financial and in-kind resources, well exceeded typical return on investment.   

3.3 Lessons Learned

One of the factors that explain why this program is a success is its ability to influence research activities in academic circles. Networks are particularly successful in terms of leveraging science capabilities and resources.  It did so by working with and understanding the funding priorities/paramaters of NSERC, the largest pool of financial capital we know of that DFO can leverage.  ARCP Directors understood that they could control how their contributions could be invested, and ensured that relevant DFO scientific personnel fully participated in the academic research activity.  This approach served the DFO equally well in either academic chair or network contributions.  Clearly, networks yielded the greatest knowledge generation, however, investments in academic chairs should continue, but be scrutinized for potential return on investment.  

Areas with the potential for future research would be to compare the ARCP to other research/knowledge generating activities at DFO to better understand the advantages and disadvantages of each model and determine which one yields the greatest benefits for DFO under what types of conditions.

3.4 Conclusions

The ARCP proved to be relevant for all parties involved.  Academic institutions are focusing their research on areas that are relevant to DFO priorities.  The knowledge generated is current and leading edge.  The funding expanded the academic institution’s reach, which also led to increasing Canada’s academic capacity in the marine, fisheries and aquaculture fields.  The ARCP focuses its efforts on emerging as well as current issues and provides DFO with relevant information for decision making purposes.   Finally, the ARCP is within the federal government’s jurisdiction, is linked to federal government priorities and is therefore appropriate for the federal government.

Except for a few minor issues, the ARCP was implemented as planned.  The ARCP has produced scientific knowledge related to DFO strategic priority areas.  The evidence showed that the interactions with the academic community have impacted positively on DFO’s credibility with the academic and other stakeholder communities.  The knowledge generated by the ARCP has benefited DFO and the evidence suggests that it has impacted on decision making at DFO.  The lack of baseline data prior to the inception of the ARCP however limits our ability to fully quantify the increases in credibility, knowledge and decision making. We concluded that the program should be continued, and made a few recommendations to improve accountability and its performance.

3.5 Recommendations

It is recommended that the ADM Science:

Recommendation 1: Strike an Evaluation Advisory Committee that would meet once a year if the ARCP is renewed. Its purpose would be to ensure the evaluability of the program i.e. ensuring performance monitoring data is valid, reliable and collected.

Recommendation 2: Assess whether the ARCP should direct funding towards creating ‘new’ academic research chairs.

Recommendation 3: Review and clarify activities and outputs and rework/remove duplicative immediate outcome statements from the ARCP logic model.

Recommendation 4: Develop and apply selection criteria for the selection/refusal of proposals.

Top of page

4.0 Management Action Plan

  Follow-up report update
Recommendations Management Action Plan Target date Completion Date
1. Set-up an Evaluation Advisory Committee for ensuring that performance monitoring data is collected, valid, and reliable. Science has already implemented procedures to improve reporting under the ARCP, and will set up mechanisms to ensure compliance with these procedures. July 2009 June 2009
2. Assess whether the ARCP should direct funding towards creating ‘new’ academic research chairs. The question of whether or not establishing new academic research chairs should be an explicit objective of the ARCP was deferred to Science senior managers who recommended that establishing new academic chairs does not need to be an explicit objective of the ARCP or referenced therein.  The reference to establishing new academic research chairs will be deleted from the program, upon program renewal, expected before the end of March 2010. March 2010 July 2009
3. Review and clarify activities and outputs and rework/remove duplicative immediate outcome statements from the ARCP logic model. Science has modified the logic model in consultation with Evaluation Directorate.  The new model will be included in the program upon its renewal, expected before the end of March 2010. March 2010 March 2010
4. Develop and apply selection criteria for the selection/refusal of ARCP proposals. Science has developed selection criteria for approving or denying approval of ARCP proposals. July 2009 July 2009