Summary of the Evaluation of the Canadian Science Advisory Secretariat (CSAS)
The Evaluation of the Canadian Science Advisory Secretariat has a two-page graphic summary of the information presented below, which is available in pdf format.
About the Evaluation
The evaluation covers 2013-14 to 2017-18 and was carried out between March 2018 and January 2019. Through a principles-based evaluation the evaluation focused on the extent to which the principles informing the CSAS science advisory process are articulated, adhered to and are leading to desired results. Evidence was gathered through interviews; administrative and financial data analyses; document review; site visits to the Pacific and Maritimes regions; and a survey of scientists/researchers (n = 254) and clients or end-users (n = 149).
CSAS and SAGE
CSAS coordinates peer-review assessments on issues related to fish stock dynamics, species at risk, invasive species, marine and freshwater ecosystems, marine protected areas, and aquaculture, and provides advice on these issues to its clients, which comprise staff from DFO policy and management programs. The total direct cost of delivering CSAS is unknown as it is not possible to track the expenditures of all the DFO programs participating in science advisory processes.
The foundation of the CSAS science advisory process are the six Science Advice for Government Effectiveness (SAGE) Principles developed by the Council of Science and Technology Advisors to support the effective use of science advice in decision making. The SAGE Principles include: (1) early issues identification; (2) inclusiveness; (3) sound science advice; (4) openness and transparency; (5) review; and, (6) uncertainty and risk.
The number of full-time equivalent (FTE) positions varies across the Secretariat and the centres based on their available budget and the average number of requests for science advice received each year.
Making a Good Thing Better
The CSAS science advisory process is a unique and important mechanism for developing and providing science advice in support of decision-making. It is aligned with DFO’s mandate and four performance indicators in the Departmental Results Framework. The inclusion of peer review assessments for developing the science advice is one of the main strengths of the CSAS science advisory process and is a best practice when using scientific information to inform fisheries management decisions. There is a growing demand for science advice within DFO and 60% of clients and end users feel that CSAS is doing a good or great job of providing them with the information they require. Additionally, 86% of CSAS clients and end-users, as well as 75% of DFO scientists report that they use one or more CSAS documents to support their work. While the CSAS is valued and well-respected within DFO there are several opportunities for improvement.
The extent to which the CSAS science advisory process adheres to the SAGE principles tends to vary across regions and by type of request. This variance is generally attributed to an insufficient number of policies, guidelines, or structures that operationalize the SAGE principles to ensure the systematic implementation of a science advisory process in compliance with these principles across Canada. In particular, there are opportunities for greater clarification regarding the operationalization of specific principles at each phase of the process, namely early issue identification, inclusiveness, sound science advice, and openness and transparency.
Requests for Science Advice
Between 2013-14 and 2017-18, 49% of the 1247 requests for science advice submitted to the CSAS were included as part of the science advisory schedule either the same year they were submitted or in subsequent years as resources and/or data for developing the science advice became available.
Among the various client sectors submitting requests for science advice ,the sectors that experienced the biggest gap between the number of requests submitted and addressed were Oceans Management, Species at Risk, Fisheries Protection Program and Aquaculture.
Inclusiveness & Conflict of Interest
73% of CSAS clients and end-users feel that the CSAS peer- review assessments involve an appropriate range of perspectives to support the development of science advice. There is however, a tension among CSAS stakeholders regarding whether only scientists and/or individuals with a scientific background should participate in the peer review assessments or if the assessments should be more inclusive of participants from industry, academia, Indigenous communities, and non-governmental organizations.
While the Chair of the CSAS peer review assessment is responsible for mitigating conflicts of interest, best practices reveal that robust conflict of interest policies and/or guidelines are also important. Currently, the CSAS does not have a conflict of interest policy or general guidelines and there is concern that external stakeholder groups (e.g. industry, non-governmental organizations) participating in the CSAS process may attempt to use the process to advance their own agenda.
Submission and Publication Compliance Rates
Of the 1022 documents analyzed in the publication data- base, 77% (n = 787) were published, 21% (n = 215) were not yet submitted for publication and 2% (n = 20) were submitted but not yet published
Science Advisory Reports and Research Documents had the lowest submission compliance rates
Science Advisory Reports and Science Responses had the lowest publication compliance rates
RecommendationsIt is recommended that the ADM, Ecosystem and Oceans Science:
- Develop mechanisms to support the standardization of formal communication between EOS scientists and CSAS clients during each phase of the science advisory process.
- Implement a multi-year science advisory schedule as part of the CSAS science advisory process to support adherence to the SAGE Principle of early issues identification and to mitigate some of the challenges impacting the extent to which the CSAS is able to address requests for science advice.
- Develop clear guidelines to support the operationalization of the SAGE principle of inclusiveness.
- Develop a conflict of interest policy to support adherence to the SAGE Principle of sound science advice with clear guidelines regarding the roles and responsibilities of peer-review assessment participants including the Chair.
- Review timeline targets and develop mechanisms to increase compliance rates for the submission and publication of CSAS documents.
For the full evaluation, visit the DFO Evaluation website: http://www.dfo-mpo.gc.ca/ae-ve/evaluations-eng.htm