DFO Review Reports prepared by Audit and Evaluation Directorate

---

July 1999

Table of Contents

Introduction

1. Chapter 1 Organization, Leadership, Accountability
   • 1.1 Who Responded
   • 1.2 Functions
   • 1.3 Public Policy Domain
   • 1.4 Scope of Units' services
   • 1.5 Unit Names
   • 1.6 Who heads these Units
   • 1.7 Formal Hierarchy
   • 1.8 Committee Structure
   • 1.9 Formal approvals
     
     
2. Chapter 2 Resources and People
   • 2.1 Unit Resources
   • 2.2 Resource Expectations
   • 2.3 Resource Level Trends
   • 2.4 Resources Transferred In
   • 2.5 Contracting
   • 2.6 Classifications and Levels
   • 2.7 Staff Qualifications
   • 2.8 Personal Characteristics and Language Skills
   • 2.9 Students
   • 2.10 Secondments
   • 2.11 Special Recruitment Programs
   • 2.12 Retirement
   • 2.13 Recruiting Plans
   • 2.14 Information Technology Auditors
   • 2.15 Learning Strategies
     
     
3. Chapter 3 Policies and Practices
   • 3.1 Work Planning
   • 3.2 Participation and Resulting Action by Program Managers
   • 3.3 Service Offerings
   • 3.4 Liaison with the Office of the Auditor General
   • 3.5 Workload
   • 3.6 Publication of Reports
   • 3.7 Quality Assurance
   • 3.8 Productivity and Service Quality
   • 3.9 Information Technology and Communication
   • 3.10 Activities Outside the Unit
   • 3.11 Joint Projects

Appendix Questionnaire

---

Introduction

The main sponsor of this study is the Director General, Audit and Evaluation Directorate, Fisheries & Oceans Canada on behalf of the Review Network. The Network is made up of many of the heads of audit, evaluation and review in the federal government of Canada. Two other sponsors are Treasury Board Secretariat (translation and publication) and Transport Canada (issues and challenges workshops); and 7 other Review Network members provided financial support for the study and assisted in developing the survey instrument.

The purpose of the study is to give Review Network members a snapshot of how their function is organized and resourced at present; of significant characteristics and issues regarding the mission, policies and practices in this area; and of the effects of the environment on their work. It resembles closely the Profile of Audit, Evaluation and Review Units in the Federal Government in 1995, the first detailed comprehensive portrayal of this community.

The questionnaire was sent out in mid-February 1999 to the names on the Review Network contact lists. Most responses were submitted in writing; a small number of organizations (seven) were interviewed by phone. Responses were gathered from late February until early May. The analysis and report are the responsibility of the same consultant (Agora Management Associates) that was responsible for the previous Profile study in 1995.

Scope and Perspective

The study questionnaire deals with that part of audit, evaluation and review that is assigned to central organizational units which specialize by and large in those activities. It does not ask for data or descriptions of these activities as carried out in a 'decentralized' manner (for example, by regional audit, evaluation or review units, or by program managers on their own programs). Moreover, the focus is on internal audit, evaluation and review -- not these services as offered outside the parent organization.

The report treats information from all sources equally. There is no weighting factor reflecting the sizes of institutions from which the information came. Of course, the raw data, which has been retained in electronic form, can be analyzed further according to institution size or other characteristics of interest.

Because this is a community profile, not an evaluation of practices in individual institutions, the report does not identify sources.

Special Terms and Acronyms

AER is used throughout as shorthand for "audit and/or evaluation and/or review."

Unit is used for an organizational grouping that provides audit and/or evaluation and/or review (AER) services to its parent organization. Unit is capitalized to remind the reader that it is a special term in this report.

Institution is used for the parent organization (This term, used in federal information law, is more general than "department" and less ambiguous than "organization").

AACI Accredited Appraiser Canadian Institute

ACL Audit Command Language

ADR Alternate Dispute Resolution

ASD Alternative Service Delivery

ATIP Access to Information and Privacy

CCAF Canadian Comprehensive Auditing Foundation

CQA-ISO Certified Quality Auditor - International Standards Organization

CRF Certified Real Estate Finance

CSA Control Self Assessment

FTE Full Time Equivalent

ICIA Industrial, Commercial and Institutional Accountant

IMA Institute of Management Accountants

IRC Industrial Relations Counsellor

ISO International Standards Organization

OAG Office of the Auditor General

P.Ag. Professional Agrologist

PCO Privy Council Office

SUD System under Development

TBS Treasury Board Secretariat

TQM Total Quality Management

Limitations

Some of the data is not strictly comparable; for example, several respondents operate on a different 'year' than the April 1 to March 31 year of federal departments (The survey accepted the data they offered -- usually the best possible fit with the period specified in the question). Sometimes respondents had to give approximations or best guesses. Some numbers differ from what can be found in published sources. Also, totals on the same issue may differ in various parts of the report because some respondents did not answer all the questions.

Participation in the survey was voluntary. As it turned out, nearly three quarters of the Institutions approached did participate, resulting in a very good unstructured sample. Readers should keep this in mind when making inferences based on the findings.

Overview of the Report

Chapter 1, "Organization," identifies the Institutions covered in the study; provides basic characteristics of the participating AER Units (scope, Unit Head, line and functional reporting); and indicates how major responsibilities are apportioned. This chapter presents the results of questions 1, 2, 4-8, 21 and 22.

Chapter 2, "People and Resources," covers total resources in terms of FTEs, salary and contract dollars; and key characteristics of personnel (classification, education, years of experience etc.). This chapter presents the results of questions 9-18 and 32-39.

Chapter 3, "Roles, Services and Practices," deals with the occurrence of significant AER activity outside the direct control of the AER Unit; work planning cycle; the use of action plans, preliminary surveys and assessments; range of service offerings; overall workload; automation; and how AER Units measure their performance and ensure quality. Questions 19, 20, 23-31 and 40-42 are covered in this chapter.

The Appendix contains the questionnaire. A guide to the database produced for this study accompanies this document.

Availability

This summary of the findings is available from the office of:

Alain Larivière

Director

Audit and Evaluation Directorate

Fisheries & Oceans Canada

200 Kent St., 14th floor,

Ottawa K1A 0E6

Tel: (613) 993-5148

Study participants may also have the report and data in electronic form to allow them to do further analysis.

Acknowledgements

We would like to thank:

all those who answered the questionnaire and our subsequent phone-calls for clarification;

Fisheries and Oceans Canada for engaging us for this project; and especially

Jacqueline Gonçalves, our clear-minded, friendly, efficient and dedicated contact at F&O.

Robert Czerny and John Burrett, Agora Management Associates, Ottawa

Since leaving the federal government in 1995 after 21 years'service, Bob Czerny has served many government and private sector clients in such areas as strategic studies; organizational development; partnership and corporate sponsorship studies and education; facilitation; surveys; and audit, evaluation and review studies.

John Burrett was with the federal government for 11 years, 7 of those as a program evaluator. Since leaving in 1997, his clients have included several federal government departments as well as private and non-profit organizations, in the areas of program evaluation and review; performance measures; and research, data acquisition and analysis.

---

Chapter 1 Organization, Leadership, Accountability

1.1 Who Responded

The questionnaire went out to most of the nearly one-hundred members of the Review Network. After removing the cases in which no response could be expected -- for instance, institutions which do not divulge such information for security reasons, and cases in which the AER function is dormant or just in the process of being established --there were 79 potential respondents, representing 71 federal institutions. (In some cases, more than one person represents an institution on the Review Network; these tend to be cases where an Institution has an audit Unit separate from the evaluation or review Unit.)

Some Units that might have participated in other circumstances gave reasons for not answering; for instance, "We have been stable since 1995 survey," "Responding is too onerous in relation to the benefit for a one-person shop," "We are so different from all the other institutions."

In all, 56 responses were received (49 written and 7 verbal, the latter covering only part of the questionnaire); 14 were dual responses from the same institution. Thus, 49 institutions are represented in this survey:

15 Departments:

Agriculture and Agri-Food Canada

Canadian Heritage

Environment Canada

Fisheries and Oceans

Foreign Affairs and International Trade

Health Canada (separate responses for audit and evaluation)

Human Resources Development Canada (separate responses for audit and evaluation)

Indian and Northern Affairs Canada

Justice Canada (separate responses for audit and evaluation)

National Defense

Natural Resources Canada

Public Works and Government Services Canada

Revenue Canada (separate responses for audit and evaluation)

Transport Canada (separate responses for audit and evaluation)

Veterans Affairs Canada

34 Others:

Atlantic Canada Opportunities Agency

Atomic Energy Control Board

Business Development Bank of Canada

Canada Information Office

Canada Mortgage and Housing Corporation

Canadian Center for Management Development

Canadian Dairy Commission (separate responses for audit and evaluation)

Canadian Food Inspection Agency

Canadian Human Rights Commission

Canadian International Development Agency

Canadian Museum of Civilization

Canadian Radio-Television and Telecommunications Commission

Canadian Transportation Agency

Communications Security Establishment

Correctional Service Canada

Economic Development Canada

Farm Credit Corporation

International Development Research Centre (separate responses for audit and evaluation)

Medical Research Council of Canada

National Archives of Canada

National Capital Commission

National Energy Board

National Film Board

National Parole Board

National Research Council

National Search and Rescue Program

Natural Sciences and Engineering Research Council

Office of the Auditor General

Public Service Staff Relations Board

Royal Canadian Mounted Police

Social Sciences and Humanities Research Council

Statistics Canada

Supreme Court

Western Economic Diversification Canada

Broad budget information (question 3) was collected with respect to financial resources and FTEs and it can be used in segmenting the data in terms of institutional size. It is not reported here, but the data can be found in the database that accompanies this report.

Note that many units responding to this survey are involved in various combinations of evaluation, audit and review. For this reason, the number of units reported in an analysis or table may often exceed the number of units in the study.

1.2 Functions

Units were asked (question 1) to rank five broad functions by their order of importance to their Institution's mandate. The following table shows how many ranked a given function as number one.

Q1: Nature of Institutions'Primary Business Lines

Regulatory	14
Operational	16
Policy and Legislation	8
Government Administration	4
Research and Development	6
Other Functions	10

How to read this table: "14 respondent Units selected regulatory activities as their institution's primary function", etc.

Ten units noted that their primary function was other than those above, including: administrative and quasi-judicial statutory tribunals, final arbitration of legal disputes, museum services, social/economic development, provision of funding for First Nations, and communications.

The purpose of this information is to allow the response population to be segmented by function, for those users of the database who wish to look at a sub-set that interests them.

1.3 Public Policy Domain

Question 2 asked respondents to rank public policy domains in terms of their importance to their institution's mandate.

These domains are based on the terminology used by the Treasury Board Secretariat in the early 1990s in federal budget documents. As above, the table reports those having ranked a choice as number one.

Some respondents had obvious difficulty in choosing the primary domain; their answers might be disputed by colleagues. Moreover, two Units took a different approach entirely and have been left out of the table.

The purpose of this information is to allow the response population to be segmented by public policy domain, for those users of the database who wish to look at a sub-set that interests them.

Q2: Institution's Primary Public Policy Domains

Primary Area
Defense	2
Foreign Affairs	2
International Trade	5
International Assistance	3
Social Programs	9
Industrial Support (National and Regional)	8
Scientific-Technological Support	7
Transportation Programs	4
Heritage and Cultural Programs	6
Justice and Legal Programs	5
General Government Services	3
Public Debt Charges and Fiscal Arrangements	1

How to read this table: "2 respondent Units selected Defense as their institution's primary public policy domain," etc.

1.4 Scope of Units' services

Question 5 asked respondents to list their business line offerings. From the chart, it is clear that about 80% of the Units responding to the survey are multi-functional in nature.

Q5: Services Offered

One half of these Units combine audit, evaluation and review, while another 13 combine review with either audit or evaluation.

In 1995, just under 70% were multi-functional or did review alone (where 'review'itself is multi-functional). Thus, the trend towards integration of services within the same Unit noted in the 1995 study appears to have continued. Some evaluation and audit Units still remain alone, and it is worth noting that in 1999 no Units claimed that they only did review.

Of these groups, 41 said that they also did management assistance, providing data and advice to management as well as liaison with central agencies. The "other duties" mentioned most frequently were performance measurement and service quality standards, strategic planning and policy, statistical and data services and inspections and investigations.

1.5 Unit Names

Although many responding Unit names are the same as in 1995, there is an interesting broadening of nomenclature. Audit, evaluation and/or review is being associated with or is subsumed under a wider range of other functions--advisory services, research, risk management, data development, statistics, quality management, renewal, information management, communications (1 instance each), international (2), performance (3), program or corporate services (4), policy (4) and planning (7).

The number of Units using the traditional vocabulary of audit and/or evaluation is about the same as in 1995; so is the number of Units that have adopted the term "review." However, in 1995 there were two units that used all three key terms in their names; now, none of the responding Units do so.

Here are the names of the 56 responding Units:

AUDIT ALONE -- 7

Corporate Audit

Internal Audit (2)

Internal Audit Bureau/ Directorate/ Division/ Services Division (4)

EVALUATION ALONE -- 4

Evaluation Division/ Unit (2)

Program Evaluation Branch/ Division (2)

REVIEW ALONE -- 9

Corporate Review Branch

Directorate of Review Service

National Archives Review Division

Review Branch/ Services Branch/ Directorate/ Services Division (6)

AUDIT AND EVALUATION -- 11

Audit and Evaluation Branch/ Division/ Function/ Group/ Services/ Unit (9)

Departmental Audit and Evaluation Branch

Internal Audit and Evaluation

AUDIT AND EVALUATION AND OTHER -- 2

Direction de la Planification, de l'évaluation et de la verification

Corporate Audit, Research and Evaluation

AUDIT AND REVIEW -- 3

Audit and Review

Corporate Audit and Audit and Evaluation Directorate

Review and Internal Audit

AUDIT AND OTHER -- 2

Audit and Advisory Services

Audit and Portfolio Risk Management

EVALUATION AND OTHER -- 4

Evaluation and Data Development

Direction generale, Gestion de la qualité, de l'évaluation et de l'information

Program Evaluation and Statistics

Strategic Planning, Policy and Evaluation Branch

REVIEW AND OTHER -- 3

Direction des service intégrés, examen, renouveau

Planning and Review

Strategic Planning and Audit and Evaluation Directorate

PROGRAM/ CORPORATE SERVICES -- 3

Corporate Services (2)

Program Services Branch

OTHER -- 8

Direction des politiques, de la planification et de la collaboration internationale

Division de la mesure du rendement

Office of the Inspector General

Performance Assurance

Performance Review Branch

Planning and Assessment

Policy and International Relations

Strategic Planning, Policy and Communications

1.6 Who heads these Units

Question 6 asked for the title of the person who heads the Unit; question 7 asked for the Unit Head's classification and level. The great majority of heads of AER Units are called Director (23) or Director General (14) and range from AS5 to EX4 (or equivalent) in classification.

As in 1995, the heads of these Units are most often EXs (33). They are usually EX1 (13) or EX2 (12). Less frequently, Unit Heads are ASs (8) and ESs (2). This is similar to 1995.

This year the questionnaire asked for salary bands in cases where Unit heads do not use the standard federal public service classification scheme. Combining this information with the current salary ranges for the AS, ES and EX groups allows us to present a table of the maximum salaries of 55 heads of AER Units:

Q6: Salary Range of Unit Head

Maximum of salary range within...	Number of Unit Heads
$56,000 to $60,999	1
61K - 65K	1
66K - 70K	2
71K - 75K	7
76K - 80K	3
81K - 85K	2
86K - 90K	15
91K - 95K	1
96K - 100K	12
101K - 105K	1
106K - 110K	9
111K - 125K	0
126K - 130K	1

How to read this table: "7 Unit Heads receive salaries for which the salary range tops out at between $71,000 and $75,999", etc.

1.7 Formal Hierarchy

Question 8 asked for the line reporting relationship, that is, who supervises the Head of the Unit. All 56 respondents answered this question. 26 of these Unit Heads report directly to the chief executive officer of the institution -- the DM, President, Chairman etc. (The 1995 figure was 20 out of 51 respondents.)

20 report to a senior executive one level lower (the same figure as in 1995). 10 report to a position two levels below the chief executive officer; in 1995, 11 reported to a position two or more levels below the chief executive officer.

51 respondents indicated the title of their supervisor:

Q 8: Title of Supervisor of Unit Head

Title	Number of Units
Most Senior Official	14
Associate DM	3
ADM or DG of Policy and Planning or equivalent	8
ADM or DG of Corporate Services or equivalent	11
Position integrating Policy/Planning and Corporate Services	2
Other	13

How to read this table: 14 heads of units reported directly to the most senior official in the Institution", etc.

1.8 Committee Structure

Functionally, Unit Heads often report to a committee. 40 institutions are reported to have a senior level committee for AER purposes (question 21a). In the majority of cases, this committee covers all three major areas (question 21b):

Question 21b: Areas covered by committee

Scope of senior level committee	number of institutions
audit alone	6
review alone	2
audit and evaluation	5
audit and review	3
evaluation and review	1
audit, evaluation and review	23

How to read this table: "6 senior level committees deal with audit alone", etc.

In eight cases, a senior level committee for all three main areas also deals with other topics. These include inspections, investigations, special studies, special studies at DM's request, CSA, research, assistance to management and management-lead reviews.

In two instances, respondents identified "other" duties as well for a committee that focuses on audit: financial results in one case, quality in the other.

In addition, one agency's audit Unit deals both with the internal senior Audit and Evaluation committee and with the Finance and Audit Committee of the agency's Board of Governors.

These committees convene from once to 12 times per year (question 21c). The average is four meetings; the most typical frequency is three meetings (11 institutions).

Just over half the committees meet as needed, rather than on a regular schedule (question 21d). The schedule of meetings (question 21e) is decided by:

• the Head of the Unit (11);
• the committee itself or the committee chair (7);
• the DM or equivalent (4) or another senior official (3);
• the Board of Directors (1);
• institution by-laws (1) or ISO requirements (1).

In just under half the cases, this committee is the same as the institution's most senior committee, such as the Departmental Executive Committee (question 21f). Where it is a distinct committee, its name usually contains the terms 'audit and evaluation'(10 instances out of 20) or 'audit'alone (5).

In 7 cases out of 45, there is regional representation on the committee (question 21g), and in 9 cases persons other than government officials are involved; otherwise, the committees are made up of senior officials from headquarters. In 10 cases, the AER Unit is represented on this Committee. Moreover, in 21 out of 39 cases (question 21i), there may be a representative from the OAG (9), from TBS (7), or from the area under study (2), as well as AER staff (5). Private auditors (2) or a PCO representative (1) might also attend.

The committee is chaired (question 21h) most often by the DM, President or other most senior official (25 cases out of 41); an ADM (3); a Board Member, Commissioner or Governor (5) or the chairperson of the Board (2). In only two cases does the Head of the AER Unit chair the committee.

1.9 Formal approvals

Formal approval (question 22) of the various stages and elements of audit, evaluation and review work are the responsibilities of the offices and committees indicated in the following table. In each case, several of the 49 respondents indicated more than one approval pattern. The table shows that the senior committee for AER concentrates on broad planning and on final results. While Unit Heads have greater influence on individual projects, much of the authority for approving Terms of Reference and Work Updates is in other hands:

Question 22: Who is responsible for formal approvals

	Senior level committee for AER	Another senior committee	Most senior official	Supervisor of Unit Head	Unit Head	Other
Strategic Plan	18	3	6	5	12	1
Work Plan (annual, multi-year)	37	4	7	4	3	1
Terms of Reference	12	4	2	5	22	9
Work updates	8	2	6	5	23	4
Final reports	22	5	5	2	15	6

How to read this table: " In 12 cases, the Units'Strategic Plan is approved by the Unit Head", etc.

Other:

• Strategic Plan: independent council (board of directors);
• Work Plans: Planning and Priorities Directorate;
• Terms of reference: other ADMs, client area managers, study steering committee, independent council (board of directors);
• Work Updates: study managers;
• Final reports: associate DMs, study managers, client area managers, study steering committee.

42 respondents also indicated who approves the Human Resources Plan:

	Senior level committee for AER	Another senior committee	Most senior official	Supervisor of Unit Head	Unit Head	Other
Human Resources Plan	6	4	10	13	12	1

How to read this table: "In 12 cases, the Units'Human Resources Plan is approved by the Unit Head", etc.

CHAPTER 2 RESOURCES AND PEOPLE

2.1 Unit Resources

Basic resource information was collected from 53 responding Units (question 9a).

Because many Units used the 'totals'line only, the most reliable information is for total budgets.

53 Units provided information on FTEs. Assuming that "authorized FTEs" and "actual FTEs" are equal in the 3 instances where Units reported one but not both, we find that about 722 FTE were devoted to this field by the responding Units in the year 1998-99. Although 20 of these FTEs were in excess of authorized levels, more Units lost (16) than gained (10) FTEs.

Q 9a: Unit Resources

FTE authorized	FTE actual	Salary $000	Professional service contracts $000
701.6	722	45,526.7	36,151.5

How to read this table: "In total, 701.6 full-time staff equivalents were authorized for the Units in the study", etc.

About $46M was spent on salaries, and about $36M on contracts.

In addition, 41 Units reported expenditures of $1M for travel and $7M for other O&M expenditures.

For the 41 Units that provided a breakdown, here are the relative figures among evaluation, audit and review in their 1998-99 budgets:

Q 9a: Unit Resources

	FTE authorized	FTE actual	Salary $000	Professional service contracts $000
Evaluation	125.8	176.6	9289	13130
Audit	149.2	161.7	9368	5789
Review	98.5	111	5863	2021
TOTAL	373.5	449.3	24,520	20,940

How to read this table: "125.8 full-time staff equivalents were authorized for evaluation staff ", etc.

Too few Units used the more elaborate version of the budget question, which also asked for internal informatics expenditures, other overhead expenditures, investigations and other details, to warrant presentation.

2.2 Resource Expectations

Respondents indicate an expectation of $146.1M and 702 FTEs in 1999-2000 (question 9b). While the FTE figure can be compared with the current FTE figures (above), there is no comparable current figure for current total budget.

2.3 Resource Level Trends

Respondents were asked (question 18) about the impact of Program Review, integration and restructuring since 1994-95 on their Unit's resources relative to those of their institution. 10 said they now form a larger part of the institution than before that period, 17 said that they had remained at roughly the same proportion, and 20 said they formed a smaller portion.

2.4 Resources Transferred In

Questions 9c and 9d touch on resources within the Unit's budget versus resources from other parts of the institution.

40 Units indicated that they had received transfers of resources from elsewhere in the institution in the past. This distribution is about the same as it was in the 1995 study.

Q 9c: Transfers of resources from other groups within the organization

Always	Often	Occasionally	Rarely	Never
6	7	17	10	9

How to read this table: "17 Units occasionally have received transfers of resources from elsewhere in the institution", etc.

The majority of Units funded at least 80 percent of their work out of their own budget allocation in 1998-99:

Q 9d: Percentage of work in 1998-99

funded out of own budget

Percentage of Unit work funded out of own budget, 1998-99
0-49%	6
50-79%	3
80-89%	6
90-99%	13
100%	19

How to read this table; "13 units funded 90 to 99%

of their work from their own budget", etc.

2.5 Contracting

53 Units indicated the percentage of their work that is done on contract (question 10). The majority of the 12 Units that contract out 76-100% of their work are in small Institutions; in some cases, the contractor drafts the review plan for the client as well as carrying out the work. However, Institution size clearly does not correlate consistently with percentage of work contracted out; other factors must be at play as well. For instance, several respondents explained by phone that contracting for AER work is a temporary measure pending creation (or resumption) of an in-house AER service.

Q 10: Percentage of unit's work contracted out, in dollars

% contracted out	# Units
0-10%	8
11-25%	13
26-50%	11
51-75%	9
76-100%	12

How to read this table: "12 units contract out 76-100% of their work", etc.

2.6 Classifications and Levels

Question 14 sought information on staff classifications (groups and levels) for seven groupings in terms of function and seniority. 51 Units responded to this question.

Unit Management

While the majority of respondents used the terminology provided, many did not; therefore there was some interpretation of data for this question (e.g. interpreting 'director'as 'senior manager'). Senior Managers and Managers were defined as having overall managerial responsibilities for AER Units. The most senior positions in AER are held, generally, by AS7s and EX1s to EX3s. At the next level of Unit management, the AS7s, ES6s and EX1s predominate.

Q 14: Senior Managers

	Audit	Evaluation	Review	Other	More than one	Total
AS5	0	0	0	0	1	1
AS7	1	0	0	2	3	6
ES6	0	2	0	0	0	2
ES7	0	1	0	0	0	1
EX1	5	1	0	1	5	12
EX2	0	1	2	0	7	10
EX3	1	1	0	3	2	7
EX4	0	0	0	0	1	1
other	0	1	2	0	5	8

How to read this table: "There is one Senior Manager, covering audit alone, with an AS7 classification", etc.

Q 14. Managers

	Audit	Evaluation	Review	Other	More than one	Total
AS5	0	0	0	1	0	1
AS7	13	6	3	1	0	23
AS8	1	0	1	1	0	3
ES5	0	0	0	0	1	1
ES6	0	13	1	0	0	14
ES7	0	1	0	0	1	2
EX1	1	3	2	2	2	10
EX2	0	0	0	2	0	2
CO3	0	1	0	0	0	1
other	5	1	0	0	2	8

How to read this table: "13 Managers, covering audit alone, have an AS7 classification", etc.

Professional Staff

The question offered four categories of non-managerial professionals: project manager, and senior, intermediate or junior professional.

While the majority of respondents used the terminology provided, many did not; therefore there was some interpretation of responses to determine which professional staff to allocate to the junior, intermediate and senior levels.

The following tables present this information.

Q 14: Project Managers

	Audit	Evaluation	Review	Other	More than one	Total
AS5	0	0	0	0	7	7
AS6	34	0	0	1	3	38
AS7	14	12	16	11	5	58
AS8	0	0	4	0	0	4
ES4	0	4	0	0	0	4
ES5	0	1	1	0	2	4
ES6	0	2	1	0	3	6
EX1	0	0	3	1	0	4
FI4	0	0	3	0	0	3
other	1	0	0	0	1	2

How to read this table: "34 Project Managers for audit alone have an AS6 classification", etc.

Q 14: Senior Professionals

	Audit	Evaluation	Review	Other	More than one	Total
AS4	1	0	0	0	0	1
AS5	27	1	1	1	2	32
AS6	32	6	22	3	9	72
AS7	1	0	2	0	7	10
ES3	0	1	0	0	1	2
ES5	0	18	2	0	0	20
ES6	0	8	1	0	0	9
FI2	1	0	0	0	1	2
FI3	0	0	5	0	0	5
CO2	0	2	0	0	0	2
CS3	0	0	0	0	1	1
ED-EDS-4	0	1	0	0	0	1
OM5	0	0	0	1	0	1
PM5	0	0	0	1	0	1
PM6	0	1	1	0	0	2
other	17	6	4	1	1	29

How to read this table: "27 Senior Professional staff doing audit alone have an AS5 classification", etc.

Q 14: Intermediate Professionals

	Audit	Evaluation	Review	Other	More than one	Total
AS2	0	0	0	2	1	3
AS3	2	0	0	1	5	8
AS4	5	0	0	0	5	10
AS5	12	1	24	1	1	39
AS6	0	0	1	1	0	2
ES4	0	21	0	1	0	22
ES5	0	4	0	0	0	4
FI2	2	0	4	0	0	6
CS3	0	0	1	0	0	1
IS4	0	1	0	0	0	1
PM3	0	0	0	0	1	1
other	6	3	10	0	1	20

How to read this table: "5 Intermediate Professional staff doing audit alone have an AS4 classification", etc.

Q 14: Junior Professionals

	Audit	Evaluation	Review	Other	More than one	Total
AS1	0	0	0	2	0	2
AS2	0	0	0	5	5	10
AS3	1	1	4	0	0	6
AS4	1	0	0	2	0	3
AS5	0	0	0	1	0	1
ES2	0	2	0	0	0	2
ES3	0	7	0	0	0	7
IS2	0	0	0	1	0	1
OM2	0	0	1	1	0	2
SI3	0	1	0	0	0	1
other	0	0	5	2	0	7

How to read this table: "4 Junior Professional staff doing review alone have an AS3 classification", etc.

Support Staff, Other Specialists

As the following table shows, AER Units identified both technical and administrative support positions:

Q 14: Support

	Audit	Evaluation	Review	Other	More than one	Total
AS1	1	4	2	1	5	13
AS2	3	3	0	3	4	13
AS3	0	1	0	2	21	24
AS4	0	1	0	0	0	1
AS5	0	0	0	1	0	1
SI2	0	1	0	0	0	1
SI3	0	0	0	2	0	2
CR	4	4	3	7	2	20
OCE	0	0	0	2	1	3
SCY	4	2	1	8	2	17
other	3	1	1	12	3	20

How to read this table: "3 Support Staff assisting audit alone have an AS2 classification", etc.

Finally, the following other positions show the diversity of specialities appearing in AER Units:

Q 14: Other Specialists

		Audit	Evaluation	Review	Other	More than one	Total
Central Agency Co-ordinator	AS6	0	0	0	2	0	2
Senior Informatics Officer	AS6	0	0	1	0	0	1
Support Manager	AS6	0	0	0	1	0	1
Fraud Investigator	AS6	0	0	0	4	0	4
Reporting Manager	CO3	0	0	0	0	1	1
System Analyst	CS2	0	2	0	2	0	4
Chief, Technical Services	CS3	0	0	0	2	0	2
IT Auditor	CS3	6	0	0	0	0	6
IT Project Leader	CS4	3	0	0	0	0	3
IT Manager	CS5	1	0	0	0	0	1
Publishing Officer	GT2	0	1	0	0	0	1
Quality Review	PG6	4	0	0	0	0	4
ADR Manager	PG6	0	0	0	4	0	4

How to read this table: "6 IT Auditors doing audit alone have a CS3 classification", etc.

2.7 Staff Qualifications

The following tables, reporting questions 35-37, show the educational attainment and specialization of managers and officers in AER Units.

With respect to data quality, it should be noted that some responding Units were unable to account fully for all members.

Q 35: Educational attainment of managers and officers

Highest Education Level	Evaluation	Audit	Review	Multi-functional E/A/R	Total
Secondary	5	13	19	29	66
College	0	12	13	3	28
Bachelor	22	61	45	110	238
Master	45	16	15	48	124
Ph.D.	8	1	6	6	21
Other	0	8	0	10	18
Total	80	111	98	206	495

How to read this table: "Out of 80 evaluators, 45 hold Masters degrees", etc.

As was the case in the 1995 study, staff holding bachelor degrees predominate overall and for each type of work except evaluation. There are again, relatively few managers or staff with doctorates. The entries under "other" were almost all accounting degrees (and one Registered Nurse). These will show up in the table on designations. 45 Units answered this question as well as question 37.

Q 36: Post secondary degrees of managers and officers - area of concentration

Degree Category	Evaluation	Audit	Review	Multi-functional	Total
Administration/ Management	12	28	14	44	98
Accounting	2	25	5	27	59
Economics	12	4	6	12	34
Arts	7	7	20	18	52
Law	0	1	1	1	3
Science	9	4	2	17	32
Social Science	32	10	8	44	94
Engineering	3	3	0	10	16
Computer Science	1	3	1	11	16
Other	10	1	2	3	16
Total	88	86	59	187	420

How to read this table: "Out of 86 auditors with a post-secondary degree, 25 have their degree in accounting", etc.

Q 37: Professional designations of managers and officers

Designation	Evaluation	Audit	Review	Multi-functional	Total
Chartered Accountant (CA)	0	18	1	16	35
Certified Management Accountant (CMA)	0	16	3	11	30
Certified General Accountant (CGA)	0	9	7	18	34
Certified Internal Auditor (CIA)	0	8	5	7	20
Certified Public Accountant (CPA)	0	0	0	0	0
Certified Management Consultant (CMC)	0	0	0	0	0
Certified Information Systems Auditor (CISA)	0	5	2	6	13
Professional Engineer (PE)	1	1	0	4	6
Other	4	11	3	13	31
Total	5	68	21	75	169

How to read this table: "Out of 68 auditors with professional designations, 18 are CAs", etc.

Note: only 37 Units answered this question. Units with no audit functions may simply have skipped the question.

A number of other designations were recorded as "other":

• 9 Certified Fraud examiners in audit, review and multifunctional Units;
• auditors also have IMA, CQA-ISO Auditor, CRF, AACI, ICIA, IRC, P.Ag. designations;
• elsewhere, there are one Registered Nurse and one Professional Urban Planner.

Comparing this table to its match in the 1995 study, it appears that the incidence of CAs, CMAs and CGAs is now more evenly spread across audit and multifunctional Units.

2.8 Personal Characteristics and Language Skills

Questions 38 and 39 asked about the employment equity characteristics and official language skills profile of AER Units. Only 43 Units provided information with respect to question 38 and 42 Units for question 39; moreover, some of the responding Units appeared to provide incomplete information.

Q 38: Employment equity composition

Designated Group	Evaluation	Audit	Review	Multi-functional	Support	Total
Women	57	56	9	96	69.5	287.5
Francophone	45	50	17	74	54	240
Disabled	0	1	0	4	0	5
Visible Minority	16	13	1	6	1	37
Aboriginal	2	3	0	2	0	7

How to read this table: "Of 287.5 women in AER Units, 57 are evaluators", etc.

Q 39: Unit's official language composition

Number of Staff	Evaluation	Audit	Review	Multi-functional	Support	Total
Unilingual English	34	26	8	38	10	116
Unilingual French	0	0	0	1	3	4
Bilingual - Oral "B"	45	27	7	70	18	167
Bilingual - Oral "C"	33	13	7	43	20	116
Bilingual Oral Exemption	25	36	8	34	22	125
Total

How to read this table: "34 evaluators are unilingual English-speaking", etc.

2.9 Students

31 respondents reported, in question 11, that they do employ Co-op/summer students. This is not likely to fluctuate appreciably. Four respondents said that they would likely have more on staff in the future, 38 said they would have about the same number on staff, while 4 said they would have fewer than their current complement.

2.10 Secondments

47 respondents reported having an average of 12 staff seconded to their Unit in the past two fiscal years (question 12). There may be a decline in secondments: 3 said that they expected to have more secondees in the future, 38 said they would have about the same number and 8 said they expected to have fewer.

Secondments last from ½-year to 5 years, but the majority last either 1 or 2 years as reported by 36 respondents.

7 of the reported secondments were career audit/evaluation/review professionals, while the other 28 came from other areas such as programs and operations (13), information technology (3), and various corporate areas like administration, human resources and finance.

2.11 Special Recruitment Programs

Only 6 of 49 respondents said that they had recruited personnel from the Financial Officer or the Internal Auditor Recruitment Programs over the current and past 2 fiscal years (question 13). Other federal program recruitment sources included the Career Assignment Program (4 recruits), Management Trainee Program (6), summer student programs (2) and several from interchange and diversity programs.

2.12 Retirement

Question 15 was answered by 47 respondents to the full survey. It appears from several annotations that retirement expectations are not known accurately. Therefore, the table should be taken to indicate that at least this many staff fall into these categories.

Q 15: Number of Unit's staff, other than support staff, expected to retire over the next decade and beyond

Number of Years To Retirement	Evaluation	Audit	Review	Multi- functional	Total	Cumulative Total
0-1 year	4	1	0	7	12	12
1-3 years	9	17	1	10	37	49
4-6 years	9	7	6	25	47	96
6-8 years	7	10	8	20	45	141
8-10 years	17	19	10	18	64	205
More than 10 years	73	38	12	131	254	459

How to read this table: "9 evaluators are expected to retire in 1 to 3 years", etc.

2.13 Recruiting Plans

Respondents were asked (question 16) to estimate what percentage of their overall professional recruitment would come from inside or outside the public service, and at which of three general levels of experience.

By far the favourite choice was "inside the public service, with significant experience" --the source from which 14 of the 37 respondents would obtain over 50% of their requirements, and for 3 respondents, 100%. Any other trend was not particularly clear.

2.14 Information Technology Auditors

43 respondents answered a question about recruiting information technology auditors (question 17).

Q 17: IT auditors expected to be hired in 2-5 years

Number of Respondents	Number of IT auditors likely to be hired
1	0.5
12	1
3	1.5
5	2
1	3
2	5

How to read this table: "12 respondents would hire only one IT auditor", etc.

Twenty-three respondents, out of 43, foresee trying to recruit IT auditors over the next 2-5 years. Of these, 12 said they would hire only one.

When asked from what source they would likely obtain their IT auditors, 37 of 40 respondents cited contracting out. 19 respondents would make this their only source and 32 would use contractors to meet at least 50% of their needs. Others intend to hire from within the public service (6), use secondments (5) or hire from outside the public service (4).

2.15 Learning Strategies

In question 32, respondents were asked to rank the following as sources of information on new techniques and methodology. Only 25 Units provided a ranking; 15 other Units simply checked off some or all sources. Since 25 sets of rankings would not likely be representative of the group, "hits" on each information source, whether ranked or checked, have been recorded. All the options provided had numerous hits, from 77.5% of possible mentions for "Other Departments" to 100% for "Training--other sources":

Q 32. Sources of information on new techniques and methodology

source	# of respondents	as % of maximum possible mentions
Other Departments	31	77.5%
Review Network	32	80%
Treasury Board Secretariat	33	82.5%
Office of the Auditor General	34	85%
Internet	34	85%
Canadian Evaluation Society	35	87.5%
Institute of Internal Auditors	37	92.5%
Conferences (other sources)	39	97.5%
Training (other sources)	40	100%
Other	14

How to read this table: "31 out of 40 responding Units get information on new techniques and methodology from other departments", etc.

Other sources include: the American Evaluation Society, CCAF, consultants and industry, U.S. Government Training Institute, "like-minded agencies" internationally and the Bureau d'évaluation des organisations internationales.

The actual training situation is approximated by the information collected by question 33 on money and person-days devoted to training Unit employees in 1998-99.

Q 33 (1): Amounts spent on training in 1998-99

Type of Training	Training $ Spent	Units
Audit Techniques	$263,588	25
Evaluation Techniques	$174,724	21
Performance Measurement	$67,025	20
Information Technology	$72,181	14
Control Self-Assessment	$73,900	9
Facilitation	$93,650	8
Review Techniques	$9,850	7
Quality Management	$10,000	4
ISO 9000/14000	$25,000	2
Service Standards	$3,000	1
Environmental Management Systems	$20,000	1
Other	$153,990	29

How to read this table: "25 Units together spent a total of $263,588 on training in audit techniques", etc.

Q 33 (2) : Person-days spent on training in 1998-99

Type of Training	# Person Days in Training	Units
Audit Techniques	807	22
Performance Measurement	172	18
Evaluation Techniques	375	17
Information Technology	241.5	16
Control Self-Assessment	145	9
Facilitation	252	8
Review Techniques	40	6
Quality Management	31	4
ISO 9000/14000	111	3
Environmental Management Systems	4	3
Service Standards	11	2
Other	573	28

How to read this table: "22 Units together spent 807 person days on training in audit techniques", etc.

For the 40 Units responding to this question, the most common types of training sought in 1998-99 were on audit and evaluation techniques, performance measurement and information technology. By far the greatest total resources were spent on audit techniques ($263,588 and 807+ days) and evaluation techniques ($174,724 and 375+ days). (Note that some Units reported the dollar expenditure but not the person-days; for this reason, the actual person-days must have been more than 807 and 375 respectively. Furthermore, Units may have reported person-days but no financial expenditure for training provided within their institution. For these reasons, the dollars and the person-days should not be related to each other.)

Other types of training noted were: information technology (6), treasury, comptrollership (4), professional and personal development (3), university and college courses (3), project management (2) and language (2). Respondents also noted training in: fraud topics, teamwork, writing, benchmarking for audit, governments and global markets, internal programs, market research, retirement planning, support staff (not specified), and conferences.

Question 34 asked for Units'most significant future training needs. 32 Units mentioned a wide range of topics:

• control self-assessment (11 mentions);
• IT and information systems (10) and IT audit (3);
• performance measurement (6);
• audit, evaluation and/or review techniques in general--basic (5), advanced (3);
• facilitation (5);
• risk management (4);
• comptrollership (4);
• results based management (3);
• 2 mentions each for project planning and management, environmental management, management trends, communications;

one mention each: new concepts and tools, attest audit, data analysis (ACL), qualitative measurement, cost-benefit analysis, report preparation, presentations, OAG orientation, time reporting, team skills, Internet, service standards, critical thinking, rational problem-solving, writing.

CHAPTER 3 POLICIES AND PRACTICES

3.1 Work Planning

All 49 Units answering the full survey have a work plan (question 19).

• Of the 41 out of 49 Units indicating (in question 5) that they do audit, 36 had an audit plan, while 5 fell under a review plan.
• 31 out of 36 Units doing evaluation had an evaluation plan, while 4 others fell under review, and one under "other".
• Of the 38 Units indicating that they carry out reviews, 32 had a review plan, with all but one of the others being evaluation/audit plans.

Most plans are issued on a yearly basis:

Q 19b: Work plans issued yearly or less frequently

Time	Evaluation	Audit	Review	Other
½ year	1	1	0	1
1 year	26	32	30	21
3 years	1	1	1	1
5 years	0	1	0	0
Varies	2	1	0	3

How to read this table: "26 evaluation Units issue the work plan yearly", etc.

It is clear from this table that almost all Units use a yearly work plan. However, the next table illustrates the fact that half of the plans cover a longer time span than one year, so the annual plans are partly revisions of previous plans. Plans covering 1 or 3 years are by far the most common.

Q 19c: How many years does the plan cover

Time	Evaluation	Audit	Review	Other
½ year	1	1	1	1
1 year	12	13	10	11
2-5 years	14	20	17	10
varies	3	2	2	3

How to read this table: "12 evaluation Units'plans cover 1 year", etc.

When asked (question 20) what formal mechanisms their Unit used to determine which areas to include in the work plan, respondents answered as follows:

Q 20: Formal mechanisms for choice of work plan elements

Mechanism	# of Respondents
Multi-Year Cycle	32
Legislative requirements and conditions of program funding	33
Significance and risk assessment	50
Interviews with departmental management	50
Control Self-Assessment	13
Other	14

How to read this table: "32 Units choose work plan elements on the basis of a multi-year cycle", etc.

Other means to decide the plan's contents included environmental scanning, monitoring and business intelligence, management and Executive Committee concerns and priorities, strategic planning, and stakeholder requests.

3.2 Participation and Resulting Action by Program Managers

Two questions (questions 23 and 24) dealt with the involvement of program managers in AER studies and the subsequent actions required.

Question 23 asked if program managers were generally required or invited to provide input and commentary, aside from providing data, on the findings/recommendations of reports on their program or area before the report is finalized. This question was asked only of the 49 units answering the full survey.

Participation in studies is the norm. It is required slightly more than half the time, but is almost always either required or invited. For example, of the 36 out of 49 units which said that they did program evaluations, a total of 33 units either required or invited participation in evaluations. The rate is higher for audit and review.

Q 23: Input and commentary of Program Managers

Involvement of Program Managers	Evaluation (of 36)	Audit (of 41)	Review (of 38)
Required	20	22	20
Invited	13	18	17
Neither	0	0	0

How to read this table: "20 Units reported that program managers were invited or required to participate in evaluations", etc. Note: Almost all respondents replied to this question, however, up to 3 of the Units who said they work in a given function failed to respond. These may likely be interpreted as cases where the response "Neither" also applies. The balance of responses to add up to 49 were, for each function, made up of responses of "not applicable" or were non-responses.

Question 24a and 24b asked if management responses or action plans were required before approval of the evaluation, audit or review, and if their implementation was subsequently monitored.

Q 24a: Management responses before approval of reports

Response/Action Plan	Evaluation	Audit	Review
Always Required	20	32	22
Sometimes Required	10	5	11
Never Required	3	3	4

How to read this table: "20 Units require responses or action plans from program managers for evaluations before reports are approved" etc. The balance of responses to add up to 49 were, for each function, made up of responses of "not applicable" or were non-responses.

Q 24b: Monitoring the implementation of responses and action plans

Monitoring	Evaluation	Audit	Review
Always	19	26	19
Sometimes	10	11	15
Never	4	2	2

How to read this table: "19 Units monitor the responses and action plans of managers in response to evaluation reports", etc. The balance of responses to add up to 49 were, for each function, made up of responses of "not applicable" or were non-responses.

The responses to question 24a indicate that just more than half of the Units having conducted a study always require a response or action plan, and almost all (33 out of 36 possible for evaluation, 40/41 for audit, and 33/38 for review) require responses or action plans at least sometimes.

Responses to question 24b show that a smaller proportion of Units necessarily follow up on the implementation of the responses and action plans. Further analysis reveals that:

• Of the 20 Units which always require a response to evaluations, only 13 said they always monitor them, while 5 of these said they sometimes do, and 2 never do. Of the 10 that sometimes require a response, only half always monitor its implementation and the rest sometimes do. In one case, a unit which never requires a response always monitors responses or action plans.
• Of the 32 Units which always require a response to audits, 22 said they always monitor them, while 8 of these said they sometimes do, and 1 never does. Of the 5 that sometimes require a response, only 3 always monitor its implementation and 2 do so sometimes. In 2 cases, units which never require a response always or sometimes monitor responses or action plans.
• Of the 22 Units which always require a response to reviews, 13 said they always monitor them, while 7 of these said they sometimes do, and 2 do not. Of the 11 that sometimes require a response, only 5 always monitor its implementation and 2 do so sometimes. In 3 cases, units which never require a response always or sometimes monitor responses or action plans.

Units monitoring the implementation of responses and action plans were asked how often they follow up on this.

Q 24c: Frequency of follow-up

Frequency	Evaluation	Audit	Review
Every 3 months	4	5	4
Every 6 months	4	5	2
Yearly	8	13	9
More than yearly	3	4	5
Other	10	10	12

How to read this table:"4 Units said evaluations are followed-up every 3 months", etc.

2 of the relevant units did not respond regarding review.

This result resembles that of the 1995 study, where "yearly" or "other" were the most frequent choices. Under the category of "other" the most frequent entry was "as needed or required" (5 times for evaluation and audit and 7 cases for review). Longer times for follow-up were indicated in "other" but these were included in the table as "More than yearly." Other responses showing up once or twice included "varies, informal, not regularly, as time permits, as in multiyear plan." Interestingly, "depending on the nature of the project/on the nature of the recommendation and action plan" only shows up from two Units.

Units which monitored implementation of responses and action plans were asked if they report the results of the activity.

Q 24d: Reporting the results of monitoring

Report Results	Evaluation	Audit	Review
Always	15	20	15
Sometimes	10	15	15
Never	2	0	0

How to read this table: "16 Units said they always report the results of monitoring action plans", etc. 2 relevant units did not respond for both evaluation and audit, and 4 for review indicated that this was "not applicable", or did not respond.

Most, but not all, Units monitoring implementation of action plans or responses finally report on this to the AER committee or equivalent body. This would be consistent with an approach where a report is made only if necessary.

3.3 Service Offerings

Question 29 asked Units to identify their current activities, projects and services, in 1997-98 and 1998-99, in terms of the proportion of time spent on each; whether they consider these activities to be Core or Non-core; and trend in demand.

The three aspects of the question were not answered consistently. However, a good indication of the prevalence of service offerings is if there was information under at least one of the three headings. On this basis, the current service offerings of 48 Units include:

Q 29: Activities, Projects and Services

Activities, Projects, Services	Number of Units Offering the Service
Audits	38
Y2K audits	35
Performance measurement	33
Program Evaluations	32
Assessment of management processes and controls	26
SUD audit	26
Evaluation frameworks	25
Special investigations	23
Other IT audits	20
Implementation audit	19
Q 29: Activities, Projects, Services, continued	Number of Units Offering the Service
Client/ Stakeholder survey	18
Control self-assessment	18
Evaluation of ASD	17
Regulatory review/ evaluation	17
Service Standards	15
Policy evaluation	15
Environmental audit	14
Total quality management	13
Employee survey	11
Other	21

How to read this Table: "38 of 48 responding Units have included audits in their service offerings for 1997-98 and 1998-99", etc.

Other activities include: review of implementation of legislation; reviews (2); ISO9000 (2); management assistance; risk; OAG liaison (2); interdepartmental projects; third party audits; research on learning in museums; ethics program and policy; strategic planning; special studies; quality assurance; performance reports; information management; inspections; advise and assist projects.

Q29: Core and Non-Core Activities

Activities, Projects, Services	Core	Non-Core
Audits	31	0
Program Evaluations	28	0
Performance measurement	25	2
Assessment of management processes and controls	20	2
Y2K audits	20	6
Evaluation frameworks	19	3
SUD audit	18	3
Other IT audits	14	2
Regulatory review/ evaluation	13	2
Special investigations	13	7
Policy evaluation	11	2
Implementation audit	9	5
Client/ Stakeholder survey	8	4
Evaluation of ASD	7	7
Control self-assessment	7	8
Environmental audit	6	4
Total quality management	5	6
Service standards	4	8
Employee survey	4	5
Other	14	4

How to read this Table: "25 of 40 responding Units regard performance measurement as a Core offering, and 2 units regard it as a Non-core offering, in 1997-98 and 1998-99", etc.

47 Units indicated the trend in these offerings--increasing, steady or decreasing. The following table shows the services in decreasing order of prevalence, that is, in terms of the numbers of Units that indicated that these services are holding steady or increasing in importance:

Two messages from this table are that performance measurement is well entrenched as a new trend in audit, evaluation and review practices; and that while Y2K work has peaked, the demand for IT work overall is strong.

The data on percentages varied greatly in quality and does not lend itself to summary presentation.

Q29: Trends in Activities, Projects and Services

Activity, Project or Service	increasing	steady	steady + increasing	decreasing
Performance measurement	24	9	33	0
Audits	7	25	32	5
Program Evaluations	11	16	27	4
Assessment of management processes and controls	9	17	26	0
Y2K audits	12	10	22	14
Evaluation frameworks	9	13	22	3
SUD audit	9	12	21	4
Other IT audits	10	9	19	2
Special investigations	4	13	17	5
Policy evaluation	8	7	15	2
Implementation audit	4	11	15	3
Control self-assessment	8	6	14	3
Regulatory review/ evaluation	5	9	14	3
Client/ Stakeholder survey	5	9	14	4
Service Standards	6	6	12	1
Environmental audit	4	8	12	2
Evaluation of ASD	4	7	11	4
Total quality management	4	5	9	2
Employee survey	3	4	7	4

How to read this Table: "For 7 out of 47 responding Units, the demand for audits is increasing; the demand is steady for another 25 Units, and it is decreasing for a further 5 Units", etc.

3.4 Liaison with the Office of the Auditor General

Liaison with the OAG (question 30) is the responsibility of 37 out of 49 Units.

Of these, 32 provided an estimate of the time spent in this activity. The highest estimate was 36 person-months.

Q 30: Estimated time spent in liaison with OAG each year

Workload for liaison with OAG	Number of Units
Minimal	3
1 - 2 person-weeks	7
1 person-month	3
2 - 3 person months	5
4 - 6 person months	6
More	8

How to read this table: "Seven Units estimate that they spend one to two person-weeks per year on liaison with the OAG", etc.

35 respondents explained what this liaison entails. While many answers elaborate what "co-ordination and liaison" may mean--from introducing people, scheduling meetings and exchanging information through to logistical support--about one third provided details on a more active involvement. For example, the Units might scope new work, liaise on current work, challenge findings, negotiate on report contents, negotiate in their institution for responses, advise on management responses, respond to recommendations, brief and advise senior management on work done by OAG, prepare follow-up on OAG reports, monitor follow-up work and action plans, and produce briefing material for Public Accounts Committee appearances and other uses.

3.5 Workload

All 56 respondent Units indicated (question 27) how many projects they completed in 1997-98.

Q 27: Total Projects Completed 1997-98

Type of Project	Totals
Evaluations	227
Audits	517.5
Reviews	147
Management Assistance Projects	160
Other	140

How to read this table: "Respondents said they completed 227 evaluations in 1997-98", etc.

There is considerable variation in the number of projects completed in any one category, so reporting a mean would be misleading. For evaluation, review, management assistance and other projects, about half of the Units had completed no projects in that year, while about one third of audit Units had completed none. On the other hand, for each category, a relatively small group of Units account for a high percentage of projects. These results follow from the marked disparity of the size of evaluation/audit/review Units and in the types and depth of projects done.

Few respondents specified what "other" projects were.

Of the 56 Units, 33 said that their workload for that year was typical, while 17 said it was not. In the latter case, most of these Units were involved in reorganization or change, in their first year or developmental stage, occupied with other types of assignment such as performance measurement or quality assurance, or at a given point in their cycles where projects were not reaching completion. This is only one proxy measure for workload. Another measure would be the number of projects underway.

Of the above 1,191 projects completed, 253 (21%) were characterized as unplanned special requests.

3.6 Publication of Reports

Questions 41 and 42 dealt with Units'policies (41) and mechanisms (42) with respect to publication of reports. The policy question was open-ended and received 48 clear responses. 42 out of 48 indicate that reports are available to the public; the other six restrict reports to internal readership. Note that not all responding institutions are bound by the TB Review Policy, which requires that reports be made available to the public. 46 respondents mentioned an average of 2 mechanisms each for making reports accessible to the public:

Q42: Mechanisms to make reports available

in departmental library	Unit web site	Departmental web site	TBS web site	other
23	13	19	17	21

How to read this table: "23 Units place their reports in the departmental library", etc.

Among the "other" responses, 10 indicate that they are reactive, sending the reports upon request (usually via ATIP). Another 4 send the reports automatically to particular audiences: union representatives, industry representatives, organizations that were involved in the study.

3.7 Quality Assurance

Question 25 asked if the Unit had any formal mechanisms for quality assurance in their evaluation, audit, review or management assistance work.

A small majority of Units use formal mechanisms to assure quality in their work (63% of those doing evaluation; 63% for audit; 68% for review; and 48% for management assistance).

Q 25b: Formal quality assurance mechanisms

	Evaluation	Audit	Review	Management
Client survey	11	14	14	8
Post project interviews	4	5	6	4
Employee feedback surveys	4	5	6	5
Internal progress reports	8	15	12	8
Project management systems	10	11	9	8
Annual reports on Unit's work	19	20	18	13
Periodic independent assessments	6	10	11	7
Other	5	5	7	4

How to read this table: " 11 Units said they used client surveys to check on the quality of their evaluation work" etc. Note: "Post project interviews" did not appear on the French version of the questionnaire. Therefore, this category may be slightly under-represented.

The most prominent means of quality assurance were annual reports on the Unit, internal progress reports and client surveys. Client feedback was noted as being an important means of quality assurance in the 1995 study, as well.

Other means recorded for quality assurance included "performance review, annual appraisals, and time reporting systems." These represent quality control at the individual level. Annual reports to the OAG and other senior committees were cited several times. One group used control self-assessment.

3.8 Productivity and Service Quality

Question 31 asked "What techniques, tools or processes have been introduced over the past 5 years in order to increase your Unit's productivity and service quality?" Note that the question asks about innovation in the past five years, not the total incidence of various approaches. 28 respondents mentioned one or more approaches.

The frequently-mentioned approaches fall into four domains: general principles of good management; good management specific to AER Units; automation; and planning focus on risk and control self-assessment. Here are examples:

• general principles of good management include good staff and management; full and frank feedback on performance; SMART Goals and related rewards; criteria for good management; performance measurement; core team matrix; de-layered management; empowerment; work flexibility; project management system; strategic planning retreats;
• good management of AER Units includes particular methods (audit tools, concurrent review methodology, evaluation frameworks, standard audit reports); publications (Review Handbook, procedures manual); partnership and collaborative approaches to evaluation; dedicated planning and practices unit that deals with quality assurance; quality initiatives; quality checklist and published service standards;
• automation includes ACL software (mentioned more frequently than any other); audit tracking and management software; Lotus Notes GroupWare and Lotus workflow platform for audit process; option finder technology; flowchart software; software for data analysis; automated working papers; presentation software; and one mention of hardware--laptops for on-site audits;
• planning on the basis of control self-assessment, risk assessment models, and increased focus on high risk areas, comptrollership and management issues.

Five other areas were mentioned less frequently:

• ISO 9002 Certification, TQM, Balridge criteria, National Quality Institute;
• training and professional development;
• client surveys and focus groups for project feedback;
• influence of the Audit/Evaluation Committee;
• annual report.

3.9 Information Technology and Communication

Units were asked (question 40) about their use of several types of computer applications. Of the 47 Units responding:

- 19 Units said that they use a computer-based time reporting system.

- 14 Units said they use a computer-based project management system.

- 14 Units said they use automated working papers.

3.10 Activities Outside the Unit

Respondents were asked (question 26) if other groups in their institution conduct projects which are essentially evaluation, audit or review. Almost all said this happened frequently (21) or at least infrequently (22). Only 6 said this never occurred.

Where other parts of the institution do engage in AER activities, question 26 also asked if the AER Units has any involvement:

Question 26b: Involvement of Unit in other group's studies

For the most part, AER Units sometimes assist other units carrying out their own studies. Cross-divisional assistance happens least in audit.

36 respondents commented on the manner of involvement. The following were mentioned more than once:

• advice on terms of reference (6) and methodology (7);
• participating in study committees (4);
• advising on or helping to choose contractors (2);
• contract management (2);
• reviewing reports (3).

3.11 Joint Projects

Question 28 asked how often Units worked on joint projects with partners outside their immediate Unit organization.

Question 28: Work on joint projects with outside partners

Partners	Often	Occasionally	Rarely
Other groups in the department	17	20	7
Other departments	6	22	9
Other governments	5	9	12
Non-governmental organizations	4	7	9
Industry	0	4	10
Other	0	1	2

How to read this table: "17 Units said they often work on joint projects with other groups

in their department", etc.

While almost all respondents reported whether or not they worked with other groups in their institution, a substantial number did not report anything at all for the other categories. This can be interpreted to mean they did not know the answer or that they simply left blanks instead of specifying "never". For this reason, the "never" responses are probably seriously under-counted and are not reported here.

Most collaboration by respondents to this survey is done within the Unit's own institution or with other institutions within the federal government, as was the case in the 1995 survey. Roughly the same proportion of Units work jointly with other governments as was the case in 1995.

---

APPENDIX

QUESTIONNAIRE

Introduction

This questionnaire has been designed by and for Heads of Review in the Government of Canada. It replicates, with some modification, a survey carried out in 1995 and reported under the title A Profile of Audit, Evaluation and Review Units in the Federal Government in 1995.

Please make every effort to complete the questionnaire. The usefulness of the results to yourself and to your colleagues depends on the completeness of coverage.

Given the great variety of institutions in the federal government, the perspectives and terminology of some questions may not match your situation very well. Please answer as well as you can. You may also seek assistance from Agora Management Associates who are conducting the survey; phone Robert Czerny (613-728-0658) or John Burrett (613-747-3444).

At the end of the survey, we invite your comments on any of your answers and on the survey itself.

Please complete this questionnaire by February 26, 1999 and return it in the envelope provided to:

Agora Management Associates

2030 Gatineau View Crescent

Gloucester, ON

K1J 7X1

Definitions and scope

For purposes of this questionnaire, please use these key terms as follows:

Institution: a Government of Canada entity named in Schedules I and II of the Financial Administration Act as a department; departmental program, division, branch or corporation (commonly called 'agency'); or crown corporation.

Unit: a discrete organizational component of an Institution that includes one or more of audit, evaluation and/or review among its main business offering(s). (It may also have other business offerings, including management assistance.)

Audit: 'internal'audits of your Institution's programs and structures, and your Unit's participation in multi-institutional or government-wide audits; exclude 'external'audit, that is, auditing provided to outside organizations that do not report to the head of your Institution.

Evaluation: evaluations of your own Institution's policies, programs and structures, and your Unit's participation in multi-institutional or government-wide evaluations; exclude 'external'evaluations, that is, evaluation services provided to outside organizations that do not report to the head of your Institution

Review: any activity that your Unit identifies as "review." For example, some Units call studies that combine audit and evaluation methodologies "reviews." Other Units use the term for those of their studies that lack some features of full-fledged, classical audits or evaluations; or for less independent (manager-led) studies to which they provide some input, e.g. study design. If your Unit has a definition of "review," please attach it.

Management Assistance: management consulting services provided by professionals of your Unit to clients within the Institution.

Other Business Offerings: any other service provided by the professional staff of your Unit to clients within the Institution. This can include providing briefings and information to executives.

---

Name of Institution

__

 Description of Institution's Business Lines

What is the nature of your Institution's business?

1. Please rank the following functions by order of importance indicating their significance to your Institution's mandate (1 being the most important). Categories which do not apply should be left blank.

_______ Regulatory

_______ Operational

_______ Policy and Legislation

_______ Government Administration

_______ Research and Development

_______ Other functions, please specify ____

2. Please rank the following public policy domains by order of importance indicating their significance to your Institution's mandate (1 being the most important). Categories which do not apply should be left blank.

_______ Defence

_______ Foreign Affairs

_______ International Trade

_______ International Assistance

_______ Social Programs

_______ Industrial Support (National and Regional)

_______ Scientific-Technological Support

_______ Transportation Programs

_______ Heritage and Cultural Programs

_______ Justice and Legal Programs

_______ General Government Services

_______ Public Debt Charges and Fiscal Arrangements

Institutional Budget

3. Please provide the following information on your Institution's budget for 1998-99 in the following format, if possible. If your Institution's summary budget information is organized differently, please use another format at a similarly high level of aggregation.

             1998-99 ($ 000)

Budget:

i) Operating              _

ii) Grants & Contributions   _

iii) Capital              _

iv) Total              _

Total FTEs              _

Description of Your Unit

4. Name of Unit (e.g., Review Branch, Corporate Audit and Evaluation Directorate, etc.):

_________

5. Business line offerings of the Unit (check all that apply )

Audit

Evaluation

Review

Management assistance

Other, please specify __

6. Title of the person who heads the Unit:

______

7. Classification and level of Unit Head: For Institutions that do not use the common occupational groups (e.g. AS, ES, PM, EX) of federal government departments, please provide the classification in full rather than acronym, and indicate the salary range in terms of closest $10K range (e.g. $70,000 to $79,999).

___

8. Reporting level:

a) Does the Unit Head report directly to the most senior official (Deputy Minister (DM) or equivalent, e.g. president)?

Yes No

If you answered " no ", answer question 8-b, if not, go to question 8-c.

b) Please indicate the number of levels between the Unit Head and the most senior official:

_____levels

(e.g. there is one level between the Unit Head and the most senior official in the case of a Director General of Corporate Review Branch reporting via an Assistant Deputy Minister (ADM) to the Deputy Minister (DM))

c) Title of supervisor of the Unit Head: (check one)

Most senior official (DM or equivalent, e.g. president)

Associate DM

Senior ADM

ADM, Policy & Planning or equivalent

ADM, Corporate Services or equivalent

Another ADM, please specify title___

Other, please specify _____

d) Does the Unit Head sit on the Institution's most senior committee (such as the Departmental Executive Committee)?

Yes
No

UNIT RESOURCES

9. Allocation of Unit Resources:

a) What is your Unit's 1998-99 budget (in $thousands)? Please show the budget's allocation to Evaluation, Audit, Review and other services, as well as the total.

Note: There are two versions of this Table. Please use the more complex Version B instead of Version A if any of the following conditions apply:

- significant travel expenditures

- significant expenditures on internal informatics from the Unit's budget

- management of the Unit is tracked as an overhead item

- significant amount of Investigation activity

- significant amount of Management Assistance activity

Version A	FTEs		Salary ($'000)	Professional Services Contracts **($'000)	Other O&M ($'000)
	Autho rized	Actual*
Evaluations
Audits
Reviews
Other Services
TOTAL

* 'Actual'may vary from 'authorized'due to personnel movements, use of casuals, etc.

**Include consulting services and other professional services obtained from outside the

institution. Do not include contract staff.

Version B	FTEs		Salary ($'000)	Profe ssional Services Contracts ** ($'000)	Travel ($'000)	Other O&M ($'000)
	Autho rized	Actual*
Internal Informatics
Other Overhead (Unit mgmt)
Evaluations
Investigations
Other Audits
Reviews
Management Assistance
Other Services
TOTAL

* 'Actual'may vary from 'authorized'due to personnel movements, use of casuals, etc.

**Include consulting services and other professional services obtained from outside the

institution. Do not include contract staff.

Question 9 continued:

b) What is your Unit's expected budget for 1999-2000 (in dollars and FTEs) ?

____$ ____ FTEs

c) Do you normally receive transfers of resources (travel money, contracts, staff) from other units or groups within your institution in carrying out evaluation, audit or review work? (check one)

Always    Often   Occasionally Rarely Never

d) Approximately what percentage (%) of the work done by your Unit in 1998-99 is being funded out of your own budget allocation, rather than through cost recovery or transfers of resources from other parts of the institution?

_______%

e) Approximately what percentage (%) of the staff of your Unit in 1998-99 are on contract (rather than salaried employees)?

_______%

10. Contracting Out:

Approximately what percentage (%) of your Unit's audit/evaluation/review work is contracted out (in terms of dollars spent on contracted projects, or parts of projects, as opposed to salaries and overhead devoted to projects conducted by in-house staff)?

______%

11. Students:

a) Do you employ Co-op and/or summer students in your Unit?

Yes No

b) Do you think you will generally have more, the same number or fewer Co-op and/or summer students on staff in the future? (check one)

More About the same Less

12. Secondments:

a) In your current and past two fiscal years, what percentage of your staff was seconded to your Unit?

______%

b) Do you think you will generally have more, the same number or fewer seconded personnel on staff in the future? (check one)

More About the same Fewer

c) How long have secondments in your Unit generally lasted?

________ Years

d) Have the staff brought into your Unit on secondment usually been career
professionals from within the review field or have they come from other
areas of expertise?

Usually career audit/ evaluation/ review professionals

Usually from other areas, please specify

13. Recruitment & Development Programs:

a) In your current and past two fiscal years, have you recruited personnel from the Financial Officer Recruitment and Development (FORD) and/or the Internal Auditor Recruitment and Development Program (IARD) for your Unit?

Yes     No

If you answered " yes", answer question 13-b, if not, go to question 13-c.

b) please indicate your degree of satisfaction with the following, where 5 is "very satisfied" and 1 is "very dissatisfied"

Information about the program(s)    1 2 3 4 5

Quality of personnel recruited         1 2 3 4 5

Administration of the program(s)    1 2 3 4 5

c) Please name any other federal government program (such as Career Assignment Program (CAP), Management Trainee Program (MTP), the Accelerated Economist Program, etc.) which you have approached as recruitment sources in the current and past two fiscal years.

_______________

14. Unit Staff:

Looking at the staff that you have now (including the head of the Unit and support staff), please complete the table on the following page.

The first column asks for level of seniority within the Unit. Please use the following descriptors:

For Managers of the Unit (not project managers): Senior Manager, Manager

For professional non-managerial staff (evaluator/auditor/reviewers and others such as IT professionals; includes management of projects): Project Manager; Senior; Intermediate; Junior

For support staff: Support

Example: entries for the upper levels of an integrated audit/evaluation shop might look like this:

Level of Seniority	Classification and Level (eg. AS-07)	Number of staff with this classification and level
		Eval	Aud	Rev	Other	Covers more than one area (explain).
Project Manager	AS - 06	2	2
Manager	AS - 07					2 Managers of both audit and evaluation

PLEASE ATTACH A COPY OF THE LATEST ORGANIZATION CHART, IF AVAILABLE.

(Note: if there is not enough space, please photocopy this page and complete the table on an extra sheet).

Level of Seniority	Classification and Level (e.g. AS-07)	Number of staff with this classification and level
		Eval	Aud	Rev	Other	Covers more than one area (explain).

 15. Retirement:

How many of your Unit's staff (including management, but excluding support staff) do you expect will retire over the next decade and beyond?

Number of Years to Retirement	Evaluation	Audit	Review	Multi-functional E/A/R
0-1 year
1-3 years
4-6 years
6-8 years
8-10 years
More than 10 years

16. Recruiting Plans:

Over the next five years, what percentage (%) of the overall professional recruitment for your Unit do you expect or intend to hire from the following sources?

Outside the public service, with:     Significant A/E/R experience     _____%

            Some A/E/R experience              _____%

             No A/E/R experience              _____%

Within the public service, with: Significant A/E/R experience   _____%

             Some A/E/R experience      _____% 

No A/E/R experience          _____%

17. Information Technology Auditors:

a) How many auditors specializing in Information Technology (IT) do you believe you will need to recruit over the next 2-5 years? _______

b) What percentage (%) of your IT auditing requirement do you expect to obtain from the following sources? Do not include support staff.

Contracted out             ___________%

Seconded                      ___________%

Hired from outside Public Service           ___________%

Hired from within Public Service             ___________%

Other, please specify ______________  ___________%

18. Resource Level Trends:

a) What has been the cumulative impact of Program Review, integration, restructuring, downsizing etc. from

   1994-95 to 1998-99 on the resources of your Unit?

FTEs 1994-95 __________ 1998-99__________

Total $ 1994-95__________ 1998-99__________

b) How does this resource trend compare with that for your Institution as a whole in the same period?

we now form a larger portion of the Institution than we did before

we now form a smaller portion of the Institution than we did before

we form roughly the same portion of the Institution as we did before

FUNCTIONING OF THE UNIT

19. Work Planning:

a) Do you prepare a work plan for your Unit detailing the projects (review, audit, evaluation and other projects) that your Unit will conduct? (check the appropriate boxes)

Evaluation Yes
Evaluation No
Evaluation N/A
Audit Yes
Audit No
Audit N/A
Review Yes
Review No
Review N/A
Other Yes
Other No
Other N/A

If you answered " yes", answer question 19-b, if not, go to question 21.

b) Is the plan issued yearly or less frequently? (check the appropriate box for each of evaluation, audit and review)

Evaluation Yearly
Every ____ years
Varies
N/A

Audit Yearly
Every ____ years
Varies
N/A

Review Yearly
Every ____ years
Varies
N/A

Other Yearly
Every ____ years
Varies
N/A

c) How many years does the plan cover?

Evaluation Every ____ years
Varies
N/A

Audit Every ____ years
Varies
N/A

Review Every ____ years Varies N/A Other Every ____ years
Varies
N/A

20. Which of the following formal mechanisms does your Unit use to determine which areas to include in the plan? (check all that apply)

Multi-year cycle (universe coverage)
Legislative requirements, conditions of program funding
Significance and risk assessment
Interviews with departmental management
Control self-assessment
Other, please specify

21. Committees

Is there a senior level Committee in your Institution for purposes of audit, evaluation and/or review?

Yes
No

If you answered " yes", answer question 21-b, if not, go to question 22.

b) Which areas does this committee cover? (check all that apply)

Audit
Evaluation
Review
Other, please specify

__

c) Approximately how often does the committee meet per year?

_____ times per year

d) Does the committee meet

As needed?
On a set schedule?

e) Who decides the schedule for the committee meetings? _________

f) Is this committee the same as your Institution's most senior committee (such as the Departmental Executive Committee)?

Yes
   No.

Please give title of the committee ___

g) Please outline the composition of the committee _______________

h) Who chairs the committee?

i) Do external advisors or observers attend meetings of this committee (e.g. members of the Auditor General's Office)?

Yes, please specify who_______________
No

22. Who approves the following documents?

In each space provided, identify one or more of the following:

SC = the senior-level committee described in question 21;

DC = a different committee;

MSO = most senior official (e.g. Deputy Minister) of the Institution

DS = direct supervisor as identified in question 8

HU = Head of Unit as identified in question 6

OTHER (please specify)

Terms of Reference for Individual Projects _________

Final Reports on Individual Projects ______________

Annual or Multi-Year Work Plan

Work Updates__

Strategic Plan __

Human Resource Plan____

23. Participation of Program Managers

Are managers of programs generally required or invited to provide input and commentary, beyond provision of data, on the findings and recommendations of a study regarding their area, before the report is finalized? (check all that apply)

Evaluations Required
Invited
Neither
N/A

Audits Required
Invited
Neither
N/A

Reviews Required
Invited
Neither
N/A

24. Management Responses:

a) Do you require management responses or action plans from managers before reports are passed on for approval?

Evaluations Always
Sometimes
Never
N/A

Audits Always
Sometimes
Never
N/A

Reviews Always
Sometimes
Never
N/A

b) Do you monitor the implementation of management responses or action plans?

Evaluations Always
Sometimes
Never
N/A

Audits Always
Sometimes
Never
N/A

Reviews Always
Sometimes
Never
N/A

If you answered "always" or "sometimes", answer question 24-c, if not, go to question 25.

c) How often do you follow up on responses and action plans?

Evaluations:

Every 3 months   
Every 6 months   
Yearly    
Other, please specify

____

Audits:       

Every 3 months   
Every 6 months   
Yearly    
Other, please specify

____

Reviews: 

Every 3 months  
Every 6 months   
Yearly   
Other, please specify

____

d) Do you report the results of this monitoring to the audit/ evaluation/ review committee or equivalent body?

Evaluations
Always
Sometimes
Never
N/A
Audits
Always
Sometimes
Never
N/A
Reviews
Always
Sometimes
Never
N/A

25. Quality Assurance:

a) Are there formal mechanisms in place for measuring the performance of your Unit?   

Evaluation
Yes
No
N/A
Audit
Yes
No
N/A
Review
Yes
No
N/A Management
Yes
No
N/A Assistance
Yes
No
N/A

If you answered " yes " to any part, answer question 25-b, if not, go to question 26.

b) Please indicate which of the following mechanisms are used:

a) Client survey Evaluation
Audit
Review
Management Assistance

b) Post-project interviews Evaluation
Audit
Review
Management Assistance

c) Employee feedback survey Evaluation
Audit
Review
Management Assistance

d) Internal Progress reports Evaluation
Audit
Review
Management Assistance

e)Project management systems Evaluation
Audit
Review
Management Assistance

f) Annual reports on unit's work Evaluation
Audit
Review
Management Assistance

g) Periodic independent assessments Evaluation
Audit
Review
Management Assistance

h) Other, please specify ____

26. Activities Outside Your Unit:

Do other units in your institution conduct projects which are essentially evaluation, audit or review?

Yes, frequently    
Yes, infrequently
Never

If you answered " yes", answer question 26-b, if not, go to question 27.

b) Is your Unit involved in these projects?

Evaluations
Always
Sometimes
Never
Audits
Always
Sometimes
Never
Reviews
Always
Sometimes
Never

If you answered "always" or "sometimes", answer question 26-c, if not, go to question 27.

c) Please explain the arrangements or circumstances under which your involvement takes place:

27. Workload:

How many projects were completed by your Unit in 1997-98? (Note: completed projects are defined as projects where final results were provided in some form to the client, who was then in a position to take action on the results.)

Type of Project	Number of Projects Completed
Evaluations
Audits
Reviews
Management Assistance Projects
Other, please specify

b) Is this representative of your Unit's regular yearly workload?

Yes
No, please explain

c) How many of the above projects consisted of unplanned special requests?

_________

28. Joint Projects

Do you work on joint projects with partners outside of your Unit? (This does not include consultants that you are paying to work on your projects).

Partners	Often	Occasionally	Rarely	Never
Other groups in the department
Other departments
Other governments
Non-governmental organizations
Industry
Other, please specify

29. Types of Activities, Projects or Services

The purpose of this question is to detect trends in offerings - what sort of work is on the rise, what is holding steady or waning. Accordingly, rough estimates are acceptable.

To what extent do the following represent part of your current (1997-98 and 1998-99) activities (please indicate a rough percentage)? Please also indicate (with a check mark) whether you consider them Core activities or Non-core activities, and indicate (with a check mark) what trend in demand do you foresee?

Type of Activity, Project or Service	Current (%) of total activity	Core	Non-Core	Trend:
				Increasing	Steady	Decreasing
Audits (Program Operations)
Program Evaluations
Evaluation frameworks
Evaluation of alternative service delivery
Performance measurement
Service standards
Implementation audit
Assessment of management processes and controls
Regulatory review/evaluation
Policy evaluation
Client/Stakeholder survey
Employee survey
Systems under development audit
Y2K audits
Other IT audits
Environmental audit
Control self-assessment
Total quality management
Special investigations
Other, please specify
Other, please specify
Other, please specify

 30. Liaison with the Office of the Auditor General (OAG):

a) Is your Unit responsible for liaison with the Office of the Auditor General in your
department or agency?

Yes 
No      
Not Applicable

If you answered " yes ", answer question 30-b, if not, go to question 31.

b) What types of activities does this responsibility entail?

_______________</ p>

c) How much staff time, including yours, do you estimate your Unit spends each year
in this liaison? _______ (number of work-months)

31. What techniques, tools or processes have been introduced over the past 5 years in order to improve your Unit's productivity and service quality? (for example, Total Quality Management or ISO Certification)

32. Through which sources does your Unit keep in touch with techniques and methodologies? Please rank the following, 1 being the most significant source, over the past 5 years.

Other departments                      _______

Treasury Board Secretariat         _______

Office of the Auditor General     _______

Internet                           _______

Review Network                     _______

Institute of Internal Auditors       _______

Canadian Evaluation Society      _______

Training (from other sources)      _______

Conferences (from other sources) _______

Other, please specify_____

Skills, Competencies and Experience

33. Approximately how much money and person-days are you devoting to training Unit

employees in 1998-99?

Type of Training	Money ($)	Person-Days
Audit Techniques
Evaluation Techniques
Review Techniques
Control Self-Assessment
Facilitation
Performance Measurement
Service Standards
Quality Management
Information Technology
ISO 9000/14000
Environmental Management Systems
Other, please specify
Other, please specify
Other, please specify

 34. What do you anticipate to be your Units` most significant training needs in the near future?

______

35. Referring to the staff in the Unit (including management, but excluding support staff), how many employees fall into the following:

Highest Education Level	Evaluation	Audit	Review	Multi-functional E/A/R
Secondary education completed
College Education
Bachelor's Degree
Master's Degree
Ph.D.
Other formal education, please specify

36. Number of staff (including management, but excluding support staff), in your Unit, with degrees in the following areas of expertise:

Degree Category	Evaluation	Audit	Review	Multi-functional E/A/R
Administration/ Management
Accounting
Economics
Arts (e.g. English Literature, History)
Law
Science
Social Science
Engineering
Computer Science
Other, please specify

37. Number of staff (including management, but excluding support staff), in your Unit with one or more designation:

Designation	Evaluation	Audit	Review	Multi-functional E/A/R
Chartered Accountant (CA)
Certified Management Accountant (CMA)
Certified General Accountant (CGA)
Certified Internal Auditor (CIA)
Certified Public Accountant (CPA)
Certified Management Consultant (CMC)
Certified Information Systems Auditor (CISA)
Professional Engineer (PE)
Other, please specify

38. What is your Unit's employment equity composition (including management and

administrative support staff and not including students and consultants on contract)?

Designated Group	Evaluation	Audit	Review	Multi-functional E/A/R	Support
Women
Francophone
Disabled
Visible Minority
Aboriginal

39. What is your Unit's language composition?

No. of staff (including management)	Evaluation	Audit	Review	Multi-functional E/A/R	Support
Unilingual English
Unilingual French
Bilingual - Oral "B"
Bilingual - Oral "C"
Bilingual Oral Exemption

INFORMATION TECHNOLOGY AND COMMUNICATION

40. Which of the following does your Unit use?

a) computer-based time reporting system            
Yes       
No
b) computer-based project management system  
Yes       
No
c) automated working papers                           
Yes         
No

41. What is your Unit's policy on publication of reports? _______________

42. What mechanisms do you use to make reports accessible to the public? (Check all that apply)

Publish and place in departmental library
Publish on your web site
Publish on departmental web site
Publish on Treasury Board web site
Other, please specify________

Comments

Please elaborate on any of your answers, referring to the related question(s) by number, or comment on the survey.

Thank you for taking the time to participate in this survey.

As a precaution, and to facilitate discussion if we need to call you for clarification, please keep a photocopy of your response.

Name of Contact: ____________

Telephone: ____________

Fax: ____________

Please return the questionnaire (using the envelope provided) to the following address before February 26, 1999:

Agora Management Associates

2030 Gatineau View Crescent

Gloucester, ON

k1j 7x1