Survey Methodology and Data Quality

The diversity of recreational fishing licensing procedures across Canada required the use of several approaches in order to conduct a national survey across the various jurisdictions.

This section aims to describe the procedures used in the 2010 survey as well as the strengths and limitations of the data in order that they could be effectively used and analysed. Information on the methodology used and the data quality of survey results would of particular importance when making comparisons with data from other surveys and sources of information or when drawing conclusions regarding changes over time.

5.1 Sampling Procedures

5.1.1 General

Resident and non-resident samples were selected using stratified, systematic random sampling of licence databases with stratification based on licence category. The only exceptions were in the case of sampling the resident anglers in two jurisdictions: Quebec and Newfoundland and Labrador (Sections 5.1.2 and 5.1.3).

In British Columbia (freshwater) and in the Northwest Territories, the licence strata were further defined by the area in which the licences were sold. In Saskatchewan, Nova Scotia and Ontario, resident samples were further stratified by area of residence.

The sample sizes were determined from the reliability estimates for days fished for each stratum as calculated from the results of the 2005 Survey of Recreational Fishing in Canada.

5.1.2 Resident Anglers in Quebec and Newfoundland and Labrador

Due to limitations in the licence databases available for residents of Quebec and Newfoundland and Labrador, sampling for the populations of resident anglers was conducted by using household samples of the general population. The households in each province were geographically stratified. The required sample of angling households was determined in consultation with representatives of each province based on the distribution of households in the 2006 census.

The overall sample size was determined using the proportion of angling households to all households in each geographic area as estimated for 2005 combined with reliability of days fished for each area in 2005.

Households were pre-screened by telephone to establish the eligibility for inclusion in the survey. Profiles were obtained by determining for each selected household the age (under/over base age for adult anglers in the province) and sex of all individuals residing in the household. It was then further requested for those individuals identified as adults, how many in the household fished for recreation in 2010. In households where anglers were identified, one was selected at random to respond to the questionnaire. For verification purposes, the selected respondent was asked for the number of days fished in 2010.

The actual pre-screening operations were handled by private research firms under joint contract with DFO and the respective provincial governments. The results of the pre-screening were provided to DFO for population estimation and survey weighting procedures.

5.1.3 Non-resident Anglers in Quebec

Historically, the Quebec licence database was used to sample the non-resident anglers in the province. However, with the implementation of the province's Act Respecting Access to Documents Held by Public Bodies and the Protection of Personal Information, non-resident anglers are not required to provide complete personal and contact information when obtaining their recreational fishing licence.Footnote 13 Therefore, as in 2005, it was not possible to survey the non-resident angling population in Quebec for 2010 due to the absence of a survey frame (i.e., a list containing all necessary information from which the survey sample can be drawn).

The number of active non-resident anglers in Quebec was estimated at 46,579 in 2000. Based on the observed trend in non-resident licence sales in Quebec since 2000, falling at an average annual rate of 2%, it would be reasonable to expect that the number of active non-resident anglers in Quebec would also have been on a similar downward trend in 2010. There was no attempt, however, to impute the non-surveyed portion of the Quebec angling population in this report due to methodological constraints.

5.1.4 Special Sampling: Atlantic Salmon Licence Holders

In Newfoundland and Labrador, a special sample of resident Atlantic salmon licence holders was selected to ensure adequate sampling for special analyses of these important sub-populations required for an Atlantic-wide analysis of Atlantic salmon angling. A similar effort was made in Quebec, but with supplementary resident samples of both Atlantic salmon licence holders and general licence-holders to augment the results found in the household pre-screening phase of the survey. These licences were subsequently allocated for weighting based on the administrative region of residence as found in the pre-screening survey.

The decision to select special resident samples to augment the pre-screening phase was made because results of earlier surveys had shown that there was inadequate information for certain licence types based solely on the pre-screening results. Augmenting the samples allowed for the detailed analyses required by both provinces.

5.2 Data Collection, Processing, Coding and Estimation

The survey was launched in January 2011 through a coordinated press release led by DFO Headquarters (Ottawa). Each jurisdiction was responsible for survey mail out and coding of survey responses.

Due to variations across questionnaires, and specialized identification requirements, all coding was provided on the documents by each participating jurisdiction before being sent to Ottawa for processing. In some cases, all that was required after assessing completeness of the questionnaire was identification coding on each document. In others, far more involved coding was required ranging from special codes for sub-components of questions to geographic codes in order to allow for sub-provincial/territorial data analysis.

A major requirement in the editing and coding phase was verification that the information provided by anglers was both reasonable and sound for the jurisdiction involved. This included assessing species caught by area as well as the availability of the specific species, determining whether or not the catch levels indicated were within acceptable limits, cross-checking administrative lists to eliminate substitutions (someone responding other than the designated respondent), etc. Once questionnaires had been completely assessed and coded, they were sent to Ottawa for analysis.

Working under contract to DFO Headquarters, BriLev Consulting Inc. coordinated the questionnaire imaging, data capture and validation of survey responses in cooperation with the coordinators for each jurisdiction. This was done on a staggered basis, depending on the province or territory. All documents were verified prior to data capture, and those with significant levels of incomplete coding, missing information, etc. were set aside. Data capture procedures were written in-house for each jurisdiction. All data collection, coding, and data capture were completed in the Fall of 2011.

The required programming to produce survey estimates consisted of:

  • systematic editing procedures;
  • substitution algorithms for missing data;
  • data output programs covering all data collected; and
  • special output programs for publication purposes.

5.3 Weighting Procedures

Weighted estimates were produced to ensure that the raw survey data reflect the estimates for the population. This procedure for most jurisdictions where licence sales were known was a simple application of inverse weighting by stratum (population of licence-holders divided by resultant sample). In all jurisdictions where additional stratification was done based on geographic information, adjustments were made on the basis of both licence-holding and geographic stratification. In jurisdictions where anglers could hold different types of licence, either by choice or due to regulations, adjustments to weighting were made to ensure that there was no double counting of anglers across licence types.

For residents of Quebec and Newfoundland a more complex procedure was required. First, it was necessary to estimate the total population by age-group and sex in each stratum based on the pre-screening data. Household weights, determined from the estimated households in each stratum divided by the sample of households pre-screened, were used to generate estimates of population. Of primary interest were the estimated populations of adult anglers by sex.

The definition of adult varied with Quebec using 15 years of age and, Newfoundland and Labrador, 18. Using the pre-screening data, it was possible to determine the proportion of anglers in each geographic area of each province.

The estimates of population in each stratum of interest were then compared to official statistics on population available from Statistics Canada. Population data was based on the 2006 census with post-censal estimates as at July 1, 2010. The initial estimates of population and the respective estimates of anglers in each stratum were adjusted to reflect these statistics. Respondent weights for survey estimates were then derived using standard inverse weighting functions.

5.4 Post-Survey Adjustment Procedures

Preliminary survey results were distributed to various jurisdictions from late 2011 to early 2012. All jurisdictions were provided with the opportunity to review the preliminary estimates.

The province of Ontario conducted a non-response follow-up survey in mid-2011. The results of this survey were used to adjust the estimates based on the results of the mail survey. The adjustment procedure resulted in an overall reduction of 17% in activity information and, for expenditure information, 6%.

In all other jurisdictions, adjustments were made provided that they were based on identifiable errors or changes required due to revised administrative information. The area of primary concern to most jurisdictions was catch and retention of specific species by geographic area.

Most errors were the result of incorrect identification of species and the changes were relatively straightforward. Revisions were completed by February 2012 and the survey results were finalized.

5.5 Data Quality

5.5.1 Sampling Error

Information on reliability is presented in terms of the coefficient of variation of the mean, at one standard deviation. Because exact estimates of variance cannot be calculated, the coefficient of variation of the mean is used as a proxy for the assessing the variability of the data. Annex A.13 presents the statistical reliability of key variables for all anglers.

The ranges of coefficient of variation (c.v.) below provide guidelines as to the use of the data:

Coefficient of variation Guideline
less than 16.5% Data can be used without condition
16.5% to 33.5% Data should be used with caution
greater than 33.5% Data should be used with extreme caution

Source: Statistics Canada.

5.5.2 Non-Sampling Error

Many factors that are not related to sampling also affect the reliability of the data produced in the survey. For example, respondents may have made errors in interpreting questions, the answers may have been incorrectly entered on the questionnaires or errors may have been introduced during the data capture or tabulation process.

During the data collection phase, efforts were made to reduce the occurrence of non-sampling errors in the survey.Footnote 14 These efforts included a complete verification of the reported data, validity and consistency edits and ongoing consultations with the coordinators in each jurisdiction.

5.5.3 Data Limitations

Despite all efforts to improve data accuracy, the survey results have limitations. Understanding these limitations will help the reader make informed decisions before conducting further research and analysis using the estimates in this report:

  1. Estimates of recreational saltwater fisheries in Quebec and the Atlantic provinces: Anglers are not required to hold a licence to fish in saltwater for most species. The survey results (harvests, days fished, etc.) pertaining to recreational saltwater fisheries in Quebec and Newfoundland and Labrador are based on the pre-screening estimates of angler populations who fished in 2010. In the other provinces, estimates of recreational saltwater fishing activities are based only on anglers who held a freshwater fishing licence.
  2. Quebec non-resident estimates: there are no survey estimates on recreational fishing activities of anglers who fished in Quebec but lived outside the province in 2010 (see Section 5.1.3 for details).
  3. The survey did not cover nor did it attempt to measure the illegal, unreported activities related to recreational fisheries in Canada, for example, the days fished and harvests of individuals who conducted recreational fishing activities without a licence where/when a licence or permit is required.

5.6 Data comparability

The 2010 detailed provincial and territorial results are directly comparable with the 2005 surveys, with one main exception. While most of the core survey questions remained unchanged throughout these survey cycles, investment information was refined in 2010 to include estimates for purchase of new or used boating equipment and special vehicles. It is assumed that most of these purchases identified in earlier years focussed on new purchases, however, we cannot say so with any degree of certainty.

The issue of potentially obtaining information on the non-resident anglers in Quebec through alternate sources will be addressed in future surveys.