Survey Methodology and Data Quality

The diversity of recreational fishing licensing procedures across Canada required the use of several approaches in order to conduct a national survey across the various jurisdictions.
 
This section aims to illustrate the procedures used in the 2005 survey as well as the strengths and limitations of the data in order that they could be effectively used and analysed.  Information on the methodology used and the data quality of survey results would of particular importance when making comparisons with data from other surveys and sources of information or when drawing conclusions regarding changes over time.

5.1 Sampling Procedures

5.1.1 General

In each jurisdiction, two types of survey questionnaires were mailed: one covering residents and another covering non-residents. ^{Footnote 19} Resident and non-resident samples were selected using stratified, systematic random sampling of licence bases with stratification based on licence category.  The only exceptions were in the case of sampling the resident anglers in two jurisdictions: Quebec and Newfoundland and Labrador (Sections 6.1.2 and 6.1.3).

In British Columbia (freshwater) and in the Northwest Territories, the licence strata were further defined by the area in which the licences were sold. In Nova Scotia and Ontario, resident samples were further stratified by area of residence. 

The sample sizes were determined from the reliability estimates for days fished for each stratum as calculated from the results of the 2000 Survey of Recreational Fishing in Canada.

5.1.2 Resident Anglers in Quebec and Newfoundland and Labrador

Due to limitations in the licence databases available for residents of Quebec and Newfoundland and Labrador, sampling for the populations of resident anglers was conducted by using household samples of the general population. The households in each province were geographically stratified. The required sample of angling households was determined in consultation with representatives of each province based on the distribution of angling households in the 2001 census.

The overall sample size was determined using the proportion of angling households to all households in each geographic area as estimated for 2000 combined with reliability of days fished for each area in 2000.

Households were pre-screened by telephone to establish the eligibility for inclusion in the survey. Profiles were obtained by determining for each selected household the age (under/over base age for adult anglers in the province) and sex of all individuals residing in the household. It was then further requested for those individuals identified as adults, how many in the household fished for recreation in 2005. In households where anglers were identified, one was selected at random to respond to the questionnaire. For verification purposes, the selected respondent was asked for the number of days fished in 2005.

The actual pre-screening operations were handled by private research firms under joint contract with DFO and the respective provincial governments. The results of the pre-screening were provided to the responsible agency and, subsequently to DFO for weighting procedures. Mailing labels for selected respondents to the survey were also provided for the fieldwork phase.

5.1.3 Non-resident Anglers in Quebec

Historically, the Quebec licence database has been used to sample the non-resident anglers in the province. However, with the implementation of the province's Act Respecting Access to Documents Held by Public Bodies and the Protection of Personal Information, non-resident anglers are no longer required to provide complete personal and contact information when obtaining their recreational fishing licence. ^{Footnote 20}

Therefore, in 2005, it was not possible to survey the non-resident angling population in Quebec due to the absence of a survey frame (i.e., a list containing all necessary information from which the survey sample can be drawn).

The number of active non-resident anglers in Quebec was estimated at 50,806 in 1995 and 46,579 in 2000.  Based on the observed trend in most provinces and territories throughout the years, it would be reasonable to expect that  the number of active non-resident anglers in Quebec would also have been on a downward trend in 2005. There was no attempt, however, to impute the non-surveyed portion of the Quebec angling population in this report due to methodological constraints.  

5.1.4 Special Sampling: Atlantic Salmon Licence Holders

In Newfoundland and Labrador, a special sample of resident Atlantic salmon licence holders was selected to ensure adequate sampling for special analyses of these important sub-populations required  for an Atlantic-wide analysis of Atlantic salmon angling. A similar effort was made in Quebec, but with special resident samples of both Atlantic salmon licence holders and general licence-holders to augment the results found in the household pre-screening phase of the survey. These licences were subsequently allocated for weighting based on the administrative region of residence.

The decision to select special resident samples to augment the pre-screening phase was made because results of earlier surveys had shown that there was inadequate information for certain licence types based solely on the pre-screening results. Augmenting the samples allowed for the detailed analyses required by both provinces.

5.2 Data Collection, Processing, Coding and Estimation

The survey was launched in January 2006 through a coordinated press release led by DFO Headquarters (Ottawa). Each jurisdiction was responsible for survey mailout and coding of survey responses.  

A major requirement in the editing and coding phase was a verification that the information provided by anglers was both reasonable and sound for the jurisdiction involved. This included assessing species caught by area as well as the availability of the specific species, determining whether or not the catch levels indicated were within acceptable limits, cross-checking administrative lists to eliminate substitutions (someone responding other than the designated respondent), etc. Once questionnaires had been completely assessed and coded, they were shipped to Ottawa for analysis.  

DFO Headquarters coordinated the questionnaire imaging, data capture and validation of survey responses in cooperation with the coordinators for each jurisdiction. This was done on a staggered basis, depending on the province or territory. All documents were verified prior to data capture, and those with significant levels of incomplete coding, missing information, etc. were set aside. Data capture procedures were written in-house for each jurisdiction.  All data collection, coding, and data capture were completed in Fall 2006.

The required programming to produce survey estimates consisted of:

• systematic editing procedures;
• substitution algorithms for missing data;
• data output programs covering all data collected; and
• special output programs for publication purposes.

5.3 Weighting Procedures

Weighted estimates were produced to ensure that the raw survey data reflect the estimates for the population. This procedure for most jurisdictions where licence sales were known was a simple application of inverse weighting by stratum (population of licence-holders divided by resultant sample). In all jurisdictions where additional stratification was done based on geographic information, adjustments were made on the basis of both licence-holding and geographic stratification. In jurisdictions where anglers could hold different types of licence, either by choice or due to regulations, adjustments to weighting were made to ensure that there was no double counting of anglers across licence types.

For residents of Quebec and Newfoundland a more complex procedure was required. First, it was necessary to estimate the total population by age-group and sex in each stratum based on the pre-screening data. Household weights, determined from the estimated households in each stratum divided by the sample of households pre-screened, were used to generate estimates of population. Of primary interest were the estimated populations of adult anglers by sex. The definition of adult varied with Quebec using 15 years of age and, Newfoundland, 18. Using the pre-screening data, it was possible to determine the proportion of anglers in each geographic area of each province.

The estimates of population in each stratum of interest were then compared to official statistics on population available from Statistics Canada. Population data was based on the 2001 census with post-censal estimates as at July 1, 2005. The initial estimates of population and the respective estimates of anglers in each stratum were adjusted to reflect these statistics. Respondent weights for survey estimates were then derived using standard inverse weighting functions.

5.4 Post-Survey Adjustment Procedures

Preliminary survey results were distributed to various jurisdictions from late 2006 to early 2007. All jurisdictions were provided with the opportunity to review the preliminary estimates.

Adjustments were made provided that they were based on identifiable errors or changes required due to revised administrative information.  The area of primary concern to most jurisdictions was catch and retention of specific species by geographic area.

Most errors a result of incorrect identification of species and the changes were relatively straightforward. Another area where significant adjustments were required was refinement of population estimates. Changes were made based on updated information on licence sales, however, in the two provinces where population samples had been used, a considerable number of refinements were requested to ensure that the estimated number of anglers was accurately reflected.

Revisions were completed by Spring 2007 and survey results were finalized.

5.5 Data Quality

5.5.1 Sampling Error

Information on reliability is presented in terms of the coefficient of variation of the mean, at one standard deviation. Because exact estimates of variance cannot be calculated, the coefficient of variation of the mean is used as a proxy for the assessing the variability of the data.  Annex A.13 presents the statistical reliability of key variables for all anglers.

Source: Statistics Canada.

5.5.2 Non-Sampling Error

Many factors that are not related to sampling also affect the reliability of the data produced in the survey.  For example, respondents may have made errors in interpreting questions, the answers may have been incorrectly entered on the questionnaires or errors may have been introduced during the data capture or tabulation process.

During the data collection phase, efforts were made to reduce the occurrence of non-sampling errors in the survey. ^{Footnote 21}  These efforts included a complete verification of the reported data, validity and consistency edits and ongoing consultations with the coordinators in each jurisdiction.

5.5.3 Data Limitations

Despite all efforts to improve data accuracy, the survey results have limitations.  Understanding these limitations will help the reader make informed decisions before conducting further research and analysis using the estimates in this report:

1. Estimates of recreational saltwater fisheries in Quebec: Quebec residents were not required to purchase a saltwater fishing licence in 2005. Therefore, the survey results (harvests, days fished, etc.) pertaining to recreational saltwater fisheries in Quebec covered only the recreational fishing activities of Quebec residents who also fished in saltwater during the 2005 fishing season.
2. Quebec non-resident estimates: there are no survey estimates on recreational fishing activities of anglers who fished in Quebec but lived outside the province in 2005 (see Section 6.8 for details).
3. The survey did not cover not did it attempt to measure the illegal, unreported activities related to recreational fisheries in Canada, for example, the days fished and harvests of individuals who conducted recreational fishing activities without a licence where/when a licence or permit is required.

5.6 Data comparability

The 2005 detailed provincial and territorial results are directly comparable with the 1995 and 2000 surveys.  The core survey questions remained unchanged throughout these survey cycles. 

The only exception would be the absence of survey data for Quebec non-residents. This ultimately has an impact on the Canadian estimates. However, since the majority of active anglers in Quebec has always been the province's resident anglers, the undercoverage in 2005 is expected to have minimal impact in overall comparability of survey results. 

The issue of potentially obtaining information on the non-resident anglers in Quebec through alternate sources will be addressed in future surveys.