Science Advisory Report  2015/039

Science Advice to Guide a Research Study using the Fluidigm® Biomark Platform for Microbe Detection in Wild and Farmed Salmon

Summary

• Over 90% of juvenile salmon migrating from freshwater into the ocean will die before returning to freshwater to spawn. While mortality is believed to be highest during the first few months in the marine environment, there is ongoing mortality due to many different reasons. Because the bulk of the current knowledge on salmon diseases comes from observations of cultured fish, the relative importance of infectious and non-infectious causes in wild migrating pacific salmon remains largely unknown.
• The Genome BC Strategic BC Salmon Health initiative (SSHI) is underway and its strategic role will be to detect the microbes and potential resultant diseases that may affect the productivity and performance of BC salmon, their evolutionary history, and the potential role of exchanges between wild and cultured salmon. This research project is highly multidisciplinary involving several research areas namely: Genomics, Epidemiology, Histopathology, Immunology, Virology, Parasitology, Salmon Ecology and Bioinformatics. The project will proceed through four sequential phases (stages).
• The SSHI project focuses principally on microbes that are recognized globally, and either known or suspected to cause disease in salmon (or related to opportunistic infections of immune compromised fish).  It utilizes genomic methods to identify and verify which microbes are present in wild and cultured (federal hatcheries and salmon farms) finfish in BC, Canada. Later phases of the project will develop challenge studies for those microbes that are agreed upon to have the greatest potential to negatively impact wild salmon.  These studies will allow us to better understand under what conditions, if any, these microbes cause disease.
• The scope of this CSAS process is on Phase 2a of this project with the stated major objective to evaluate the sensitivity, specificity and repeatability of assays designed to quantitatively assess the presence and load of microbes, in multiple samples simultaneously using a high throughput micro-fluidics platform (the Fluidigm® BioMark; hereafter BioMark).  This technology utilizes a specific targeted amplification (STA) step, the effects of which were assessed within this report.
• There are no agreed upon minimum standards for analytic performance characteristics as outlined by the World Organization for Animal Health (OIE).  Stage 1 of the OIE evaluation (diagnostic tests) is perhaps the closest analogue; however, the OIE has not explicitly provided guidelines for validation of multiplex assays (e.g. the STA) or new technologies (e.g. the BioMark platform). 
• In the present context, the intended purpose of the assays tested on the BioMark platform is for conducting research (which is not an OIE-listed purpose, but nor is it specifically excluded). Therefore, the conceptual testable hypothesis that can be statistically evaluated (null vs. alternate) within the context of this project is the estimation and comparison of microbe and pathogen prevalence, and thus probably the closest to the purposes listed by OIE.
• For CFIA-regulated or OIE-listed diseases, a procedure for reporting to the Canadian Food Inspection Agenda (CFIA) was discussed and agreed upon:
• The project team (lab) use OIE recommended assays, where available.
• If samples are suspected to contain a CFIA reportable disease agent, the project team (lab) will follow the CFIA mandatory notification directive for research laboratories for such events (Directive: Mandatory Notification of Reportable Aquatic Animal Diseases by Researchers).
• If samples are suspected to contain an OIE-listed disease agent that is not CFIA reportable (or immediately notifiable to CFIA), that CFIA be notified per requirements for a diagnostic lab.
• Positive results for reportable disease agents are required to be reported to the CFIA within 24 hours of detection, and such preliminary results are not to be reported outside of the project team.  CFIA has the regulatory responsibility to confirm the presence of all reportable diseases in Canada. 
• The project team (lab) will liaise with the CFIA to seek clarification if required.
• The SSHI project management team will only be notified following a CFIA investigation (if applicable).
• Per the Terms of Reference, this CSAS review assessed: the analytical sensitivity, specificity and repeatability of microbial assays on the BioMark platform; the comparability of assay results between the BioMark and ABI 7900 platforms; the effect(s) of the STA; and the benefits, limitations, uncertainties and proposed uses of this methodology.
• Assay sensitivity – For virtually all BioMark assays, a Limit of Detection (LOD) of samples undergoing an STA enrichment step was 1-10 copies per chamber, similar to the LOD of the starting material. The threshold cycle (Ct) cutpoint associated with the LOD was between 27 and 29.  Positive control samples were collected from as many reliable sources as possible and showed satisfactory results for most of the 47 assays; for two viruses, ISAV7 and IPNV, the assays did not detect all strain variants, consistent with results from other platforms and expected based on the degree of homology.  The authors of the review suggested that there may also be some strain variants of bacterial microbes not included among those tested that based on in silico analyses may not be detected. 
• Assay specificity – 13 viral and 12 bacterial assays showed high specificity against all closely-related species tested. Assays for 22 parasites were mostly specific but some distantly-related species appeared positive in certain tissue samples presumably because of co-infection by multiple parasite species. While the authors showed that for these pairs of microbes, co-detection was not the norm (consistent with a co-infection rather than cross-amplification scenario), the reviewers felt that nucleotide sequencing would provide better evidence for mixed infection.
• In all, there was no variance in assay sensitivity or specificity from expectations on other platforms, any variance from 100% analytical specificity (ASp) did not impede resolution of known salmon microbes (with one exception), and measures of analytical performance were within an accepted range for most assays.
• Assay repeatability – Twenty-six (26) endemic microbes were assessed for repeatability across 240 samples from BC salmon. Repeatability (within a dynamic array) and reproducibility (across dynamic arrays) were examined. Binary agreement (positive/negative) was assessed between replicates and found to be 98% overall. 
• Platform comparison – The study compared the performance of BioMark and ABI 7900 HT using the primer/probe concentrations recommended for the BioMark platform, and obtained comparable results for 22 available assays. These conditions were not individually optimized for conventional quantitative PCR platforms. 
• Effect of STA – The use of STA allows for enhanced sensitivity of detection of the target and is required because of the small assay volumes inherent in any microfluidics platform (7 nl for the 96.96 dynamic array).  This volume is approximately 1:1000th the volume of a traditional quantitative PCR (qPCR) platform. The cycle threshold cutpoint and LOD per chamber for STA and non-STA are the same, Ct 27-29 and ~1-10 copies, respectively, but a minimum of ~1,000 copies per µl are required for the starting material if the STA is not applied.  In essence the STA enriches targeted products by approximately 1000x. Other than the LOD effect on the starting material, no consistent biases were identified regarding the relative abundance of targets.  Moreover, no significant differences between STA and non-STA frequencies of false positives were observed across the assays, suggesting that the enrichment step did not impact sensitivity or specificity of the assays.  Target enrichment STA did not significantly affect repeatability and reproducibility, nor individual results (positive/negative). Overall, repeatability was good to excellent for most assays, and within ranges observed on other platforms.
• Strengths and weaknesses - A number of strengths were identified, including individual benefits with respect to: depth of coverage; identification of co-infections; cost per test; time-savings; efficiency; flexibility; expandability; refinement of purpose; analytical sensitivity; interchangeability; limited tissue requirements; upgrading to facilitate PCR-sequencing; simultaneous RNA quality assessments; detection of a second probe; and use of non-lethal samples.  Conversely, a number of weaknesses were also identified, including: poor curve quality for some assays; enhanced training requirements; high initial cost of instrument; volume of data and level of analysis required (“big data”); need to improve software algorithms; instrument better suited for large studies and datasets; and cautions for interpretation of results.  See Assessment section for a more comprehensive discussion on these points.
• Uncertainty – a number of sources of uncertainty were identified in the analytical approach, including: the use of gBlocks for closely-related species; the use of tissue samples as positive controls which may enhance possibility of detection of co-infections when multiple microbes are assessed at once; the grouping of control samples (by viruses, bacteria and parasites) for specificity studies; and the use of a single serial dilution (did not assess technical variation in pipetting accuracy).  These limitations are more fully discussed in the Assessment section, along with associated recommendations.
• The results of this peer review indicated that this research project can proceed to Phase 2b. However, the Science advice includes a number of recommended improvements to the experimental design, lab procedures and the associated working paper (Research Document) which must be addressed.  Further details are provided in the Assessment section of this report, but these recommendations include:
• Dozens of replicates of the same assays were performed to obtain the limit of detection (LOD) and assay linearity for each primer/probe design, however, this was done using a single serial dilution of artificial positive controls (APC). The authors successfully demonstrated the accuracy of measurement within a single sample for each point, including the impact of six independent STA reactions, however the evaluation of possible variations in liquid handling (during the preparation of serial dilution of APC) was not included.  To more fully address the inherent variability, the evaluation of inter-control variability using multiple independently prepared serial dilutions is recommended.
• All positive detections obtained against positive control samples other than the one(s) targeted should be subjected to DNA sequencing for confirmation. For sequencing, it would be best to target another gene (classical or nested PCR), which will distinguish potential cross-reactions or co-infections. Similar sequencing should be conducted on a subset of tested field samples to confirm/validate specificity associated with microbe monitoring studies.
• To ensure the same detection efficiency (i.e. same Ct values), a comparison should be conducted between both platforms using STA samples, under the prescribed conditions for both platforms (e.g., a temperature of hybridization of 60°C for all primers and probes). This comparison would ideally be conducted using real positive samples (i.e. without gBlocks), where possible.
• A framework to guide the interpretation of research findings is strongly recommended (e.g. scoring methodology).  Consideration should also be given to the broader implications of potential findings and how this will be communicated both internally (within the Department) and more broadly externally (the public).  Both regulators and policy makers would benefit greatly from a consistent framework for the interpretation of project results. 
• To account for the uncertainty around some assay processes, it is recommended that a quality management system be implemented, which would provide methods and process checks, verification and calibration of equipment and instruments, critical reagents, and the introduction of new assays.  In addition, each aspect of the quality assurance/quality control (QA/QC) process should consider and mitigate the risks identified (e.g. via standard operating procedures (SOPs) or a quality control manual).
• A verification of the effectiveness of the ‘clean-up step’ after the STA is recommended.
• The verification of specificity across microbe groupings (viruses, bacteria, and parasites) using real samples and excluding tissues and gBlocks across all assays is recommended.
• It is recommended that the efficiencies of the calculated Cts be tested for robustness.  This can be examined by changing the efficiencies (manually in formulas) and verifying the effect on the calculated Cts.

This Science Advisory Report is from the December 2-4, 2014 national advisory process on the Review of the Fluidigm BioMark platform: Evaluation to assess fitness for purpose in microbial monitoring. The accompanying Research document and Proceedings from this meeting will be posted on the Fisheries and Oceans Canada (DFO) Science Advisory Schedule as they become available.

Complete document (PDF, 240 KB)

Accessibility Notice

This document is available in PDF format. If the document is not accessible to you, please contact the Secretariat to obtain another appropriate format, such as regular print, large print, Braille or audio version.