Critical assessment of automated flow cytometry data analysis techniques

Nima Aghaeepour, Greg Finak, FlowCAP Consortium, DREAM Consortium, Holger Hoos, Tim R Mosmann, Ryan Brinkman, Raphael Gottardo, Richard H Scheuermann, David Dougall, Alireza Hadj Khodabakhshi, Phillip Mah, Gerlinde Obermoser, Josef Spidlen, Ian Taylor, Sherry A Wuensch, Jonathan Bramson, Connie Eaves, Andrew P Weng, Edgardo S Fortuno 3rd, Kevin Ho, Tobias R Kollmann, Wade Rogers, Stephen De Rosa, Bakul Dalal, Ariful Azad, Alex Pothen, Aaron Brandes, Hannes Bretschneider, Robert Bruggner, Rachel Finck, Robin Jia, Noah Zimmerman, Michael Linderman, David Dill, Gary Nolan, Cliburn Chan, Faysal El Khettabi, Kieran O'Neill, Maria Chikina, Yongchao Ge, Stuart Sealfon, István Sugár, Arvind Gupta, Parisa Shooshtari, Habil Zare, Philip L De Jager, Mike Jiang, Jens Keilwagen, Jose M Maisog, George Luta, Andrea A Barbo, Peter Májek, Jozef Vilček, Tapio Manninen, Heikki Huttunen, Pekka Ruusuvuori, Matti Nykter, Geoffrey J McLachlan, Kui Wang, Iftekhar Naim, Gaurav Sharma, Radina Nikolic, Saumyadipta Pyne, Yu Qian, Peng Qiu, John Quinn, Andrew Roth, Pablo Meyer, Gustavo Stolovitzky, Julio Saez-Rodriguez, Raquel Norel, Madhuchhanda Bhattacharjee, Michael Biehl, Philipp Bucher, Kerstin Bunte, Barbara Di Camillo, Francesco Sambo, Tiziana Sanavia, Emanuele Trifoglio, Gianna Toffolo, Slavica Dimitrieva, Rene Dreos, Giovanna Ambrosini, Jan Grau, Ivo Grosse, Stefan Posch, Nicolas Guex, Jens Keilwagen, Miron Kursa, Witold Rudnicki, Bo Liu, Mark Maienschein-Cline, Tapio Manninen, Heikki Huttunen, Pekka Ruusuvuori, Matti Nykter, Petra Schneider, Michael Seifert, Jose M G Vilar, Nima Aghaeepour, Greg Finak, FlowCAP Consortium, DREAM Consortium, Holger Hoos, Tim R Mosmann, Ryan Brinkman, Raphael Gottardo, Richard H Scheuermann, David Dougall, Alireza Hadj Khodabakhshi, Phillip Mah, Gerlinde Obermoser, Josef Spidlen, Ian Taylor, Sherry A Wuensch, Jonathan Bramson, Connie Eaves, Andrew P Weng, Edgardo S Fortuno 3rd, Kevin Ho, Tobias R Kollmann, Wade Rogers, Stephen De Rosa, Bakul Dalal, Ariful Azad, Alex Pothen, Aaron Brandes, Hannes Bretschneider, Robert Bruggner, Rachel Finck, Robin Jia, Noah Zimmerman, Michael Linderman, David Dill, Gary Nolan, Cliburn Chan, Faysal El Khettabi, Kieran O'Neill, Maria Chikina, Yongchao Ge, Stuart Sealfon, István Sugár, Arvind Gupta, Parisa Shooshtari, Habil Zare, Philip L De Jager, Mike Jiang, Jens Keilwagen, Jose M Maisog, George Luta, Andrea A Barbo, Peter Májek, Jozef Vilček, Tapio Manninen, Heikki Huttunen, Pekka Ruusuvuori, Matti Nykter, Geoffrey J McLachlan, Kui Wang, Iftekhar Naim, Gaurav Sharma, Radina Nikolic, Saumyadipta Pyne, Yu Qian, Peng Qiu, John Quinn, Andrew Roth, Pablo Meyer, Gustavo Stolovitzky, Julio Saez-Rodriguez, Raquel Norel, Madhuchhanda Bhattacharjee, Michael Biehl, Philipp Bucher, Kerstin Bunte, Barbara Di Camillo, Francesco Sambo, Tiziana Sanavia, Emanuele Trifoglio, Gianna Toffolo, Slavica Dimitrieva, Rene Dreos, Giovanna Ambrosini, Jan Grau, Ivo Grosse, Stefan Posch, Nicolas Guex, Jens Keilwagen, Miron Kursa, Witold Rudnicki, Bo Liu, Mark Maienschein-Cline, Tapio Manninen, Heikki Huttunen, Pekka Ruusuvuori, Matti Nykter, Petra Schneider, Michael Seifert, Jose M G Vilar

Abstract

Traditional methods for flow cytometry (FCM) data processing rely on subjective manual gating. Recently, several groups have developed computational methods for identifying cell populations in multidimensional FCM data. The Flow Cytometry: Critical Assessment of Population Identification Methods (FlowCAP) challenges were established to compare the performance of these methods on two tasks: (i) mammalian cell population identification, to determine whether automated algorithms can reproduce expert manual gating and (ii) sample classification, to determine whether analysis pipelines can identify characteristics that correlate with external variables (such as clinical outcome). This analysis presents the results of the first FlowCAP challenges. Several methods performed well as compared to manual gating or external variables using statistical performance measures, which suggests that automated methods have reached a sufficient level of maturity and accuracy for reliable use in FCM data analysis.

Conflict of interest statement

At the time of this study, J.Q. was an employee of Tree Star Inc., and P. Májek and J.V. were employees of ADINIS s.r.o. and consultants for Cytobank Inc., which make commercial FCM analysis software. G.N. is a consultant, equity holder and member of a scientific advisory board and/or board of directors at Nodality, DVS Sciences, Beckton Dickinson, Cell Signaling Technologies, BINA Technologies, and 5AM Ventures.

Figures

Figure 1. F -measure results of cell…
Figure 1. F-measure results of cell population identification challenges.
Average manual and algorithm F-measures are represented against the manual consensus cluster as a function of the number of populations included, ranked from most consistent to least consistent. For a given population, consistency was defined as the agreement among manual gates, calculated as the average manual F-measures against the manual consensus cluster for that population. All populations across all samples were included in this calculation, and, as such, the numbers on the x axis should be multiplied by 12 and 30 (for GvHD and HSCT, respectively) to reflect the total number of populations in all samples in the reference. Individual manual gating results are plotted as gray lines. (a) Graft-versus-host disease (GvHD) data set. (b) Hematopoietic stem cell transplant (HSCT) data set.
Figure 2. Per-population pairwise comparisons of the…
Figure 2. Per-population pairwise comparisons of the cell population identification challenges.
Average F-measures of all pairs of results for the five cell populations across all samples in the hematopoietic stem cell transplant (HSCT) data set are represented as heat maps. The heat-map color in individual squares reflects the pairwise agreement between each method for each cell population independently, and the position in the matrix reflects the pattern of agreement across all methods on the basis of hierarchical clustering. The manual-gate consensus cluster for each sample was used as a reference for matching of the automated results of that sample. Pairwise F-measures between all algorithms and manual gates for the HSCT data set are shown. The dendrogram groups the algorithms and manual gates on the basis of the similarities between their pairwise F-measures. EC, ensemble clustering.
Figure 3. Comparison of manual-gate consensus and…
Figure 3. Comparison of manual-gate consensus and ensemble clustering results.
Dots are color-coded by population membership as determined by ensemble clustering, with donor-derived (CD45.2+) granulocytes/monocytes in green and donor-derived lymphocytes in red. Colored polygons enclose regions corresponding to the consensus clustering of manual gates. Fluorochromes used: FITC, fluorescein isothiocyanate; PE, phycoerythrin; APC, allophycocyanin. (a,b) Sample for which all of the cell populations have been accurately identified. (c,d) Sample in which the tail of the blue population has been misclassified as orange by the algorithms, resulting in a lower F-measure for the blue population. The red, blue, green, purple and orange cell populations match cell population 1–5 of Figure 2, respectively.
Figure 4. Acute myeloid leukemia (AML) subject…
Figure 4. Acute myeloid leukemia (AML) subject detected as an outlier by the algorithms.
(a) Total number of misclassifications for each sample in the test set (sample nos. 180–359) of the AML data set. (b–g) Forward scatter (FSC)/side scatter (b–d) and FSC/CD34 (e–g) plots of representative normal (b,e) and AML (c,f) samples and the outlier sample no. 340 (d,g), with the CD34+ cells highlighted in red. Cell proportions of the CD34+ population are reported as blast frequency (freq.) percentages.

References

    1. Baumgarth N, Roederer M. A practical approach to multicolor flow cytometry for immunophenotyping. J. Immunol. Methods. 2000;243:77–97. doi: 10.1016/S0022-1759(00)00229-5.
    1. Tanner SD, et al. Flow cytometer with mass spectrometer detection for massively multiplexed single-cell biomarker assay. Pure Appl. Chem. 2008;80:2627–2641. doi: 10.1351/pac200880122627.
    1. Bendall SC, Nolan GP, Roederer M, Chattopadhyay PK. A deep profiler′s guide to cytometry. Trends Immunol. 2012;33:323–332. doi: 10.1016/j.it.2012.02.010.
    1. Newell EW, Sigal N, Bendall SC, Nolan GP, Davis MM. Cytometry by time-of-flight shows combinatorial cytokine expression and virus-specific cell niches within a continuum of CD8+ T cell phenotypes. Immunity. 2012;36:142–152. doi: 10.1016/j.immuni.2012.01.002.
    1. Lugli E, Roederer M, Cossarizza A. Data analysis in flow cytometry: the future just started. Cytometry A. 2010;77:705–713. doi: 10.1002/cyto.a.20901.
    1. Quinn J, et al. A statistical pattern recognition approach for determining cellular viability and lineage phenotype in cultured cells and murine bone marrow. Cytometry A. 2007;71:612–624. doi: 10.1002/cyto.a.20416.
    1. Lo K, Brinkman RR, Gottardo R. Automated gating of flow cytometry data via robust model-based clustering. Cytometry A. 2008;73:321–332. doi: 10.1002/cyto.a.20531.
    1. Finak G, Bashashati A, Brinkman R, Gottardo R. Merging mixture components for cell population identification in flow cytometry. Adv. Bioinformatics. 2009;2009:247646. doi: 10.1155/2009/247646.
    1. Pyne S, et al. Automated high-dimensional flow cytometric data analysis. Proc. Natl. Acad. Sci. USA. 2009;106:8519–8524. doi: 10.1073/pnas.0903028106.
    1. Naumann U, Luta G, Wand MP. The curvHDR method for gating flow cytometry samples. BMC Bioinformatics. 2010;11:44. doi: 10.1186/1471-2105-11-44.
    1. Suchard MA, et al. Understanding GPU programming for statistical computation: studies in massively parallel massive mixtures. J. Comput. Graph. Stat. 2010;19:419–438. doi: 10.1198/jcgs.2010.10016.
    1. Zare H, Shooshtari P, Gupta A, Brinkman RR. Data reduction for spectral clustering to analyze high throughput flow cytometry data. BMC Bioinformatics. 2010;11:403. doi: 10.1186/1471-2105-11-403.
    1. Qian Y, et al. Elucidation of seventeen human peripheral blood B-cell subsets and quantification of the tetanus response using a density-based method for the automated identification of cell populations in multidimensional flow cytometry data. Cytometry B Clin. Cytom. 2010;78(suppl. 1):S69–S82. doi: 10.1002/cyto.b.20554.
    1. Sugár IP, Sealfon SC. Misty Mountain clustering: application to fast unsupervised flow cytometry gating. BMC Bioinformatics. 2010;11:502. doi: 10.1186/1471-2105-11-502.
    1. Aghaeepour N, Nikolic R, Hoos HH, Brinkman RR. Rapid cell population identification in flow cytometry data. Cytometry A. 2011;79:6–13. doi: 10.1002/cyto.a.21007.
    1. Ge Y, Sealfon SC. flowPeaks: a fast unsupervised clustering for flow cytometry data via K-means and density peak finding. Bioinformatics. 2012;28:2052–2058. doi: 10.1093/bioinformatics/bts300.
    1. Aghaeepour N, et al. Early immunologic correlates of HIV protection can be identified from computational analysis of complex multivariate T-cell flow cytometry assays. Bioinformatics. 2012;28:1009–1016. doi: 10.1093/bioinformatics/bts082.
    1. Zare H, et al. Automated analysis of multidimensional flow cytometry data improves diagnostic accuracy between mantle cell lymphoma and small lymphocytic lymphoma. Am. J. Clin. Pathol. 2012;137:75–85. doi: 10.1309/AJCPMMLQ67YOMGEW.
    1. Costa ES, et al. Automated pattern-guided principal component analysis vs expert-based immunophenotypic classification of B-cell chronic lymphoproliferative disorders: a step forward in the standardization of clinical immunophenotyping. Leukemia. 2010;24:1927–1933. doi: 10.1038/leu.2010.160.
    1. Roederer M, Nozzi JL, Nason MC. SPICE: exploration and analysis of post-cytometric complex multivariate datasets. Cytometry A. 2011;79:167–174. doi: 10.1002/cyto.a.21015.
    1. Azad A, Pyne S, Pothen A. Matching phosphorylation response patterns of antigen-receptor-stimulated T cells via flow cytometry. BMC Bioinformatics. 2012;13:S10.
    1. Bendall SC, et al. Single-cell mass cytometry of differential immune and drug responses across a human hematopoietic continuum. Science. 2011;332:687–696. doi: 10.1126/science.1198704.
    1. Qiu P, et al. Extracting a cellular hierarchy from high-dimensional cytometry data with SPADE. Nat. Biotechnol. 2011;29:886–891. doi: 10.1038/nbt.1991.
    1. Aghaeepour N, et al. RchyOptimyx: cellular hierarchy optimization for flow cytometry. Cytometry A. 2012;81:1022–1030. doi: 10.1002/cyto.a.22209.
    1. Chan C, et al. Statistical mixture modeling for cell subtype identification in flow cytometry. Cytometry A. 2008;73:693–701. doi: 10.1002/cyto.a.20583.
    1. El Khettabi, F. & Kyriakidis, P. The L2 discrepancy framework to mine high-throughput screening data for targeted drug discovery: application to AIDS antiviral activity data of the National Cancer Institute. (Data Mining for Biomedical Informatics workshop, SIAM Conf. Data Mining 2006).
    1. Naim, I. et al. Swift: scalable weighted iterative sampling for flow cytometry clustering. in Acoustics Speech and Signal Processing 509–512 (IEEE, 2010).
    1. Hahne F, et al. flowCore: a Bioconductor package for high throughput flow cytometry. BMC Bioinformatics. 2009;10:106. doi: 10.1186/1471-2105-10-106.
    1. Yang P, Yang YH, Zhou BB, Zomaya AY. A review of ensemble methods in bioinformatics. Current Bioinformatics. 2010;5:296–308. doi: 10.2174/157489310794072508.
    1. Maecker HT, et al. Standardization of cytokine flow cytometry assays. BMC Immunol. 2005;6:13. doi: 10.1186/1471-2172-6-13.
    1. Prill RJ, et al. Towards a rigorous assessment of systems biology models: the DREAM3 challenges. PLoS ONE. 2010;5:e9202. doi: 10.1371/journal.pone.0009202.
    1. Stolovitzky G, Prill RJ, Califano A. Lessons from the DREAM2 Challenges. Ann. NY Acad. Sci. 2009;1158:159–195. doi: 10.1111/j.1749-6632.2009.04497.x.
    1. Meyer P, et al. Verification of systems biology research in the age of collaborative competition. Nat. Biotechnol. 2011;29:811–815. doi: 10.1038/nbt.1968.
    1. Califano A, Kellis M, Stolovitzky G. Preface: RECOMB Systems Biology, Regulatory Genomics, and DREAM 2011 special issue. J. Comput. Biol. 2012;19:101. doi: 10.1089/cmb.2012.010p.
    1. Bain, B.J. Blood Cells: A Practical Guide 4th edn., Ch. 9, 398–468 (Wiley-Blackwell, 2006).
    1. Maddox AM, et al. Philadelphia chromosome-positive adult acute leukemia with monosomy of chromosome number seven: a subgroup with poor response to therapy. Leuk. Res. 1983;7:509–522. doi: 10.1016/0145-2126(83)90046-2.
    1. Tecimer C, Loy BA, Martin AW. Acute myeloblastic leukemia (M0) with an unusual chromosomal abnormality: translocation (1;14)(p13;q32) Cancer Genet. Cytogenet. 1999;111:175–177. doi: 10.1016/S0165-4608(98)00234-9.
    1. Peters JM, Ansari MQ. Multiparameter flow cytometry in the diagnosis and management of acute leukemia. Arch. Pathol. Lab. Med. 2011;135:44–54.
    1. Hornik, K. & Bohm, W. Hard and soft Euclidean consensus partitions. in Data Analysis, Machine Learning and Applications (eds. Preisach, C., Burkhardt, H., Schmidt-Thieme, L. & Decker, R.) 147–154 (Springer, 2008).
    1. Hornik K. A clue for cluster ensembles. J. Stat. Softw. 2005;14:1–25. doi: 10.18637/jss.v014.i12.
    1. Gentleman R, Temple Lang D. Statistical analyses and reproducible research. J. Comput. Graph. Statist. 2007;16:1–23. doi: 10.1198/106186007X178663.
    1. Lee JA, et al. MIFlowCyt: the minimum information about a Flow Cytometry Experiment. Cytometry A. 2008;73:926–930. doi: 10.1002/cyto.a.20623.
    1. Brinkman RR, et al. High-content flow cytometry and temporal data analysis for defining a cellular signature of graft-versus-host disease. Biol. Blood Marrow Transplant. 2007;6:691–700. doi: 10.1016/j.bbmt.2007.02.002.
    1. Aghaeepour, N, Khodabakhshi, A.H. & Brinkman, R.R. Clustering Theory Workshop (Neural Information Processing Systems, 2009).
    1. Hesterberg T, Moore DS, Monaghan S, Clipson A, Epstein R. Bootstrap methods and permutation tests. Introduction to the Practice of Statistics. 2005;16:1–70.
    1. Dym CL, Wood WH, Scott MJ. Rank ordering engineering designs: pairwise comparison charts and Borda counts. Res. Eng. Des. 2002;13:236–242. doi: 10.1007/s00163-002-0019-8.
    1. Maecker HT, et al. Standardization of cytokine flow cytometry assays. BMC Immunol. 2005;6:13. doi: 10.1186/1471-2172-6-13.
    1. Maecker HT, McCoy JP, Nussenblatt R. Standardizing immunophenotyping for the human immunology project. Nat. Rev. Immunol. 2012;12:191–200. doi: 10.1038/nri3158.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe