Abstract
High-spatial-resolution mass spectrometry imaging (HSR-MSI) provides precise spatial information on thousands of biomolecules without labelling across a tissue section. Deep learning methods, trained on large numbers of images, can be used to further improve resolution. However, the limited amount of HSR-MSI data that are publicly available mean that super-resolution reconstruction of images obtained by MSI using deep learning is still a challenge. Here we develop a deep learning framework based on transfer learning called MSI from optical super-resolution (MOSR) that substantially reduces the requirement for sample size. Needing only ten HSR-MSI images, the method transfers knowledge learned from abundant optical images (~15,000) to MSI tasks. Compared with the deep learning model without transfer learning, the MOSR model obtains better image quality with higher peak signal-to-noise ratios and structural similarity index values. It also achieves higher training efficiency and a stronger generalization performance. The MOSR model predicts HSR-MSI images with very small sample size and could transform applications with super-resolution MSI.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
9,800 Yen / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
¥14,900 per year
only ¥1,242 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The public optical imaging data can be found at Allen Brain Atlas database (https://atlas.brain-map.org/) under dataset id numbers 67787347, 68162246, 68442901, 72270832 and so on. To facilitate access, we collated all the optical imaging data that we used to a public available data repository via Figshare at https://doi.org/10.6084/m9.figshare.22639936.v1 (ref. 69). Each image in the Figshare is named by its image id number in the Allen Brain Atlas database. The public MSI imaging data are available from the METASPACE Platform under dataset ids 100um_M2_003_Recal (https://metaspace2020.eu/datasets?q=2017-07-18_17h21m08s), FullBrain_Norh_neg (https://metaspace2020.eu/datasets?q=2020-09-01_10h04m59s), 20200904_FullBrain_Norh_pos_2 (https://metaspace2020.eu/datasets?q=2020-09-04_11h03m04s) and 20200827_Brain_Cer_Nor_neg_i (https://metaspace2020.eu/datasets?q=2021-04-15_13h34m52s).
Code availability
In this work we used open-source image and video restoration toolbox (BasicSR; https://github.com/xinntao/BasicSR) to build ESRGAN models. All custom code is available via GitHub at https://github.com/USTC-xlab/MOSR (ref. 70).
References
Amstalden van Hove, E. R., Smith, D. F. & Heeren, R. M. A. A concise review of mass spectrometry imaging. J. Chromatogr. A 1217, 3946–3954 (2010).
Buchberger, A. R., DeLaney, K., Johnson, J. & Li, L. Mass spectrometry imaging: a review of emerging advancements and future insights. Anal. Chem. 90, 240–265 (2018).
Hansen, R. L. & Lee, Y. J. High-spatial resolution mass spectrometry imaging: toward single cell metabolomics in plant tissues. Chem. Rec. 18, 65–77 (2018).
Römpp, A. & Spengler, B. Mass spectrometry imaging with high resolution in mass and space. Histochem. Cell Biol. 139, 759–783 (2013).
Hsieh, Y. et al. Matrix-assisted laser desorption/ionization imaging mass spectrometry for direct measurement of clozapine in rat brain tissue. Rapid Commun. Mass Spectrom. 20, 965–972 (2006).
Bunch, J., Clench, M. R. & Richards, D. S. Determination of pharmaceutical compounds in skin by imaging matrix-assisted laser desorption/ionisation mass spectrometry. Rapid Commun. Mass Spectrom. 18, 3051–3060 (2004).
Prideaux, B. & Stoeckli, M. Mass spectrometry imaging for drug distribution studies. J. Proteom. 75, 4999–5013 (2012).
Takáts, Z., Wiseman Justin, M., Gologan, B. & Cooks, R. G. Mass spectrometry sampling under ambient conditions with desorption electrospray ionization. Science 306, 471–473 (2004).
Wiseman Justin, M. et al. Desorption electrospray ionization mass spectrometry: imaging drugs and metabolites in tissues. Proc. Natl Acad. Sci. USA 105, 18120–18125 (2008).
Nilsson, A. et al. Investigating nephrotoxicity of polymyxin derivatives by mapping renal distribution using mass spectrometry imaging. Chem. Res. Toxicol. 28, 1823–1830 (2015).
Li, B. et al. Interrogation of spatial metabolome of Ginkgo biloba with high-resolution matrix-assisted laser desorption/ionization and laser desorption/ionization mass spectrometry imaging. Plant Cell Environ. 41, 2693–2703 (2018).
Cobice, D. F. et al. Mass spectrometry imaging for dissecting steroid intracrinology within target tissues. Anal. Chem. 85, 11576–11584 (2013).
Yin, R., Burnum-Johnson, K. E., Sun, X., Dey, S. K. & Laskin, J. High spatial resolution imaging of biological tissues using nanospray desorption electrospray ionization mass spectrometry. Nat. Protoc. 14, 3445–3470 (2019).
Qi, K. et al. Cholesterol was identified as a biomarker in human melanocytic nevi using DESI and DESI/PI mass spectrometry imaging. Talanta 231, 122380 (2021).
Thoma, C. Making DESI-MSI desirable. Nat. Rev. Urol. 14, 325–325 (2017).
Balluff, B., Heeren, R. M. A. & Race, A. M. An overview of image registration for aligning mass spectrometry imaging with clinically relevant imaging modalities. J. Mass Spectrom. Adv. Clin. Lab 23, 26–38 (2022).
Bokhart, M. T., Nazari, M., Garrard, K. P. & Muddiman, D. C. MSiReader v1.0: evolving open-source mass spectrometry imaging software for targeted and untargeted analyses. J. Am. Soc. Mass. Spectrom. 29, 8–16 (2018).
Van de Plas, R., Yang, J., Spraggins, J. & Caprioli, R. M. Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping. Nat. Methods 12, 366–372 (2015).
Van Malderen, S. J., van Elteren, J. T. & Vanhaecke, F. Submicrometer imaging by laser ablation-inductively coupled plasma mass spectrometry via signal and image deconvolution approaches. Anal. Chem. 87, 6125–6132 (2015).
Westerhausen, M. T. et al. Super-resolution reconstruction for two- and three-dimensional LA-ICP-MS bioimaging. Anal. Chem. 91, 14879–14886 (2019).
Titus, J. & Geroge, S. A comparison study on different interpolation methods based on satellite images. Int. J. Eng. Res. Technol. 2, 82–85 (2013).
Dianyuan, H. Comparison of commonly used image interpolation methods. In Proc. 2nd International Conference on Computer Science and Electronics Engineering 10, 1556–1559 (Atlantis Press, 2013).
Zhao, C., Guo, L., Dong, J. & Cai, Z. Mass spectrometry imaging-based multi-modal technique: next-generation of biochemical analysis strategy. Innovation 2, 100151–100151 (2021).
Porta Siegel, T. et al. Mass spectrometry imaging and integration with other imaging modalities for greater molecular understanding of biological tissues. Mol. Imag. Biol. 20, 888–901 (2018).
DeLaney, K., Phetsanthad, A. & Li, L. Advances in high-resolution maldi mass spectrometry for neurobiology. Mass Spectrom. Rev. 41, 194–214 (2022).
Holzlechner, M., Eugenin, E. & Prideaux, B. Mass spectrometry imaging to detect lipid biomarkers and disease signatures in cancer. Cancer Rep. 2, e1229–e1229 (2019).
Spraggins, J. M. & Caprioli, R. M. High-speed MALDI-TOF imaging mass spectrometry: rapid ion image acquisition and considerations for next generation instrumentation. J. Am. Soc. Mass. Spectrom. 22, 1022–1031 (2011).
Batson, J. & Royer, L. Noise2Self: blind denoising by self-supervision. In Proc. 36th International Conference on Machine Learning Vol. 97 (eds. Kamalika, C. & Ruslan, S.) 524–533 (PMLR, 2019).
Krull, A., Buchholz, T.-O. & Jug, F. Noise2Void-learning denoising from single noisy images. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 2129–2137 (IEEE, 2019).
Li, Y. et al. DLBI: deep learning guided Bayesian inference for structure reconstruction of super-resolution fluorescence microscopy. Bioinformatics 34, i284–i294 (2018).
Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).
Buchholz, T., Jordan, M., Pigino, G. & Jug, F. Cryo-CARE: content-aware image restoration for cryo-transmission electron microscopy data. In Proc. IEEE 16th International Symposium on Biomedical Imaging 502–506 (IEEE, 2019).
de Haan, K., Ballard, Z. S., Rivenson, Y., Wu, Y. & Ozcan, A. Resolution enhancement in scanning electron microscopy using deep learning. Sci Rep. 9, 12050 (2019).
Heinrich, L., Bogovic, J. A. & Saalfeld, S. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2017, Vol. 10434 (eds Descoteaux, M. et al.) 135–143 (Springer, Cham, 2017).
Sreehari, S. et al. Multi-resolution data fusion for super-resolution electron microscopy. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 1084–1092 (IEEE, 2017).
Park, J. et al. Computed tomography super-resolution using deep convolutional neural network. Phys. Med. Biol. 63, 145011 (2018).
You, C. et al. CT Super-Resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE). IEEE Trans. Med. Imag. 39, 188–203 (2020).
Zhang, Z. et al. CT super resolution via zero shot learning. Preprint at https://arxiv.org/abs/2012.08943 (2020).
Chen, H. et al. Real-world single image super-resolution: a brief review. Inform. Fusion 79, 124–145 (2022).
Yang, W. et al. Deep learning for single image super-resolution: a brief review. IEEE Trans. Multimedia 21, 3106–3121 (2019).
Yang, L., Hanneke, S. & Carbonell, J. A theory of transfer learning with applications to active learning. Mach. Learn. 90, 161–189 (2013).
Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Proc. 27th International Conference on Neural Information Processing Systems – Volume 2 3320–3328 (MIT Press, 2014).
He, J. et al. A sensitive and wide coverage ambient mass spectrometry imaging method for functional metabolites based molecular histology. Adv. Sci. 5, 1800250 (2018).
Veselkov, K. A. et al. Chemo-informatic strategy for imaging mass spectrometry-based hyperspectral profiling of lipid signatures in colorectal cancer. Proc. Natl Acad. Sci. USA 111, 1216–1221 (2014).
Wang, X. et al. ESRGAN: enhanced super-resolution generative adversarial networks. In Proc. European Conference on Computer Vision – ECCV 2018 Workshops (eds Leal-Taixé, L. & Roth, S.) 63–79 (Springer, 2019).
Lein, E. S. et al. Genome-wide atlas of gene expression in the adult mouse brain. Nature 445, 168–176 (2007).
Ledig, C. et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 105–114 (IEEE, 2017).
Horé, A. & Ziou, D. Image quality metrics: PSNR vs. SSIM. In Proc. 20th International Conference on Pattern Recognition 2366–2369 (IEEE, 2010).
Fang, L. et al. Deep learning-based point-scanning super-resolution imaging. Nat. Methods 18, 406–416 (2021).
Culley, S. et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat. Methods 15, 263–266 (2018).
Aggarwal, C. C. Neural Networks and Deep Learning (Springer, 2018).
Yasaka, K. & Abe, O. Deep learning and artificial intelligence in radiology: current applications and future directions. PLoS Med. 15, e1002707 (2018).
Zhang, L. et al. Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Trans. Med. Imag. 39, 2531–2540 (2020).
Large, R. & Knof, H. A comparison of negative and positive ion mass spectrometry. Org. Mass Spectrom. 11, 582–598 (1976).
Tyler, B. J. et al. Denoising of mass spectrometry images via inverse maximum signal factors analysis. Anal. Chem. 94, 2835–2843 (2022).
Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).
Barbastathis, G., Ozcan, A. & Situ, G. On the use of deep learning for computational imaging. Optica 6, 921–943 (2019).
Moen, E. et al. Deep learning for cellular image analysis. Nat. Methods 16, 1233–1246 (2019).
Wang, H. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103–110 (2019).
Arentz, G. et al. in Advances in Cancer Research Vol. 134 (eds. Drake, R. R. & McDonnell, L. A.) 27–66 (Academic, 2017).
Wolrab, D., Jirásko, R., Chocholoušková, M., Peterka, O. & Holčapek, M. Oncolipidomics: mass spectrometric quantitation of lipids in cancer research. Trends Anal. Chem. 120, 115480 (2019).
Palmer, A. et al. FDR-controlled metabolite annotation for high-resolution imaging mass spectrometry. Nat. Methods 14, 57–60 (2017).
Metodiev, M. D., Steven, R. T., Loizeau, X., Takats, Z. & Bunch, J. Modality agnostic model for spatial resolution in mass spectrometry imaging: application to MALDI MSI data. Anal. Chem. 93, 15295–15305 (2021).
Zubair, F., Prentice, B. M., Norris, J. L., Laibinis, P. E. & Caprioli, R. M. Standard reticle slide to objectively evaluate spatial resolution and instrument performance in imaging mass spectrometry. Anal. Chem. 88, 7302–7311 (2016).
Blau, Y., Mechrez, R., Timofte, R., Michaeli, T. & Zelnik-Manor, L. in Computer Vision – ECCV 2018 Workshops (eds Leal-Taixé, L. & Roth, S.) 334–355 (Springer, 2019).
Ma, C., Yang, C.-Y., Yang, X. & Yang, M.-H. Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Und. 158, 1–16 (2017).
Mittal, A., Soundararajan, R. & Bovik, A. C. Making a ‘completely blind’ image quality analyzer. IEEE Signal Process. Lett. 20, 209–212 (2013).
Wang, X., Hou, Y., Hou, Z., Xiong, W. & Huang, G. Mass spectrometry imaging of brain cholesterol and metabolites with trifluoroacetic acid-enhanced desorption electrospray ionization. Anal. Chem. 91, 2719–2726 (2019).
Liao, T. Optical data for MOSR. Figshare https://doi.org/10.6084/m9.figshare.22639936.v1 (2023).
USTC-xlab. USTC-xlab/MOSR: MOSR for MSI images (v2.0.1). Zenodo https://doi.org/10.5281/zenodo.7833505 (2023).
Acknowledgements
We thank National Key R&D Program of China (grant numbers 2021YFA0804900 and 2020YFA0112203), the National Natural Science Foundation of China (grant numbers 32225020, 91849206, 91942315, 92049304 and 32121002 to W.X. and 21974130 and 91849116 to H.Z.), the Strategic Priority Research Program of the Chinese Academy of Sciences (grant number XDB39050000), the Youth Innovation Promotion Association CAS, University Synergy Innovation Program of Anhui Province (grant number GXXT-2022-033), the Key Research Program of Frontier Science (CAS, grant number ZDBS-LY-SM002), the CAS Interdisciplinary Innovation Team (grant number JCTD-2018-20), the Fundamental Research Funds for the Central Universities, USTC Research Funds of the Double First-Class Initiative (grant numbers YD9100002001 to W.X. and YD9100002005 to H.Z.), the CAS Project for Young Scientists in Basic Research (grant number YSBR-013) and the CAS Collaborative Innovation Program of Hefei Science Center (grant number 2021HSC-CIP003).
Author information
Authors and Affiliations
Contributions
H.Z. and W.X. designed research and supervised the project. T.L. performed the experiments with assistance from Z.R. and Z.C. T.L. analysed data with assistance from Z.L., J.L., M.Y., Q.C., Ziyi Wang, L.Y., S.G., L.S., Zilei Wang, C.M. and W.Q. T.L., H.Z. and W.X. wrote the paper.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Machine Intelligence thanks Dong Hye Ye and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Figs. 1–18.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liao, T., Ren, Z., Chai, Z. et al. A super-resolution strategy for mass spectrometry imaging via transfer learning. Nat Mach Intell 5, 656–668 (2023). https://doi.org/10.1038/s42256-023-00677-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42256-023-00677-7
This article is cited by
-
Cross-layer transmission realized by light-emitting memristor for constructing ultra-deep neural network with transfer learning ability
Nature Communications (2024)
-
Transfer learning enables identification of multiple types of RNA modifications using nanopore direct RNA sequencing
Nature Communications (2024)
-
Super-resolution techniques for biomedical applications and challenges
Biomedical Engineering Letters (2024)