Abstract
Removing undesired reflection from an image captured through a glass surface is a very challenging problem with many practical applications. For improving reflection removal, cascaded deep models have been usually adopted to estimate the transmission in a progressive manner. However, most existing methods are still limited in exploiting the result in prior stage for guiding transmission estimation. In this paper, we present a novel two-stage network with reflection-aware guidance (RAGNet) for single image reflection removal (SIRR). To be specific, the reflection layer is firstly estimated due to that it generally is much simpler and is relatively easier to estimate. Reflection-aware guidance (RAG) module is then elaborated for better exploiting the estimated reflection in predicting transmission layer. By incorporating feature maps from the estimated reflection and observation, RAG can be used (i) to mitigate the effect of reflection from the observation, and (ii) to generate mask in soft partial convolution for mitigating the effect of deviating from linear combination hypothesis. A dedicated mask loss is further presented for reconciling the contributions of encoder and decoder features. Experiments on five commonly used datasets demonstrate the quantitative and qualitative superiority of our RAGNet in comparison to the state-of-the-art SIRR methods. The source code and pre-trained model are available at https://github.com/liyucs/RAGNet.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability statement
The datasets used during the training are available in http://host.robots.ox.ac.uk/pascal/VOC/ and https://github.com/ceciliavision/perceptual-reflection-removal. The datasets used during the testing are available in https://github.com/fqnchina/CEILNet and https://sir2data.github.io/.
References
Pal SK, Pramanik A, Maiti J, Mitra P (2021) Deep learning in multi-object detection and tracking: state of the art. Appl Intell 51(9):6400–6429
Yang J, Ge H, Yang J, Tong Y, Su S (2022) Online multi-object tracking using multi-function integration and tracking simulation training. Appl Intell 52(2):1268–1288
Punnappurath A, Brown MS (2019) Reflection removal using a dual-pixel sensor. In: IEEE conference on computer vision and pattern recognition, pp 1556–1565
Fan Q, Yang J, Hua G, Chen B, Wipf D (2017) A generic deep architecture for single image reflection removal and image smoothing. In: IEEE international conference on computer vision, pp 3238–3247
Wei K, Yang J, Fu Y, Wipf D, Huang H (2019) Single image reflection removal exploiting misaligned training data and network enhancements. In: IEEE conference on computer vision and pattern recognition, pp 8178–8187
Asif M, Chen L, Song H, Yang J, Frangi AF (2021) An automatic framework for endoscopic image restoration and enhancement. Appl Intell 51(4):1959–1971
Chang Y, Jung C (2018) Single image reflection removal using convolutional neural networks. IEEE Trans Image Process 28(4):1954–1966
Zhang X, Ng R, Chen Q (2018) Single image reflection separation with perceptual losses. In: IEEE conference on computer vision and pattern recognition, pp 4786–4794
Yang J, Gong D, Liu L, Shi Q (2018) Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In: European conference on computer vision, pp 654–669
Heydecker D, Maierhofer G, Aviles-Rivero AI, Fan Q, Chen D, Schönlieb C-B, Süsstrunk S (2019) Mirror, mirror, on the wall, who’s got the clearest image of them all?—a tailored approach to single image reflection removal. IEEE Trans Image Process 28(12):6185–6197
Kim S, Huo Y, Yoon S-E (2020) Single image reflection removal with physically-based training images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5164–5173
Li C, Yang Y, He K, Lin S, Hopcroft JE (2020) Single image reflection removal through cascaded refinement. In: IEEE conference on computer vision and pattern Recognition, pp 3565–3574
Lei C, Huang X, Zhang M, Yan Q, Sun W, Chen Q (2020) Polarized reflection removal with perfect alignment in the wild. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1750–1758
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Peng Y-T, Cheng K-H, Fang I-S, Peng W-Y, Wu J-S (2022) Single image reflection removal based on knowledge-distilling content disentanglement. IEEE Signal Process Lett 29:568–572
Chang Y-C, Lu C-N, Cheng C-C, Chiu W-C (2021) Single image reflection removal with edge guidance, reflection classifier, and recurrent decomposition. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 2033–2042
Agrawal A, Raskar R, Nayar S.K, Li Y (2005) Removing photography artifacts using gradient projection and flash-exposure sampling. In: ACM SIGGRAPH, pp 828–835
Fu Y, Lam A, Sato I, Okabe T, Sato Y (2013) Separating reflective and fluorescent components using high frequency illumination in the spectral domain. In: IEEE international conference on computer vision, pp 457–464
Schechner Y.Y, Kiryati N, Basri R (2000) Separation of transparent layers using focus, pp 25–39
Sarel B, Irani M (2004) Separating transparent layers through layer information exchange. In: European conference on computer vision, pp 328–341
Wieschollek P, Gallo O, Gu J, Kautz J (2018) Separating reflection and transmission images in the wild. In: European conference on computer vision, pp 89–104
Xue T, Rubinstein M, Liu C, Freeman WT (2015) A computational approach for obstruction-free photography. ACM Trans Graph 34(4):1–11
Szeliski R, Avidan S, Anandan P (2000) Layer extraction from multiple images containing reflections and transparency. In: IEEE conference on computer vision and pattern recognition, pp 246–253
Li Y, Brown MS (2013) Exploiting reflection change for automatic reflection removal. In: IEEE international conference on computer vision, pp 2432–2439
Guo X, Cao X, Ma Y (2014) Robust separation of reflection from multiple images. In: IEEE conference on computer vision and pattern recognition, pp 2187–2194
Gai K, Shi Z, Zhang C (2012) Blind separation of superimposed moving images using image statistics. IEEE Trans Pattern Anal Mach Intell 34(1):19–32
Sinha SN, Kopf J, Goesele M, Scharstein D, Szeliski R (2012) Image-based rendering for scenes with reflections. ACM Trans Graph 31(4):1–10
Yang J, Li H, Dai Y, Tan RT (2016) Robust optical flow estimation of double-layer images under transparency or reflection. In: IEEE conference on computer vision and pattern recognition, pp 1410–1419
Sun C, Liu S, Yang T, Zeng B, Wang Z, Liu G (2016) Automatic reflection removal using gradient intensity and motion cues. In: ACM international conference on multimedia, pp 466– 470
Han B-J, Sim J-Y (2017) Reflection removal using low-rank matrix completion. In: IEEE conference on computer vision and pattern recognition, pp 5438–5446
Cheong JY, Simon C, Kim C-S, Park IK (2017) Reflection removal under fast forward camera motion. IEEE Trans Image Process 26(12):6061–6073
Simon C, Kyu Park I (2015) Reflection removal for in-vehicle black box videos. In: IEEE conference on computer vision and pattern recognition, pp 4231–4239
Li T, Chan Y-H, Lun DP (2020) Improved multiple-image-based reflection removal algorithm using deep neural networks. IEEE Trans Image Process 30:68–79
Han B-J, Sim J-Y (2018) Glass reflection removal using co-saliency-based image alignment and low-rank matrix completion in gradient domain. IEEE Trans Image Process 27(10):4873– 4888
Levin A, Weiss Y (2007) User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans Pattern Anal Mach Intell 29(9):1647–1654
Wan R, Shi B, Duan L-Y, Tan A-H, Gao W, Kot AC (2018) Region-aware reflection removal with unified content and gradient priors. IEEE Trans Image Process 27(6):2927–2941
Arvanitopoulos N, Achanta R, Susstrunk S (2017) Single image reflection suppression. In: IEEE conference on computer vision and pattern recognition, pp 4498–4506
Sandhan T, Young Choi J (2017) Anti-glare: tightly constrained optimization for eyeglass reflection removal. In: IEEE conference on computer vision and pattern recognition, pp 1241–1250
Li Y, Brown MS (2014) Single image layer separation using relative smoothness. In: IEEE conference on computer vision and pattern recognition, pp 2752–2759
Wan R, Shi B, Tan A-H, Kot AC (2016) Depth of field guided reflection removal. In: IEEE international conference on image processing, pp 21–25
Shih Y, Krishnan D, Durand F, Freeman W.T (2015) Reflection removal using ghosting cues. In: IEEE conference on computer vision and pattern recognition, pp 3193–3201
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
Dong Z, Xu K, Yang Y, Bao H, Xu W, Lau RW (2021) Location-aware single image reflection removal. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 5017–5026
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241
Chen Y, Zhang H, Liu L, Chen X, Zhang Q, Yang K, Xia R, Xie J (2021) Research on image inpainting algorithm of improved gan based on two-discriminations networks. Appl Intell 51 (6):3460–3474
Zhang J, Liu Y, Guo C, Zhan J (2022) Optimized segmentation with image inpainting for semantic mapping in dynamic scenes. Appl Intell, pp 1–16
Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision, pp 694–711
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680
Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338
Wan R, Shi B, Duan L-Y, Tan A-H, Kot AC (2017) Benchmarking single-image reflection removal algorithms. In: IEEE International Conference on Computer Vision, pp 3922–3930
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L (2019) Pytorch: an imperative style, high-performance deep learning library. In: Advances in neural information processing systems, pp 8026–8037
Acknowledgements
This work was partially supported by National Natural Science Foundation of China (Nos. 62172127 and U22B2035), Natural Science Foundation of Heilongjiang Province (No. YQ2022F004).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, Y., Liu, M., Yi, Y. et al. Two-stage single image reflection removal with reflection-aware guidance. Appl Intell 53, 19433–19448 (2023). https://doi.org/10.1007/s10489-022-04391-6
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-022-04391-6