Estimating Illumination Condition and Spectral Reflectance Based on the Color Variation Between Shadowed Region and Its Surrounding Regions

doi:10.12263/DZXB.20211291

Acta Electronica Sinica ›› 2023, Vol. 51 ›› Issue (8): 2098-2109. doi: 10.12263/DZXB.20211291

• PAPERS • Previous Articles Next Articles

Estimating Illumination Condition and Spectral Reflectance Based on the Color Variation Between Shadowed Region and Its Surrounding Regions

WANG Xiao¹^,³, YAO Si-yuan¹^,³(), DAI Peng-wen¹^,³, WANG Rui¹^,²^,³, CAO Xiao-chun¹^,³()

^1.State Key Laboratory of Information Security，Institute of Information Engineering，Chinese Academy of Sciences，Beijing 100093，China
^2.Zhejiang Laboratory，Hangzhou，Zhejiang 311100，China
^3.School of Cyber Security，University of Chinese Academy of Sciences，Beijing 100049，China

Received:2021-09-21 Revised:2021-11-17 Online:2023-08-25 Published:2023-10-31
Contact: YAO Si-yuan, CAO Xiao-chun E-mail:yaosiyuan@bupt.edu.cn;caoxiaochun@iie.ac.cn
Supported by:
National Natural Science Foundation of China Under Grants(U20B2066);National Key Research and Development Program of China(2020AAA0109304);Open Research Projects of Zhejiang Lab(2021KB0AB01)

基于阴影和周边区域色差的光照条件和光谱反射率计算方法

王潇¹^,³, 尧思远¹^,³(), 代朋纹¹^,³, 王蕊¹^,²^,³, 操晓春¹^,³()

^1.中国科学院信息工程研究所信息安全国家重点实验室, 北京 100093
^2.之江实验室, 浙江杭州 311100
^3.中国科学院大学网络空间安全学院, 北京 100049

通讯作者: 尧思远,操晓春 E-mail:yaosiyuan@bupt.edu.cn;caoxiaochun@iie.ac.cn
作者简介:王潇男，1991年5月生，河南平顶山人.2016年毕业于西北工业大学，获工程硕士学位.现为中国科学院信息工程研究所博士生.主要研究方向为计算机视觉、人工智能等.E-mail: wangxiao@iie.ac.cn
尧思远男，1990年5月生，江西赣州人.2014年毕业于中国电子科技集团电子科学研究院，获工学硕士学位.现为中国科学院信息工程研究所在读博士生.研究方向为目标跟踪检测、智能信号处理.
代朋纹男，1990年9月生，四川内江人.2014年毕业于重庆大学，获工学学士学位.现为中国科学院信息工程研究所博士生.主要研究方向为计算机视觉、人工智能等.E-mail: daipengwen@iie.ac.cn
王蕊（通讯作者）女，1981年11月生，北京人.2011年于中国科学院研究生院获得博士学位.中国科学院信息工程研究所信息安全国家重点实验室研究员、博士生导师、IEEE会员.主要研究方向为人工智能、计算机视觉、多媒体内容理解及深度学习等.E-mail: wangrui@iie.ac.cn
操晓春男，1980年生，北京人，2005年于中弗罗里达大学获博士学位.中国科学院信息工程研究所信息安全国家重点实验室研究员、博士生导师、IEEE高级会员.主要研究方向为人工智能、计算机视觉、网络空间安全等.
基金资助:
国家自然科学基金企业创新发展联合基金重点支持项目(U20B2066);科技创新2030-“新一代人工智能”重大项目课题(2020AAA0109304);之江实验室开放课题(2021KB0AB01)

Abstract

Abstract:

The illumination condition and the spectral reflectance are very useful for scene rendering and augmented reality but difficult to collect. Thus, it is an important task to get them from ordinary image. In this paper, we proposed a method that estimates illumination condition and spectral reflectance of natural scene without training process by analyzing the color variation between shadowed region and its surrounding regions. In natural scenes, the shadowed region was only lighted by skylight while its surrounding unshadowed regions were lighted by both sunlight and skylight. The color variation between them was the effect of shielded light source when they had the same spectral reflectance. The difference between these two colors was caused by the illumination condition of captured scene, while they were connected by their spectral reflectance. These colors provided the cue to calculate the illumination condition and the spectral reflectance. We formulized the whole process of image generation using prior knowledge and model. Then, we designed an optimization function based on the relationship between these colors to estimate the illumination condition and the spectral reflectance. The results of comparison experiment and the application experiment on shadow removal showed that the proposed method performed favorably against the state-of-the-art methods.

Key words: shadowed region, unshadowed region, color variation, illumination condition, spectral reflectance

摘要：

光照条件和光谱反射率对增强现实和场景渲染任务非常有用，但是通过仪器采集比较困难，因此根据普通图像计算光照条件和光谱反射率是重要的计算机视觉任务.本文设计了一种通过分析场景中的阴影区域和周边区域颜色之间的色差，无需训练数据即可计算场景的光照条件和光谱反射率的算法.在自然场景中，阴影区域由于遮挡只能接收和反射天空光，而其周边的非阴影区域则同时受到太阳光和天空光的影响.当阴影和非阴影区域有着相同的光谱反射率时，两块区域颜色之间的色差反映了被遮挡光源的信息.两块区域之间的颜色差异由光照条件造成，又以光谱反射率为联系.因此，可以利用这两种颜色之间的关系计算光照条件和光谱反射率.本文使用先验知识和模型公式化表达某一区域在光照下呈现在图像中的颜色，然后基于两种颜色以及两者之间的色差设计了一种优化算法，通过计算最优解来计算场景的光照条件以及光谱反射率.通过在多个数据集上的一系列对比实验，以及一个使用光照条件和光谱反射率来去除阴影区域的应用实验，证明了该算法的准确性.

关键词: 阴影区域, 非阴影区域, 色差, 光照条件, 光谱反射率

CLC Number:

TP183

WANG Xiao, YAO Si-yuan, DAI Peng-wen, WANG Rui, CAO Xiao-chun. Estimating Illumination Condition and Spectral Reflectance Based on the Color Variation Between Shadowed Region and Its Surrounding Regions[J]. Acta Electronica Sinica, 2023, 51(8): 2098-2109.

王潇, 尧思远, 代朋纹, 王蕊, 操晓春. 基于阴影和周边区域色差的光照条件和光谱反射率计算方法[J]. 电子学报, 2023, 51(8): 2098-2109.

/ Save 0 / Recommend / Download Citations

URL: https://www.ejournal.org.cn/EN/10.12263/DZXB.20211291

https://www.ejournal.org.cn/EN/Y2023/V51/I8/2098

Figures/Tables 13

References 38

1	HOLD-GEOFFROY Y, ATHAWALE A, LALONDE J F. Deep sky modeling for single image outdoor lighting estimation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2020: 6920-6928.
2	TARABALKA Y, CHANUSSOT J, BENEDIKTSSON J A. Segmentation and classification of hyperspectral images using watershed transformation[J]. Pattern Recognition, 2010, 43(7): 2367-2379.
3	XU X, LI J, HUANG X, et al. Multiple morphological component analysis based decomposition for remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(5): 3083-3102.
4	孙乐, 吴泽彬, 冯灿, 等. 一种新的两分类器融合的空谱联合高光谱分类方法[J]. 电子学报, 2015, 43(11): 2210-2217. URL
	SUN L, WU Z B, FENG C, et al. A novel two-classifier fusion method for spectral-spatial hyperspectral classification[J]. Acta Electronica Sinica, 2015, 43(11): 2210-2217. (in Chinese) URL
5	陈允杰, 马辰阳, 孙乐, 等. 基于边缘修正的高光谱图像超像素空谱核分类方法[J]. 电子学报, 2019, 47(1): 73-81. URL
	CHEN Y J, MA C Y, SUN L, et al. Edge-modified superpixel based spectral-spatial kernel method for hyperspectral image classification[J]. Acta Electronica Sinica, 2019, 47(1): 73-81. (in Chinese) URL
6	VAN NGUYEN H, BANERJEE A, CHELLAPPA R. Tracking via object reflectance using a hyperspectral video camera[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops. Piscataway: IEEE, 2010: 44-51.
7	AKHTAR N, MIAN A. Nonparametric coupled Bayesian dictionary and classifier learning for hyperspectral classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(9): 4038-4050.
8	BIOUCAS-DIAS J M, PLAZA A, CAMPS-VALLS G, et al. Hyperspectral remote sensing data analysis and future challenges[J]. IEEE Geoscience and Remote Sensing Magazine, 2013, 1(2): 6-36.
9	AKHTAR N, SHAFAIT F, MIAN A. Bayesian sparse representation for hyperspectral image super resolution[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2015: 3631-3640.
10	PANAGOPOULOS A, WANG C H, SAMARAS D, et al. Simultaneous cast shadows, illumination and geometry inference using hypergraphs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(2): 437-449.
11	PANAGOPOULOS A, VICENTE T F Y, SAMARAS D. Illumination estimation from shadow borders[C]//2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). Piscataway: IEEE, 2012: 798-805.
12	LALONDE J F, EFROS A A, NARASIMHAN S G. Estimating the natural illumination conditions from a single outdoor image[J]. International Journal of Computer Vision, 2012, 98(2): 123-145.
13	HOLD-GEOFFROY Y, SUNKAVALLI K, HADAP S, et al. Deep outdoor illumination estimation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2017: 2373-2382.
14	GEORGOULIS S, REMATAS K, RITSCHEL T, et al. Reflectance and natural illumination from single-material specular objects using deep learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(8): 1932-1947.
15	BELL S, BALA K, SNAVELY N. Intrinsic images in the wild[J]. ACM Transactions on Graphics, 2014, 33(4): 159.
16	Horn B K P, Brooks M J. Shape from shading[J]. Computer Vision, Graphics, and Image Processing, 1990, 50(2): 255-255.
17	LOMBARDI S, NISHINO K. Reflectance and natural illumination from a single image[C]//European Conference on Computer Vision. Berlin, Heidelberg: Springer, 2012: 582-595.
18	PANAGOPOULOS A, WANG C H, SAMARAS D, et al. Illumination estimation and cast shadow detection through a higher-order graphical model[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2011: 673-680.
19	PHONG B T. Illumination for computer generated pictures[J]. Communications of the ACM, 1975, 18(6): 311-317.
20	WAGADARIKAR A, JOHN R, WILLETT R, et al. Single disperser design for coded aperture snapshot spectral imaging[J]. Applied Optics, 2008, 47(10): B44-B51.
21	FORSYTH D A, PONCE J. Computer Vision - A Modern Approach[M]. Second Edition. London: Pitman, 2012.
22	PREETHAM A J, SHIRLEY P, SMITS B. A practical analytic model for daylight[C]//Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1999: 91-100.
23	FINLAYSON G D, HORDLEY S D, LU C, et al. On the removal of shadows from images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(1): 59-68.
24	HUANG X, HUA G, TUMBLIN J, et al. What characterizes a shadow boundary under the Sun and sky? [C]//2011 International Conference on Computer Vision. Piscataway: IEEE, 2012: 898-905.
25	YASUMA F, MITSUNAGA T, ISO D, et al. Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum[J]. IEEE Transactions on Image Processing, 2010, 19(9): 2241-2253.
26	CHAKRABARTI A, ZICKLER T. Statistics of real-world hyperspectral images[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2011: 193-200.
27	WANG J F, LI X, YANG J. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 1788-1797.
28	QU L Q, TIAN J D, HE S F, et al. DeshadowNet: A multi-context embedding deep network for shadow removal[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2017: 2308-2316.
29	LE H, SAMARAS D. Shadow removal via shadow image decomposition[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2020: 8577-8586.
30	AKHTAR N, MIAN A. Hyperspectral recovery from RGB images using Gaussian processes[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(1): 100-113.
31	ARAD B, BEN-SHAHAR O. Sparse recovery of hyperspectral signal from natural RGB images[C]//European Conference on Computer Vision. Cham: Springer, 2016: 19-34.
32	WU J Q, AESCHBACHER J, TIMOFTE R. In defense of shallow learned spectral reconstruction from RGB images[C]//2017 IEEE International Conference on Computer Vision Workshops (ICCVW). Piscataway: IEEE, 2018: 471-479.
33	ZHANG S P, WANG L Z, ZHANG L, et al. Learning tensor low-rank prior for hyperspectral image reconstruction[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 12001-12010.
34	GUO R Q, DAI Q Y, HOIEM D. Paired regions for shadow detection and removal[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(12): 2956-2967.
35	GONG H, COSKER D. Interactive shadow removal and ground truth for variable scene categories[C]//Proceedings of the British Machine Vision Conference 2014. Nottingham, 2014: 1-11.
36	HU X W, ZHU L, FU C W, et al. Direction-aware spatial context features for shadow detection[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 7454-7462.
37	HU X W, JIANG Y T, FU C W, et al. Mask-ShadowGAN: Learning to remove shadows from unpaired data[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2020: 2472-2481.
38	FU L, ZHOU C Q, GUO Q, et al. Auto-exposure fusion for single-image shadow removal[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 10566-10575.

方法	太阳颜色	天空颜色
文献[13]	0.235	0.357
文献[1]	0.226	0.198
本文	0.135	0.093

方法	太阳颜色	天空颜色
文献[13]	0.235	0.357
文献[1]	0.226	0.198
本文	0.135	0.093

图像名称	RMSE				SAM
图像名称	本文	Akh^[30]	Arad^[31]	A+^[32]	本文	Akh^[30]	Arad^[31]	A+^[32]
羽毛	4.91	7.89	9.17	8.56	11.39	12.02	15.06	13.09
辣椒	3.71	5.74	7.64	6.99	10.84	16.34	18.45	17.64
粘土	7.92	8.01	8.44	8.86	14.08	18.42	16.78	16.01
线轴	4.05	6.22	7.01	6.57	12.66	16.15	18.88	17.13
玻璃	6.25	6.68	7.71	7.36	10.92	11.71	13.37	11.76
柠檬	2.36	3.42	5.03	4.93	12.81	18.14	22.99	18.83

图像名称	RMSE				SAM
图像名称	本文	Akh^[30]	Arad^[31]	A+^[32]	本文	Akh^[30]	Arad^[31]	A+^[32]
羽毛	4.91	7.89	9.17	8.56	11.39	12.02	15.06	13.09
辣椒	3.71	5.74	7.64	6.99	10.84	16.34	18.45	17.64
粘土	7.92	8.01	8.44	8.86	14.08	18.42	16.78	16.01
线轴	4.05	6.22	7.01	6.57	12.66	16.15	18.88	17.13
玻璃	6.25	6.68	7.71	7.36	10.92	11.71	13.37	11.76
柠檬	2.36	3.42	5.03	4.93	12.81	18.14	22.99	18.83

图像名称	RMSE					SAM
图像名称	本文	Akh^[30]	Arad^[31]	A+^[32]	TLR^[33]	本文	Akh^[30]	Arad^[31]	A+^[32]	TLR^[33]
室外图像	5.33	5.36	6.81	5.32	5.41	3.63	3.67	4.62	3.99	3.88
室内图像	2.71	2.78	3.50	3.11	3.27	7.08	7.13	7.36	7.29	7.23

Please choose a citation manager

Content to export