电子学报 ›› 2023, Vol. 51 ›› Issue (1): 128-138.DOI: 10.12263/DZXB.20210659
李滔1, 董秀成1, 林宏伟2
收稿日期:
2021-05-24
修回日期:
2022-05-17
出版日期:
2023-01-25
作者简介:
基金资助:
LI Tao1, DONG Xiu-cheng1, LIN Hong-wei2
Received:
2021-05-24
Revised:
2022-05-17
Online:
2023-01-25
Published:
2023-02-23
Supported by:
摘要:
消费级深度相机拍摄的深度图像具有分辨率较低的问题,深度图像超分辨率重建是解决该问题的有效方法.为了提高重建性能,提出一种基于深监督跨尺度注意力网络的深度图像超分辨率重建算法.网络逐级放大,在损失函数中对每一级的输出都进行约束,实现深监督的目的.采用高阶跨尺度注意力模块,将多尺度特征尺度内及跨尺度相关性与注意力机制结合起来,实现多尺度特征的自适应调整.采用内层为宽激活残差、外层为基本残差的双层残差块作为网络基本构成元素,以提高网络对复杂非线性关系的学习能力.实验结果表明,本文算法在主观视觉效果和客观质量评价指标方面都优于当前主流的深度图像超分辨率重建算法.
中图分类号:
李滔, 董秀成, 林宏伟. 基于深监督跨尺度注意力网络的深度图像超分辨率重建[J]. 电子学报, 2023, 51(1): 128-138.
Tao LI, Xiu-cheng DONG, Hong-wei LIN . Depth Map Super-Resolution Reconstruction Based on Deeply Supervised Cross-Scale Attention Network[J]. Acta Electronica Sinica, 2023, 51(1): 128-138.
方法 | Art | Book | Dolls | Laundry | Moebius | Reindeer | Cones | Teddy | Tsukuba | Venus | 平均 | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
PSNR | PE | PSNR | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | |
2倍 | ||||||||||||||||||||||
Bicubic | 39.74 | 2.750 | 47.77 | 0.687 | 48.96 | 0.709 | 44.03 | 1.305 | 49.32 | 0.754 | 42.45 | 1.309 | 40.14 | 3.649 | 42.40 | 2.613 | 32.82 | 6.884 | 45.84 | 1.044 | 43.35 | 2.170 |
RCAN | 55.35 | 0.084 | 57.11 | 0.023 | 55.01 | 0.070 | 56.10 | 0.058 | 55.99 | 0.073 | 55.94 | 0.048 | 51.31 | 0.271 | 51.25 | 0.284 | 0.269 | 55.92 | 0.010 | 54.56 | 0.119 | |
SAN | 55.10 | 0.094 | 57.19 | 0.023 | 55.00 | 0.073 | 56.07 | 0.061 | 55.89 | 0.074 | 55.90 | 0.050 | 51.19 | 0.254 | 51.09 | 0.299 | 53.66 | 55.32 | 0.014 | 54.64 | 0.120 | |
ATGV-Net | 54.11 | 0.108 | 60.29 | 0.023 | 57.04 | 0.054 | 56.10 | 0.073 | 58.46 | 0.055 | 55.59 | 0.061 | 46.77 | 0.643 | 49.97 | 0.482 | 39.65 | 1.488 | 46.41 | 2.336 | 52.44 | 0.532 |
JID | 46.06 | 0.685 | 51.62 | 0.262 | 50.58 | 0.355 | 50.36 | 0.398 | 52.30 | 0.301 | 47.55 | 0.411 | 44.21 | 1.500 | 46.56 | 1.165 | 37.22 | 2.386 | 52.19 | 0.249 | 47.87 | 0.771 |
MSG-Net | 53.16 | 0.148 | 59.94 | 0.025 | 56.70 | 0.043 | 56.75 | 0.061 | 58.17 | 0.059 | 55.59 | 0.069 | 49.00 | 0.611 | 51.07 | 0.384 | 42.80 | 1.247 | 65.08 | 54.83 | 0.265 | |
DepthSR-Net | 53.64 | 0.050 | 55.54 | 54.27 | 0.024 | 55.23 | 0.032 | 55.29 | 53.86 | 0.033 | 50.15 | 49.78 | 0.276 | 45.46 | 0.336 | 54.06 | 0.001 | 52.73 | 0.096 | |||
RYNet | 63.09 | 0.008 | 59.45 | 0.026 | 61.23 | 0.028 | 54.10 | 0.175 | 47.49 | 0.312 | 65.58 | 0.001 | ||||||||||
Proposed | 60.33 | 0.037 | 63.05 | 0.008 | 0.026 | 60.73 | 60.14 | 0.025 | 0.167 | 52.91 | 0.255 | 51.16 | 0.233 | 0.001 | 58.67 | 0.081 | ||||||
4倍 | ||||||||||||||||||||||
Bicubic | 36.38 | 5.507 | 44.05 | 1.465 | 45.80 | 1.488 | 40.52 | 2.755 | 45.63 | 1.658 | 39.19 | 2.546 | 36.45 | 7.207 | 39.04 | 5.276 | 29.47 | 13.052 | 42.52 | 2.200 | 39.91 | 4.32 |
RCAN | 45.19 | 0.470 | 52.80 | 0.125 | 49.35 | 0.315 | 49.45 | 0.308 | 50.93 | 0.295 | 48.00 | 0.265 | 41.61 | 1.690 | 44.09 | 2.204 | 37.15 | 2.031 | 51.28 | 0.138 | 46.99 | 0.78 |
SAN | 45.06 | 0.565 | 52.90 | 0.128 | 49.70 | 0.310 | 49.47 | 0.317 | 51.25 | 0.288 | 47.91 | 0.291 | 41.57 | 1.804 | 44.38 | 1.939 | 36.79 | 2.236 | 51.06 | 0.175 | 47.01 | 0.81 |
ATGV-Net | 44.52 | 0.542 | 54.63 | 0.084 | 50.30 | 0.238 | 48.61 | 0.405 | 52.26 | 0.211 | 47.56 | 0.244 | 38.84 | 2.670 | 44.70 | 31.65 | 4.697 | 58.05 | 0.041 | 47.11 | 1.09 | |
DJFR | 36.73 | 3.109 | 44.27 | 0.779 | 45.38 | 1.019 | 40.99 | 1.601 | 46.07 | 0.978 | 39.91 | 1.392 | 37.11 | 4.570 | 39.22 | 3.540 | 29.68 | 8.252 | 43.32 | 0.993 | 40.27 | 2.62 |
JID | 42.29 | 1.238 | 49.73 | 0.417 | 48.56 | 0.610 | 45.85 | 0.867 | 49.51 | 0.576 | 44.93 | 0.690 | 39.03 | 3.157 | 43.18 | 2.508 | 32.66 | 4.699 | 49.38 | 0.427 | 44.51 | 1.52 |
MSG-Net | 45.23 | 0.550 | 54.94 | 0.089 | 50.85 | 0.206 | 50.21 | 0.256 | 52.86 | 0.192 | 48.27 | 0.260 | 39.85 | 2.277 | 44.69 | 1.581 | 35.48 | 4.350 | 57.34 | 0.046 | 47.97 | 0.98 |
DepthSR-Net | 46.42 | 52.35 | 0.056 | 49.97 | 50.22 | 51.54 | 0.157 | 48.40 | 40.77 | 1.249 | 45.37 | 1.129 | 52.01 | 0.014 | 47.48 | 0.45 | ||||||
RYNet | 0.236 | 52.66 | 0.160 | 0.138 | 1.819 | 37.73 | 1.286 | 60.28 | 0.52 | |||||||||||||
Proposed | 48.88 | 0.219 | 57.13 | 0.047 | 0.150 | 52.96 | 0.138 | 54.30 | 0.146 | 51.63 | 0.136 | 45.17 | 0.935 | 45.72 | 1.966 | 40.01 | 1.115 | 0.014 | 50.82 | |||
8倍 | ||||||||||||||||||||||
Bicubic | 33.39 | 10.534 | 40.79 | 3.098 | 42.75 | 3.075 | 37.38 | 5.600 | 42.23 | 3.610 | 36.14 | 4.967 | 33.29 | 12.932 | 35.99 | 10.268 | 26.31 | 20.858 | 39.35 | 4.454 | 36.76 | 7.94 |
RCAN | 38.67 | 2.476 | 46.59 | 0.681 | 45.86 | 0.955 | 42.98 | 1.366 | 46.88 | 0.912 | 41.40 | 1.189 | 35.38 | 5.823 | 38.77 | 5.798 | 30.30 | 8.041 | 44.61 | 0.941 | 41.14 | 2.82 |
SAN | 39.21 | 1.637 | 48.26 | 0.425 | 45.71 | 0.841 | 44.23 | 1.008 | 47.08 | 0.752 | 42.36 | 0.784 | 35.58 | 5.010 | 39.01 | 5.377 | 30.59 | 6.432 | 45.31 | 0.574 | 41.73 | 2.28 |
ATGV-Net | 38.15 | 2.829 | 47.50 | 0.630 | 45.65 | 0.990 | 42.71 | 2.139 | 47.18 | 0.850 | 41.86 | 1.092 | 33.90 | 7.339 | 39.01 | 5.520 | 26.87 | 12.184 | 47.27 | 0.406 | 41.01 | 3.40 |
DJFR | 34.43 | 5.463 | 41.48 | 1.544 | 43.27 | 1.976 | 38.45 | 2.928 | 42.54 | 2.132 | 37.12 | 2.559 | 32.39 | 7.674 | 37.27 | 6.077 | 25.11 | 14.445 | 41.22 | 1.843 | 37.33 | 4.66 |
JID | 39.27 | 1.794 | 47.81 | 0.602 | 46.16 | 0.992 | 42.74 | 1.483 | 46.61 | 1.003 | 41.16 | 1.185 | 35.04 | 5.403 | 39.53 | 4.984 | 28.42 | 8.064 | 46.35 | 0.622 | 41.31 | 2.61 |
MSG-Net | 40.45 | 1.411 | 49.24 | 0.356 | 47.34 | 0.609 | 44.53 | 1.044 | 48.70 | 0.508 | 43.24 | 0.744 | 35.61 | 5.625 | 39.31 | 4.550 | 29.62 | 10.059 | 47.79 | 0.294 | 42.58 | 2.52 |
DepthSR-Net | 41.02 | 0.892 | 49.03 | 47.21 | 0.469 | 45.81 | 0.648 | 48.47 | 0.417 | 44.23 | 0.420 | 3.797 | 5.990 | 47.26 | 0.194 | 43.19 | ||||||
RYNet | 0.215 | 48.40 | 0.441 | 37.68 | 3.226 | 40.97 | 3.665 | 31.44 | 1.48 | |||||||||||||
Proposed | 42.58 | 0.696 | 52.09 | 0.166 | 47.55 | 0.495 | 50.27 | 0.358 | 45.30 | 0.351 | 37.09 | 40.03 | 4.606 | 32.22 | 3.846 | 52.82 | 0.113 | 44.81 | 1.48 |
表1 无噪声深度图像超分辨率重建结果的PSNR(dB)和PE(%)比较
方法 | Art | Book | Dolls | Laundry | Moebius | Reindeer | Cones | Teddy | Tsukuba | Venus | 平均 | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
PSNR | PE | PSNR | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | |
2倍 | ||||||||||||||||||||||
Bicubic | 39.74 | 2.750 | 47.77 | 0.687 | 48.96 | 0.709 | 44.03 | 1.305 | 49.32 | 0.754 | 42.45 | 1.309 | 40.14 | 3.649 | 42.40 | 2.613 | 32.82 | 6.884 | 45.84 | 1.044 | 43.35 | 2.170 |
RCAN | 55.35 | 0.084 | 57.11 | 0.023 | 55.01 | 0.070 | 56.10 | 0.058 | 55.99 | 0.073 | 55.94 | 0.048 | 51.31 | 0.271 | 51.25 | 0.284 | 0.269 | 55.92 | 0.010 | 54.56 | 0.119 | |
SAN | 55.10 | 0.094 | 57.19 | 0.023 | 55.00 | 0.073 | 56.07 | 0.061 | 55.89 | 0.074 | 55.90 | 0.050 | 51.19 | 0.254 | 51.09 | 0.299 | 53.66 | 55.32 | 0.014 | 54.64 | 0.120 | |
ATGV-Net | 54.11 | 0.108 | 60.29 | 0.023 | 57.04 | 0.054 | 56.10 | 0.073 | 58.46 | 0.055 | 55.59 | 0.061 | 46.77 | 0.643 | 49.97 | 0.482 | 39.65 | 1.488 | 46.41 | 2.336 | 52.44 | 0.532 |
JID | 46.06 | 0.685 | 51.62 | 0.262 | 50.58 | 0.355 | 50.36 | 0.398 | 52.30 | 0.301 | 47.55 | 0.411 | 44.21 | 1.500 | 46.56 | 1.165 | 37.22 | 2.386 | 52.19 | 0.249 | 47.87 | 0.771 |
MSG-Net | 53.16 | 0.148 | 59.94 | 0.025 | 56.70 | 0.043 | 56.75 | 0.061 | 58.17 | 0.059 | 55.59 | 0.069 | 49.00 | 0.611 | 51.07 | 0.384 | 42.80 | 1.247 | 65.08 | 54.83 | 0.265 | |
DepthSR-Net | 53.64 | 0.050 | 55.54 | 54.27 | 0.024 | 55.23 | 0.032 | 55.29 | 53.86 | 0.033 | 50.15 | 49.78 | 0.276 | 45.46 | 0.336 | 54.06 | 0.001 | 52.73 | 0.096 | |||
RYNet | 63.09 | 0.008 | 59.45 | 0.026 | 61.23 | 0.028 | 54.10 | 0.175 | 47.49 | 0.312 | 65.58 | 0.001 | ||||||||||
Proposed | 60.33 | 0.037 | 63.05 | 0.008 | 0.026 | 60.73 | 60.14 | 0.025 | 0.167 | 52.91 | 0.255 | 51.16 | 0.233 | 0.001 | 58.67 | 0.081 | ||||||
4倍 | ||||||||||||||||||||||
Bicubic | 36.38 | 5.507 | 44.05 | 1.465 | 45.80 | 1.488 | 40.52 | 2.755 | 45.63 | 1.658 | 39.19 | 2.546 | 36.45 | 7.207 | 39.04 | 5.276 | 29.47 | 13.052 | 42.52 | 2.200 | 39.91 | 4.32 |
RCAN | 45.19 | 0.470 | 52.80 | 0.125 | 49.35 | 0.315 | 49.45 | 0.308 | 50.93 | 0.295 | 48.00 | 0.265 | 41.61 | 1.690 | 44.09 | 2.204 | 37.15 | 2.031 | 51.28 | 0.138 | 46.99 | 0.78 |
SAN | 45.06 | 0.565 | 52.90 | 0.128 | 49.70 | 0.310 | 49.47 | 0.317 | 51.25 | 0.288 | 47.91 | 0.291 | 41.57 | 1.804 | 44.38 | 1.939 | 36.79 | 2.236 | 51.06 | 0.175 | 47.01 | 0.81 |
ATGV-Net | 44.52 | 0.542 | 54.63 | 0.084 | 50.30 | 0.238 | 48.61 | 0.405 | 52.26 | 0.211 | 47.56 | 0.244 | 38.84 | 2.670 | 44.70 | 31.65 | 4.697 | 58.05 | 0.041 | 47.11 | 1.09 | |
DJFR | 36.73 | 3.109 | 44.27 | 0.779 | 45.38 | 1.019 | 40.99 | 1.601 | 46.07 | 0.978 | 39.91 | 1.392 | 37.11 | 4.570 | 39.22 | 3.540 | 29.68 | 8.252 | 43.32 | 0.993 | 40.27 | 2.62 |
JID | 42.29 | 1.238 | 49.73 | 0.417 | 48.56 | 0.610 | 45.85 | 0.867 | 49.51 | 0.576 | 44.93 | 0.690 | 39.03 | 3.157 | 43.18 | 2.508 | 32.66 | 4.699 | 49.38 | 0.427 | 44.51 | 1.52 |
MSG-Net | 45.23 | 0.550 | 54.94 | 0.089 | 50.85 | 0.206 | 50.21 | 0.256 | 52.86 | 0.192 | 48.27 | 0.260 | 39.85 | 2.277 | 44.69 | 1.581 | 35.48 | 4.350 | 57.34 | 0.046 | 47.97 | 0.98 |
DepthSR-Net | 46.42 | 52.35 | 0.056 | 49.97 | 50.22 | 51.54 | 0.157 | 48.40 | 40.77 | 1.249 | 45.37 | 1.129 | 52.01 | 0.014 | 47.48 | 0.45 | ||||||
RYNet | 0.236 | 52.66 | 0.160 | 0.138 | 1.819 | 37.73 | 1.286 | 60.28 | 0.52 | |||||||||||||
Proposed | 48.88 | 0.219 | 57.13 | 0.047 | 0.150 | 52.96 | 0.138 | 54.30 | 0.146 | 51.63 | 0.136 | 45.17 | 0.935 | 45.72 | 1.966 | 40.01 | 1.115 | 0.014 | 50.82 | |||
8倍 | ||||||||||||||||||||||
Bicubic | 33.39 | 10.534 | 40.79 | 3.098 | 42.75 | 3.075 | 37.38 | 5.600 | 42.23 | 3.610 | 36.14 | 4.967 | 33.29 | 12.932 | 35.99 | 10.268 | 26.31 | 20.858 | 39.35 | 4.454 | 36.76 | 7.94 |
RCAN | 38.67 | 2.476 | 46.59 | 0.681 | 45.86 | 0.955 | 42.98 | 1.366 | 46.88 | 0.912 | 41.40 | 1.189 | 35.38 | 5.823 | 38.77 | 5.798 | 30.30 | 8.041 | 44.61 | 0.941 | 41.14 | 2.82 |
SAN | 39.21 | 1.637 | 48.26 | 0.425 | 45.71 | 0.841 | 44.23 | 1.008 | 47.08 | 0.752 | 42.36 | 0.784 | 35.58 | 5.010 | 39.01 | 5.377 | 30.59 | 6.432 | 45.31 | 0.574 | 41.73 | 2.28 |
ATGV-Net | 38.15 | 2.829 | 47.50 | 0.630 | 45.65 | 0.990 | 42.71 | 2.139 | 47.18 | 0.850 | 41.86 | 1.092 | 33.90 | 7.339 | 39.01 | 5.520 | 26.87 | 12.184 | 47.27 | 0.406 | 41.01 | 3.40 |
DJFR | 34.43 | 5.463 | 41.48 | 1.544 | 43.27 | 1.976 | 38.45 | 2.928 | 42.54 | 2.132 | 37.12 | 2.559 | 32.39 | 7.674 | 37.27 | 6.077 | 25.11 | 14.445 | 41.22 | 1.843 | 37.33 | 4.66 |
JID | 39.27 | 1.794 | 47.81 | 0.602 | 46.16 | 0.992 | 42.74 | 1.483 | 46.61 | 1.003 | 41.16 | 1.185 | 35.04 | 5.403 | 39.53 | 4.984 | 28.42 | 8.064 | 46.35 | 0.622 | 41.31 | 2.61 |
MSG-Net | 40.45 | 1.411 | 49.24 | 0.356 | 47.34 | 0.609 | 44.53 | 1.044 | 48.70 | 0.508 | 43.24 | 0.744 | 35.61 | 5.625 | 39.31 | 4.550 | 29.62 | 10.059 | 47.79 | 0.294 | 42.58 | 2.52 |
DepthSR-Net | 41.02 | 0.892 | 49.03 | 47.21 | 0.469 | 45.81 | 0.648 | 48.47 | 0.417 | 44.23 | 0.420 | 3.797 | 5.990 | 47.26 | 0.194 | 43.19 | ||||||
RYNet | 0.215 | 48.40 | 0.441 | 37.68 | 3.226 | 40.97 | 3.665 | 31.44 | 1.48 | |||||||||||||
Proposed | 42.58 | 0.696 | 52.09 | 0.166 | 47.55 | 0.495 | 50.27 | 0.358 | 45.30 | 0.351 | 37.09 | 40.03 | 4.606 | 32.22 | 3.846 | 52.82 | 0.113 | 44.81 | 1.48 |
方法 | Art | Book | Dolls | Laundry | Moebius | Reindeer | Cones | Teddy | Tsukuba | Venus | 平均 | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
PSNR | PE | PSNR | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | |
Bicubic | 31.42 | 9.075 | 34.57 | 3.995 | 34.96 | 3.813 | 33.52 | 5.093 | 34.92 | 3.896 | 32.93 | 5.264 | 31.36 | 9.493 | 32.97 | 7.455 | 25.85 | 15.353 | 34.01 | 5.336 | 32.65 | 6.877 |
RCAN | 35.75 | 2.229 | 42.54 | 0.540 | 40.55 | 0.954 | 39.12 | 1.424 | 41.55 | 0.906 | 38.71 | 1.015 | 32.40 | 5.565 | 35.94 | 3.718 | 27.99 | 8.457 | 41.12 | 0.803 | 37.57 | 2.561 |
SAN | 35.67 | 2.067 | 0.542 | 40.90 | 0.912 | 38.94 | 1.329 | 0.875 | 38.38 | 0.944 | 31.94 | 5.642 | 35.91 | 3.514 | 28.15 | 7.978 | 41.59 | 0.742 | 37.58 | 2.455 | ||
ATGV-Net | 35.01 | 2.876 | 40.70 | 0.714 | 40.19 | 0.875 | 37.96 | 1.199 | 40.35 | 0.960 | 37.84 | 1.145 | 32.78 | 5.540 | 35.84 | 3.810 | 26.70 | 9.325 | 39.57 | 0.979 | 36.69 | 2.742 |
MSG-Net | 35.46 | 2.393 | 40.89 | 0.584 | 40.24 | 0.819 | 38.29 | 1.091 | 39.84 | 1.000 | 37.89 | 1.281 | 32.88 | 5.468 | 36.02 | 3.389 | 27.15 | 9.580 | 40.02 | 0.750 | 36.87 | 2.636 |
DepthSR-Net | 36.17 | 41.77 | 41.35 | 27.50 | 7.502 | |||||||||||||||||
RYNet | 37.33 | 1.197 | 42.66 | 0.351 | 42.23 | 0.465 | 40.29 | 0.633 | 42.67 | 0.531 | 39.55 | 0.620 | 34.61 | 3.455 | 37.37 | 2.403 | 6.264 | 41.37 | 0.330 | 38.64 | 1.625 | |
Proposed | 1.998 | 42.21 | 0.535 | 40.83 | 0.812 | 39.20 | 0.984 | 41.46 | 0.847 | 38.68 | 0.916 | 32.99 | 4.997 | 36.27 | 3.261 | 28.80 | 40.58 | 0.810 | 37.72 | 2.265 |
表2 有噪声深度图像8倍超分辨率重建结果的PSNR(dB)和PE(%)比较
方法 | Art | Book | Dolls | Laundry | Moebius | Reindeer | Cones | Teddy | Tsukuba | Venus | 平均 | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
PSNR | PE | PSNR | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | PSRN | PE | |
Bicubic | 31.42 | 9.075 | 34.57 | 3.995 | 34.96 | 3.813 | 33.52 | 5.093 | 34.92 | 3.896 | 32.93 | 5.264 | 31.36 | 9.493 | 32.97 | 7.455 | 25.85 | 15.353 | 34.01 | 5.336 | 32.65 | 6.877 |
RCAN | 35.75 | 2.229 | 42.54 | 0.540 | 40.55 | 0.954 | 39.12 | 1.424 | 41.55 | 0.906 | 38.71 | 1.015 | 32.40 | 5.565 | 35.94 | 3.718 | 27.99 | 8.457 | 41.12 | 0.803 | 37.57 | 2.561 |
SAN | 35.67 | 2.067 | 0.542 | 40.90 | 0.912 | 38.94 | 1.329 | 0.875 | 38.38 | 0.944 | 31.94 | 5.642 | 35.91 | 3.514 | 28.15 | 7.978 | 41.59 | 0.742 | 37.58 | 2.455 | ||
ATGV-Net | 35.01 | 2.876 | 40.70 | 0.714 | 40.19 | 0.875 | 37.96 | 1.199 | 40.35 | 0.960 | 37.84 | 1.145 | 32.78 | 5.540 | 35.84 | 3.810 | 26.70 | 9.325 | 39.57 | 0.979 | 36.69 | 2.742 |
MSG-Net | 35.46 | 2.393 | 40.89 | 0.584 | 40.24 | 0.819 | 38.29 | 1.091 | 39.84 | 1.000 | 37.89 | 1.281 | 32.88 | 5.468 | 36.02 | 3.389 | 27.15 | 9.580 | 40.02 | 0.750 | 36.87 | 2.636 |
DepthSR-Net | 36.17 | 41.77 | 41.35 | 27.50 | 7.502 | |||||||||||||||||
RYNet | 37.33 | 1.197 | 42.66 | 0.351 | 42.23 | 0.465 | 40.29 | 0.633 | 42.67 | 0.531 | 39.55 | 0.620 | 34.61 | 3.455 | 37.37 | 2.403 | 6.264 | 41.37 | 0.330 | 38.64 | 1.625 | |
Proposed | 1.998 | 42.21 | 0.535 | 40.83 | 0.812 | 39.20 | 0.984 | 41.46 | 0.847 | 38.68 | 0.916 | 32.99 | 4.997 | 36.27 | 3.261 | 28.80 | 40.58 | 0.810 | 37.72 | 2.265 |
方法 | Middlebury | ToFMark | NYU_Depth | 平均 | ||||
---|---|---|---|---|---|---|---|---|
PSNR | PE | PSNR | PE | PSRN | PE | PSRN | PE | |
Bicubic | 37.27 | 3.516 | 40.00 | 3.204 | 41.44 | 2.643 | 41.27 | 2.68 |
RCAN | 47.34 | 0.575 | 45.43 | 0.656 | 45.84 | 1.023 | 45.90 | 1.00 |
SAN | 47.39 | 0.606 | 45.58 | 0.694 | 46.10 | 0.986 | 46.15 | 0.97 |
ATGV-Net | 47.34 | 0.787 | 45.23 | 0.677 | 46.31 | 1.050 | 46.35 | 1.04 |
MSG-Net | 48.07 | 0.691 | 45.47 | 0.735 | 46.18 | 1.018 | 46.25 | 1.00 |
DepthSR-Net | 47.84 | 0.369 | 45.59 | 45.88 | 0.830 | 45.96 | 0.81 | |
RYNet | 0.554 | 48.02 | 0.694 | 48.10 | 0.68 | |||
Proposed | 50.89 | 0.342 | 47.15 | 0.525 |
表3 三个深度图像集重建结果的PSNR(dB)和PE(%)比较(4倍)
方法 | Middlebury | ToFMark | NYU_Depth | 平均 | ||||
---|---|---|---|---|---|---|---|---|
PSNR | PE | PSNR | PE | PSRN | PE | PSRN | PE | |
Bicubic | 37.27 | 3.516 | 40.00 | 3.204 | 41.44 | 2.643 | 41.27 | 2.68 |
RCAN | 47.34 | 0.575 | 45.43 | 0.656 | 45.84 | 1.023 | 45.90 | 1.00 |
SAN | 47.39 | 0.606 | 45.58 | 0.694 | 46.10 | 0.986 | 46.15 | 0.97 |
ATGV-Net | 47.34 | 0.787 | 45.23 | 0.677 | 46.31 | 1.050 | 46.35 | 1.04 |
MSG-Net | 48.07 | 0.691 | 45.47 | 0.735 | 46.18 | 1.018 | 46.25 | 1.00 |
DepthSR-Net | 47.84 | 0.369 | 45.59 | 45.88 | 0.830 | 45.96 | 0.81 | |
RYNet | 0.554 | 48.02 | 0.694 | 48.10 | 0.68 | |||
Proposed | 50.89 | 0.342 | 47.15 | 0.525 |
结构 | C1 | C2 | C3 | C4 | C5 |
---|---|---|---|---|---|
高阶跨尺度注意力模块 | × | √ | × | √ | √ |
双层残差块 | × | × | √ | √ | √ |
深监督 | × | × | × | × | √ |
平均PSNR | 41.14 | 43.15 | 43.37 | 43.73 | 44.81 |
平均PE | 2.82 | 2.06 | 1.99 | 1.80 | 1.48 |
表4 消融研究(8倍)
结构 | C1 | C2 | C3 | C4 | C5 |
---|---|---|---|---|---|
高阶跨尺度注意力模块 | × | √ | × | √ | √ |
双层残差块 | × | × | √ | √ | √ |
深监督 | × | × | × | × | √ |
平均PSNR | 41.14 | 43.15 | 43.37 | 43.73 | 44.81 |
平均PE | 2.82 | 2.06 | 1.99 | 1.80 | 1.48 |
方法 | RCAN | SAN | ATGV-Net | DJFR | JID |
---|---|---|---|---|---|
时间 | 0.45 | 4.99 | 0.39 | 5.62 | 869.60 |
显存开销 | 16 | 15.7 | — | 0.5 | — |
方法 | MSG-Net | DepthSR-Net | RYNet | Proposed | — |
时间 | 0.37 | 1.87 | 0.71 | 1.17 | — |
显存开销 | 0.8 | 43.3 | 62.7 | 20.5 | — |
表5 深度图像重建的平均运行时间(秒)和显存开销(MB)比较
方法 | RCAN | SAN | ATGV-Net | DJFR | JID |
---|---|---|---|---|---|
时间 | 0.45 | 4.99 | 0.39 | 5.62 | 869.60 |
显存开销 | 16 | 15.7 | — | 0.5 | — |
方法 | MSG-Net | DepthSR-Net | RYNet | Proposed | — |
时间 | 0.37 | 1.87 | 0.71 | 1.17 | — |
显存开销 | 0.8 | 43.3 | 62.7 | 20.5 | — |
1 | HORNACEK M, RHEMANN C, GELAUTZ M, et al. Depth super resolution by rigid body self-similarity in 3d[C]//IEEE Conference on Computer Vision and Pattern Recognition. Portland: IEEE, 2013: 1123-1130. |
2 | LEI J, LI L, YUE H, et al. Depth map super- resolution considering view synthesis quality[J]. IEEE Transactions on Image Processing, 2017, 26(4): 1732-1745. |
3 | MAC A O, CAMPBELL N D F, NAIR A, et al. Patch based synthesis for single depth image super-resolution[C]//European Conference on Computer Vision. Berlin: Springer, 2012: 71-84. |
4 | XIE J, FERIS R S, SUN M T. Edge-guided single depth image super resolution[J]. IEEE Transactions on Image Processing, 2015, 25(1): 428-438. |
5 | XIE J, FERIS R S, YU S S, et al. Joint super resolution and denoising from a single depth image[J]. IEEE Transactions on Multimedia, 2015, 17(9): 1525-1537. |
6 | MANDAL S, BHAVSAR A, SAO A K. Depth map restoration from undersampled data[J]. IEEE Transactions on Image Processing, 2016, 26(1): 119-134. |
7 | RIEGLER G, et al. Atgv-net: Accurate depth super-resolution[C]//European Conference on Computer Vision. Cham: Springer, 2016: 268-284. |
8 | SONG X, DAI Y, QIN X. Deeply supervised depth map super-resolution as novel view synthesis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 29(8): 2323-2336. |
9 | HUANG L, ZHANG J, ZUO Y, et al. Pyramid-structured depth MAP super-resolution based on deep dense-residual network[J]. IEEE Signal Processing Letters, 2019, 26(12): 1723-1727. |
10 | SONG X, DAI Y, ZHOU D, et al. Channel attention based iterative residual learning for depth map super-resolution[C]//IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 5631-5640. |
11 | HUI T W, LOY C C, TANG X. Depth map super-resolution by deep multi-scale guidance[C]//European Conference on Computer Vision. Cham: Springer, 2016: 353-369. |
12 | KIECHLE M, HAWE S, KLEINSTEUBER M. A joint intensity and depth cosparse analysis model for depth map super-resolution[C]//IEEE International Conference on Computer Vision. Piscataway: IEEE, 2013: 1545-1552. |
13 | GUO C, LI C, GUO J, et al. Hierarchical features driven residual learning for depth map super-resolution[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2545-2557. |
14 | ZUO Y, WU Q, FANG Y, et al. Multi-scale frequency reconstruction for guided depth map super-resolution via deep residual network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(2): 297-306. |
15 | LI Y, HUANG J B, AHUJA N, YANG M H. Joint image filtering with deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(8): 1909-1923. |
16 | LI T, LIN H, DONG X, et al. Depth image super-resolution using correlation-controlled color guidance and multi-scale symmetric network[J]. Pattern Recognition, 2020, 107: 107513. |
17 | LI T, DONG X, LIN H. Guided depth map super-resolution using recumbent Y network[J]. IEEE Access, 2020, 8: 122695-122708. |
18 | ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//European Conference on Computer Vision. Munich: Springer, 2018: 286-301. |
19 | DAI T, CAI J, ZHANG Y, et al. Second-order attention network for single image super-resolution[C]//IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 11065-11074. |
20 | HU Y, LI J, HUANG Y, et al. Channel-wise and spatial feature modulation network for single image super-resolution[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(11): 3911-3927. |
21 | ZHANG Y, LI K, LI K, et al. Residual non-local attention networks for image restoration[C]//International Conference on Learning Representations. New Orleans: Brown Walker Press, 2019. |
22 | MEI Y, FAN Y, ZHOU Y, et al. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining[C]//IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 5690-5699. |
23 | NAJIBI M, SAMANGOUEI P, CHELLAPPA R, et al. SSH: Single stage headless face detector[C]//IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 4875-4884. |
24 | LIU Y, CHENG M M, HU X, et al. Richer convolutional features for edge detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(8): 1939-1946. |
25 | CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//European Conference on Computer Vision. Munich: Springer, 2018: 801-818. |
26 | 李雅倩, 盖成远, 肖存军, 等. 基于细化多尺度深度特征的目标检测网络[J]. 电子学报, 2020, 48(12): 2360-2366. |
LI Y, GAI C, XIAO C, et al. Object detection networks based on refined multi-scale depth feature [J]. Acta Electronica Sinica, 2020, 48(12): 2360-2366. (in Chinese) | |
27 | ZHAO H, SHI J, QI X, et al. Pyramid scene parsing network[C]//IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2881-2890. |
28 | CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848. |
29 | SHI W, CABALLERO J, HUSZAR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 1874-1883. |
30 | HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition.Piscataway: IEEE, 2016: 770-778. |
31 | FAN Y, YU J, HUANG T S. Wide-activated deep residual networks based restoration for BPG-compressed images[C]//IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 2621-2624. |
32 | JIN T, HUANG S, LI Y, et al. Low-rank HOCA: Efficient high-order cross-modal attention for video captioning[C]//2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong: ACL, 2019: 2001-2011. |
33 | Middlebury. The Middlebury Stereo Datasets[DB/OL]. (2015). . |
[1] | 李钦, 刘伟, 牛朝阳, 宝音图, 惠周勃. 低信噪比下基于分裂EfficientNet网络的雷达信号调制方式识别[J]. 电子学报, 2023, 51(3): 675-686. |
[2] | 范兵兵, 何庭建, 张聪炫, 陈震, 黎明. 联合遮挡约束与残差补偿的特征金字塔光流计算方法[J]. 电子学报, 2023, 51(3): 648-657. |
[3] | 张聿远, 闫文君, 张立民. 基于多模态特征融合网络的空时分组码识别算法[J]. 电子学报, 2023, 51(2): 489-498. |
[4] | 许新征, 李杉. 基于特征膨胀卷积模块的轻量化技术研究[J]. 电子学报, 2023, 51(2): 355-364. |
[5] | 张永梅, 孙捷. 基于动静态特征双输入神经网络的咳嗽声诊断COVID-19算法[J]. 电子学报, 2023, 51(1): 202-212. |
[6] | 袁海英, 成君鹏, 曾智勇, 武延瑞. Mobile_BLNet:基于Big-Little Net的轻量级卷积神经网络优化设计[J]. 电子学报, 2023, 51(1): 180-191. |
[7] | 王神龙, 雍宇, 吴晨睿. 基于伪孪生神经网络的低纹理工业零件6D位姿估计[J]. 电子学报, 2023, 51(1): 192-201. |
[8] | 郭晓轩, 冯其波, 冀振燕, 郑发家, 杨燕燕. 多线激光光条图像缺陷分割模型研究[J]. 电子学报, 2023, 51(1): 172-179. |
[9] | 贾童瑶, 卓力, 李嘉锋, 张菁. 基于深度学习的单幅图像去雾研究进展[J]. 电子学报, 2023, 51(1): 231-245. |
[10] | 何滢婕, 刘月峰, 边浩东, 郭威, 张小燕. 基于Informer的电池荷电状态估算及其稀疏优化方法[J]. 电子学报, 2023, 51(1): 50-56. |
[11] | 吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294. |
[12] | 李雪莹, 王田路, 梁鹏, 王翀. 基于系统模型的用户评论中非功能需求的自动分类[J]. 电子学报, 2022, 50(9): 2079-2089. |
[13] | 琚长瑞, 秦晓燕, 袁广林, 李豪, 朱虹. 尺度敏感损失与特征融合的快速小目标检测方法[J]. 电子学报, 2022, 50(9): 2119-2126. |
[14] | 张志昌, 于沛霖, 庞雅丽, 朱林, 曾扬扬. SMGN:用于对话状态跟踪的状态记忆图网络[J]. 电子学报, 2022, 50(8): 1851-1858. |
[15] | 张亚洲, 俞洋, 朱少林, 陈锐, 戎璐, 梁辉. 一种量子概率启发的对话讽刺识别网络模型[J]. 电子学报, 2022, 50(8): 1885-1893. |
阅读次数 | ||||||
全文 |
|
|||||
摘要 |
|
|||||