信号的语义刻画与度量

doi:10.12263/DZXB.20210238

电子学报 ›› 2022, Vol. 50 ›› Issue (9): 2068-2078.DOI: 10.12263/DZXB.20210238

信号的语义刻画与度量

石光明¹^,², 高大化¹^,², 杨旻曦¹, 谢雪梅¹, 董明皓³, 李雷达¹, 于凯¹

^1.西安电子科技大学人工智能学院, 陕西西安 710071
^2.鹏城实验室, 广东深圳 518055
^3.西安电子科技大学生命科学技术学院, 陕西西安 710071

收稿日期:2021-02-09 修回日期:2022-02-02 出版日期:2022-09-25
- 作者简介:
- 石光明　男， 1965年06月出生于江西省南昌市. 长江学者特聘教授， IEEE Fellow， IET Fellow. 现任西安电子科技大学人工智能学院教授. 主要研究方向为人工智能、语义通信.E-mail: gmshi@xidian.edu.cn
  高大化　男， 1979年08月出生于河南省开封市. 现任西安电子科技大学人工智能学院教授. 主要研究方向为智能信息处理、智能感知. E-mail: dhgao@xidian.edu.cn
  杨旻曦　男， 1996年10月出生于四川省成都市. 现为西安电子科技大学人工智能学院博士研究生. 主要研究方向为表征学习、语义通信.E-mail: mxyang@stu.xidian.edu.cn
  谢雪梅　女， 1967年01月出生于陕西省西安市. 现任西安电子科技大学人工智能学院教授. 主要研究方向为场景理解与视频分析、多模态融合.
  董明皓　男， 1984年05月出生于陕西省西安市. 现为西安电子科技大学分子与神经影像教育部工程研究中心副教授. 主要研究方向为脑机混合智能、人体效能增强.E-mail: dminghao@xidian.edu.cn
  李雷达　男， 1982年10月出生于江苏省徐州市. 现任西安电子科技大学人工智能学院教授. 主要研究方向为图像感知质量评价.E-mail: ldli@xidian.edu.cn
  于凯　男，1996年11月出生于山东省日照市莒县. 2018至2021年于西安电子科技大学人工智能学院攻读硕士学位. 主要研究方向为图像识别、增量学习.E-mail: yukai_nathan@163.com
- 基金资助:
- 国家重点研发计划课题 (2019YFA0706604); 国家自然科学基金 (61871304); 鹏城实验室重大攻关项目 (PCL2021A12)

Semantic Characterization and Measurement of Signals

SHI Guang-ming¹^,², GAO Da-hua¹^,², YANG Min-xi¹, XIE Xue-mei¹, DONG Ming-hao³, LI Lei-da¹, YU Kai¹

^1.School of Artificial Intelligence，Xidian University，Xi’an，Shaanxi 710071，China
^2.Peng Cheng Laboratory，Shenzhen，Guangdong 518055，China
^3.School of Life Science and Technology，Xidian University，Xi’an，Shaanxi 710071，China

Received:2021-02-09 Revised:2022-02-02 Online:2022-09-25 Published:2022-10-26

摘要/Abstract

摘要：

相比于基于比特数据的信息处理及通信技术，人类通过语义处理和传递信息的方式，在面对智能体间传递处理海量信息这一问题时显得更为高效和自然.然而由于目前缺乏关于语义度量和刻画的数学描述，涉及语义的应用无法兼顾可解释性和泛化性，无法发挥语义的高效自然的优势.本文围绕语义的度量和刻画，首先依据信息科学和神经科学相关结论，讨论了语义的内涵，并指出语义具有模块化、多模态、层级化的特点；接着提出了一种多模态信号的语义刻画和度量的数学描述；然后为了验证所提信号语义的刻画和度量的可行性和有效性，在MNIST（Mixed National Institute of Standards and Technology database）手写数字识别和水声目标识别两个应用中进行了实验，获得比传统深度学习更好的性能；最后将语义用于视频编码，实现了远超传统方法的压缩比，展现了语义在通信领域的实用价值.这为未来建立以语义为基础的新型信息处理与通信技术奠定了理论和实践基础.

关键词: 语义, 语义刻画, 语义度量, 语义基元, 语义计算, 语义识别

Abstract:

Compared with the modern information processing and communication technology based on bits, the semantic-based way of processing and transmitting information used by humans is more efficient and natural in the face of the massive information that needs to be sent between agents. However, due to the lack of mathematical description of semantic measurement and characterization, semantics' applications cannot consider both interpretability and generalization. Therefore, they cannot give play to the advantages of efficient and natural semantics. This paper focuses on the measurement and characterization of semantics. Firstly, we discuss the connotation of semantics based on the relevant conclusions of information science and neuroscience, and concludes that semantics has the characteristics of modularity, multi-mode, and hierarchy. Then, a semantic description of multimodal signals and a mathematical description of their measurement are proposed. Next, to verify the feasibility and effectiveness of the characterization and measurement of the proposed signal semantics, experiments are carried out in two applications: MNIST (Mixed National Institute of Standards and Technology database) handwritten digital recognition and underwater acoustic target recognition, and the results are better than those of the traditional deep learning. Finally, the semantics is used for video coding, and a compression ratio far exceeding that of traditional methods is achieved. This lays a theoretical and practical foundation for establishing new information processing and communication technology based on semantics.

Key words: semantic, semantic characterization, semantic measurement, semantic primitive, semantic computing, semantic recognition

中图分类号:

TP391.4

石光明, 高大化, 杨旻曦, 谢雪梅, 董明皓, 李雷达, 于凯. 信号的语义刻画与度量[J]. 电子学报, 2022, 50(9): 2068-2078.

SHI Guang-ming, GAO Da-hua, YANG Min-xi, XIE Xue-mei, DONG Ming-hao, LI Lei-da, YU Kai. Semantic Characterization and Measurement of Signals[J]. Acta Electronica Sinica, 2022, 50(9): 2068-2078.

图/表 10

参考文献 40

1	KIFER M, LAUSEN G. F-logic: a higher-order language for reasoning about objects, inheritance, and scheme[J]. ACM SIGMOD Record, 1989, 18(2): 134-146.
2	BAADER F. The Description Logic Handbook: Theory, Implementation, and Applications[M]. Cambridge, UK: Cambridge University Press, 2003
3	BOLLACKER K, COOK R, TUFTS P. Freebase: a shared database of structured general human knowledge[C]//AAAI'07: Proceedings of the 22nd national conference on Artificial intelligence - Volume 2. Menlo Park, CA: AAAI Press, 2007: 1962-1963.
4	LI Z Y, DING X, LIU T. Constructing narrative event evolutionary graph for script event prediction[C]//IJCAI'18: Proceedings of the 27th International Joint Conference on Artificial Intelligence. Menlo Park, CA: AAAI Press, 2018: 4201-4207.
5	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. (2016-09-09). .
6	VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph attention networks[EB/OL]. (2017-10-30). .
7	LAFFERTY J D, MCCALLUM A, PEREIRA F C N. Conditional random fields: Probabilistic models for segmenting and labeling sequence data[C]//ICML'01: Proceedings of the Eighteenth International Conference on Machine Learning. San Francisco: Morgan Kaufmann Publishers Inc, 2001: 282-289.
8	TERENT'YEV A N, BIDYUK P I. Method of probabilistic inference from learning data in Bayesian networks[J].Cybernetics and Systems Analysis, 2007, 43(3): 391-396.
9	MIKOLOV T, KARAFIÁT M, BURGET L, et al. Recurrent neural network based language model[C]//Interspeech 2010. Chiba: ISCA, 2010: 1045-1048.
10	HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
11	GOLDBERG Y, LEVY O. word2vec Explained: deriving Mikolovet al.' s negative-sampling word-embedding method[EB/OL].(2014-02-15). .
12	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all You need[C]//NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2017: 6000-6010.
13	DEVLIN J, CHANG M-W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018-10-11). .
14	LOWE D G. Object recognition from local scale-invariant features[C]//Proceedings of the Seventh IEEE International Conference on Computer Vision. Piscataway: IEEE, 1999: 1150-1157.
15	FELZENSZWALB P F, GIRSHICK R B, MCALLESTER D, et al. Object detection with discriminatively trained part-based models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(9): 1627-1645.
16	KOH P W, NGUYEN T, TANG Y S, et al. Concept bottleneck models[EB/OL]. (2020-07-07).
17	LU C W, KRISHNA R, BERNSTEIN M, et al. Visual relationship detection with language priors[C]//Computer Vision–ECCV 2016. Cham: Springer International Publishing, 2016: 852-869.
18	WANG X L, YE Y F, GUPTA A. Zero-shot recognition via semantic embeddings and knowledge graphs[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 6857-6866.
19	CARON M, BOJANOWSKI P, JOULIN A, et al. Deep Clustering for Unsupervised Learning of Visual Features[C]//Computer Vision–ECCV 2018. Cham: Springer International Publishing, 2018: 139-156.
20	LI O, LIU H, CHEN C, et al. Deep learning for case-based reasoning through prototypes: a neural network that explains its predictions[C] // Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. Menlo Park, CA: AAAI Press, 2018: 3530-3537.
21	GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2014: 580-587.
22	CHUTE C G. Classification and retrieval of patient records using natural language: An experimental application of latent semantic analysis[C]//Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society Volume 13. Piscataway: IEEE, 1991: 1162-1163.
23	TURNEY P D, PANTEL P. From frequency to meaning: Vector space models of semantics[J]. Journal of Artificial Intelligence Research, 2010, 37: 141-188.
24	MATSUNO K. Semantic commitments as a mode of non-programmable computation in the brain[J]. Bio Systems, 1992, 27(4): 235-239.
25	SHANNON C E. A mathematical theory of communication[J]. The Bell System Technical Journal, 1948, 27(3): 379-423.
26	SHANNON C E, WEAVER W. The Mathematical Theory of Communication[M]. Urbana: University of Illinois Press, 1949
27	GÜLER B, YENER A, SWAMI A. The semantic communication game[J]. IEEE Transactions on Cognitive Communications and Networking, 2018, 4(4): 787-802.
28	BAO J, BASU P, DEAN M K, et al. Towards a theory of semantic communication[C]//2011 IEEE Network Science Workshop. Piscataway: IEEE, 2011: 110-117.
29	BASU P, BAO J, DEAN M, et al. Preserving quality of information by using semantic relationships[J]. Pervasive and Mobile Computing, 2014, 11: 188-202.
30	WILLEMS F M J, KALKER T. Semantic compaction, transmission, and compression codes[C]//Proceedings of International Symposium on Information Theory, 2005 ISIT. Piscataway: IEEE, 2005: 214-218.
31	HUBEL D H, WIESEL T N. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[J]. The Journal of Physiology, 1962, 160(1): 106-154.
32	HUTH A G, NISHIMOTO S, VU A T, et al. A continuous semantic space describes the representation of thousands of object and action categories across the human brain[J]. Neuron, 2012, 76(6): 1210-1224.
33	HANDJARAS G, RICCIARDI E, LEO A, et al. How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge[J]. NeuroImage, 2016, 135: 232-242.
34	DEFELIPE J, MARKRAM H, ROCKLAND K S. The neocortical column[J]. Frontiers in Neuroanatomy, 2012, 6: 22.
35	MOUNTCASTLE V B. The columnar organization of the neocortex[J]. Brain: a Journal of Neurology, 1997, 120 ( Pt 4): 701-722.
36	ROCKLAND K S. Five points on columns[J]. Frontiers in Neuroanatomy, 2010, 4: 22.
37	BI Y C, WANG X Y, CARAMAZZA A. Object domain and modality in the ventral visual pathway[J]. Trends in Cognitive Sciences, 2016, 20(4): 282-290.
38	HUTH A G, DE HEER W A, GRIFFITHS T L, et al. Natural speech reveals the semantic maps that tile human cerebral cortex[J]. Nature, 2016, 532(7600): 453-458.
39	SHI G M, ZHANG Z Q, GAO D H, et al. Knowledge-guided semantic computing network[J]. Neurocomputing, 2021, 426: 70-84.
40	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.

数据集大小	卷积神经网络	本文方法
1-shot	45.35	65.93
10-shot	62.61	83.13
100-shot	72.08	92.85
1000-shot	87.53	97.41
6000-shot	96.19	98.08

数据集大小	卷积神经网络	本文方法
1-shot	45.35	65.93
10-shot	62.61	83.13
100-shot	72.08	92.85
1000-shot	87.53	97.41
6000-shot	96.19	98.08

语义基元	描述
叶频	对应于轴频与叶片数的乘积
叶片数	螺旋桨的固有属性,根据轴频和叶频计算获得
音调	声音的固有属性,与声音信号的周期密切相关
音色	声音的固有属性
LOFAR(Low Frequency Array) 谱基元	反映水声信号在时、频维度上的性质
通用声音基元	通过自然声音识别任务训练的VGG (Visual Geometry Group) 网络,描述声音的一般属性

语义基元	描述
叶频	对应于轴频与叶片数的乘积
叶片数	螺旋桨的固有属性,根据轴频和叶频计算获得
音调	声音的固有属性,与声音信号的周期密切相关
音色	声音的固有属性
LOFAR(Low Frequency Array) 谱基元	反映水声信号在时、频维度上的性质
通用声音基元	通过自然声音识别任务训练的VGG (Visual Geometry Group) 网络,描述声音的一般属性

压缩方法	原始视频	MPEG编码	初级语义编码	高级语义编码
视频大小/Kb	268800	5990.4	34.1	0.5
压缩率	1.00	0.0223	1.27e-4	1.86e-6

信号的语义刻画与度量

Semantic Characterization and Measurement of Signals

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 40

相关文章 15

编辑推荐

Metrics

[1]	仇韫琦, 王元卓, 白龙, 尹芷仪, 沈华伟, 白硕. 面向知识库问答的问句语义解析研究综述[J]. 电子学报, 2022, 50(9): 2242-2264.
[2]	张志文, 刘天歌, 聂鹏举. 基于实景数据增强和双路径融合网络的实时街景语义分割算法[J]. 电子学报, 2022, 50(7): 1609-1620.
[3]	徐建博, 魏昕, 周亮. 面向跨模态通信的信息恢复技术[J]. 电子学报, 2022, 50(7): 1631-1642.
[4]	魏博文, 全红艳. 基于语义与形态特征融合的语义分割网络[J]. 电子学报, 2022, 50(11): 2688-2697.
[5]	徐频捷, 陈逸杰, 李之南, 赵地. 基于事件驱动的车道线识别算法研究[J]. 电子学报, 2021, 49(7): 1379-1385.
[6]	高留杰, 赵文, 张君福, 姜波. G2S:基于语义块的知识图谱问答语义解析[J]. 电子学报, 2021, 49(6): 1132-1141.
[7]	周勇, 陈思霖, 赵佳琦, 张迪, 王瀚正. 基于弱语义注意力的遥感图像可解释目标检测[J]. 电子学报, 2021, 49(4): 679-689.
[8]	吴绿, 张馨月, 唐茉, 王梓, 王永安. Focus+Context语义表征的场景图像分割[J]. 电子学报, 2021, 49(3): 596-604.
[9]	王东升, 王卫民, 祁云松, 王石, 曹存根. 基于错误驱动的语义文法自动扩展学习方法研究[J]. 电子学报, 2021, 49(2): 248-259.
[10]	蒋林, 向超, 朱建阳, 刘奇. 加载语义似然估计的粒子滤波重定位[J]. 电子学报, 2021, 49(2): 306-314.
[11]	张仰森, 周炜翔, 张禹尧, 吴云芳. 一种基于情感计算与层次化多头注意力机制的负面新闻识别方法[J]. 电子学报, 2020, 48(9): 1720-1728.
[12]	孟琭, 徐磊, 郭嘉阳. 一种基于改进的MobileNetV2网络语义分割算法[J]. 电子学报, 2020, 48(9): 1769-1776.
[13]	李青, 钟将, 李立力, 李琪, 张淑芳, 张剑. 一种依需聚合的语义解析图查询模型[J]. 电子学报, 2020, 48(4): 763-771.
[14]	张红斌, 蒋子良, 熊其鹏, 武晋鹏, 邬任重, 袁天, 姬东鸿. 基于改进的有效区域基因选择与跨模态语义挖掘的图像属性标注[J]. 电子学报, 2020, 48(4): 790-799.
[15]	陈震, 马龙, 张聪炫, 黎明, 吴俊劼, 江少锋. 基于语义分割的双目场景流估计[J]. 电子学报, 2020, 48(4): 631-636.