采用多头注意力机制的C&RM-MAKT预测算法

doi:10.12263/DZXB.20220790

摘要/Abstract

摘要：

针对深度知识追踪模型中普遍存在知识状态向量可解释性弱、缺失历史序列数据语义特征信息、忽视历史序列数据对预测结果影响程度等问题，本文提出了一种融合认知诊断理论和多头注意力机制的预测模型C&RM-MAKT（Cognitive & Response Model- Multi-head Attention Knowledge Tracing）.C&RM-MAKT采用Word2Vec和BiLSTM网络将时序数据变换为低维连续实值向量，引入C&RM训练出的可解释性参数来建模学生学习状态，在模型机理层面将知识状态向量扩展为知识状态矩阵.最后，C&RM-MAKT使用多头注意力机制计算出历史序列数据对预测结果的影响程度，以提高模型的可解释性与精度.预测实验结果表明：C&RM-MAKT在HNU_SYS1、HNU_SYS2、Math1和Frcsub四个数据集上都取得了最佳性能结果，尤其在HNU_SYS2中，C&RM-MAKT相较于现有知识追踪模型在AUC（Area Uder the Curve）、ACC（ACCuracy）和F1（F-Measure）指标上分别提升了4.3%、3.6%和5.9%.此外，HNU_SYS2数据集上的可解释性分析表明：C&RM-MAKT模型内部参数可解释性强，一定程度上缓解了深度模型的“黑箱”特性.

关键词: 预测算法, 知识追踪, 认知诊断, 注意力机制, LSTM网络, 时序数据, 语义特征

Abstract:

To address the problems of weak interpretability of knowledge state vectors, lackness of the semantic feature of historical sequence data, and failure to consider the influence of historical sequence data on performance prediction in existing deep knowledge tracking models, this paper proposes a predictive model C&RM-MAKT(Cognitive & Response Model- Multi-head Attention Knowledge Tracing) integrating cognitive diagnostic theory with multiple attention mechanisms. C&RM-MAKT uses Word2Vec and BiLSTM networks to transform the time series data into low-dimensional continuous real vectors, and applies C&RM to pre-train the interpretable parameters for student state modeling, and extends the knowledge state vectors into a knowledge state matrix at the model mechanism level. C&RM-MAKT utilizes multiheaded attention mechanism to estimate the influence degree of historical exercises on the performance prediction to improve the interpretability and accuracy of the model. The prediction experiment results show that C&RM-MAKT performs the best on datasets HNU_SYS1, HNU_SYS2, Math1, and Frcsub. Especially on dataset HNU_SYS2, C&RM-MAKT improves the existing knowledge tracking models by 4.3%, 3.6%, and 5.9% in terms of AUC(Area Uder the Curve), ACC(ACCuracy), and F1(F-Measure), respectively. In addition, according to the interpretability analysis on dataset HNU_SYS2, the internal parameters of the C&RM-MAKT model are highly interpretable, which alleviates the "black box" characteristics of the deep model to a certain extent.

Key words: prediction algorithm, knowledge tracking, cognitive diagnosis, attention mechanism, lstm network, time series data, semantic features

中图分类号:

TP391

王炼红, 罗志辉, 林飞鹏, 李潇瑶. 采用多头注意力机制的C&RM-MAKT预测算法[J]. 电子学报, DOI: 10.12263/DZXB.20220790.

WANG Lian-hong, LUO Zhi-hui, LIN Fei-peng, LI Xiao-yao. C&RM-MAKT Prediction Algorithm Using Multi-head Attention Mechanism[J]. Acta Electronica Sinica, DOI: 10.12263/DZXB.20220790.

图/表 11

图1 C&RM结构图

表1 C&RM中参数及含义

参数名	含义
$θ i$	学习者 $i$ 的能力特征
$η i$	学习者 $i$ 的努力特征
$Ψ = (ψ 1, ψ 2 ⋯, ψ K)$	知识特性矩阵
$Ω = (ω 1, ω 2, ⋯, ω T)$	习题特性矩阵
$Q = (a 1, a 2, ⋯, a T)$	习题知识点关联矩阵
$δ i t$	学习者 $i$ 对习题 $t$ 的掌握水平
$α i k$	学习者 $i$ 对知识点 $k$ 的掌握水平
$r i t$	学习者 $i$ 在习题 $t$ 上的作答结果

表1 C&RM中参数及含义

参数名	含义
$θ i$	学习者 $i$ 的能力特征
$η i$	学习者 $i$ 的努力特征
$Ψ = (ψ 1, ψ 2 ⋯, ψ K)$	知识特性矩阵
$Ω = (ω 1, ω 2, ⋯, ω T)$	习题特性矩阵
$Q = (a 1, a 2, ⋯, a T)$	习题知识点关联矩阵
$δ i t$	学习者 $i$ 对习题 $t$ 的掌握水平
$α i k$	学习者 $i$ 对知识点 $k$ 的掌握水平
$r i t$	学习者 $i$ 在习题 $t$ 上的作答结果

图2 LSTM模型单元结构图

图3 C&RM-MAKT整体框架

图4 习题嵌入式表示

图5 多头注意力机制

表2 数据集信息

数据集	HNU_SYS1	HNU_SYS2	Math1	Frcsub
学生数量	466	770	4209	536
习题数量	90	221	15	20
知识点数量	6	50	11	8
习题作答记录数	21450	84320	63515	10720

表3 HNU_SYS2数据集上消融实验结果

指标	模型	训练集比例
指标	模型	30%	40%	50%	60%	70%	80%
AUC	DKT	0.720	0.724	0.729	0.737	0.744	0.732
	DKT+E	0.726	0.728	0.739	0.743	0.749	0.746
	DKT+CD	0.727	0.731	0.735	0.742	0.746	0.747
	DKT+A	0.746	0.740	0.747	0.756	0.769	0.753
	DKT+E+CD+A	0.755	0.757	0.759	0.763	0.787	0.786
ACC	DKT	0.768	0.770	0.764	0.772	0.777	0.769
	DKT+E	0.772	0.778	0.779	0.779	0.782	0.783
	DKT+CD	0.773	0.777	0.780	0.779	0.780	0.782
	DKT+A	0.777	0.783	0.783	0.784	0.786	0.790
	DKT+E+CD+A	0.786	0.789	0.797	0.806	0.813	0.818
F1	DKT	0.861	0.862	0.860	0.877	0.865	0.858
	DKT+E	0.862	0.863	0.865	0.870	0.876	0.875
	DKT+CD	0.864	0.870	0.869	0.873	0.877	0.875
	DKT+A	0.886	0.881	0.872	0.875	0.896	0.887
	DKT+E+CD+A	0.890	0.901	0.917	0.923	0.924	0.917

图6 动静态知识追踪模型对比结果(a) HNU_SYS1 (b) HNU_SYS2 　　(c) Math1 　(d) Frcsub

表4 C&RM-MAKT相比DKT的性能提升 %

数据集	指标
数据集	ACC	AUC	F1
HNU_SYS1	3.9	1.5	2.7
HNU_SYS2	3.6	4.3	5.9
Frcsub	2.7	1.8	1.6
Math1	5.3	1.2	5.5

图7 C&RM-MAKT算法可解释性分析(a) 知识追踪结果热力图展示　　　　　(b) 知识掌握情况雷达图

参考文献 22

1	FAN X. Item response theory and classical test theory: an empirical comparison of their item/person statistics[J]. Educational and Psychological Measurement, 1998, 58(3): 357-381.
2	DE LA TORRE J. DINA model and parameter estimation: a didactic[J]. Journal of Educational and Behavioral Statistics, 2009, 34(1): 115-130.
3	HARTZ M C. A Bayesian framework for the unified model for assessing cognitive abilities: blending theory with practicality[J]. American Journal of Gastroenterology, 2002, 95(4): 906-909.
4	CORBETT A T, ANDERSON J R. Knowledge tracing: modeling the acquisition of procedural knowledge[J]. User Modeling and User-Adapted Interaction, 1994, 4(4): 253-268.
5	HAWKINS W J, HEFFERNAN N T. Using similarity to the previous problem to improve Bayesian knowledge tracing[C]//Proceedings of the Workshops held at Educational Data Mining 2014(WSEDM 2014). London: CEUR-WS, 2014: 136-140.
6	AGARWAL D, BAKER R, MURALEEDHARAN A. Dynamic knowledge tracing through data driven recency weights[C]//The 13th International Conference on Educational Data Mining. Morocco: Open Access, 2020: 725-729.
7	PIECH C, SPENCER J, HUANG J, et al. Deep knowledge tracing[J]. Computer Science, 2015, 3(3): 19-23.
8	YEUNG C K, YEUNG D Y. Addressing two problems in deep knowledge tracing via prediction-consistent regularization[C]//Proceedings of the 5th Annual ACM Conference on Learning at Scale. London: ACM, 2018: 1-10.
9	MINN S, YI Y, DESMARAIS M C, et al. Deep knowledge tracing and dynamic student classification for knowledge tracing[C]//2018 IEEE International Conference on Data Mining. Singapore: IEEE, 2018: 1182-1187.
10	ZHANG J, SHI X, KING I, et al. Dynamic key-value memory networks for knowledge tracing[C]//Proceedings of the 26th International Conference on World Wide Web. Perth: ACM, 2017: 765-774.
11	SUN X, ZHAO X, LI B, et al. Dynamic key-value memory networks with rich features for knowledge tracing[J]. IEEE Transactions on Cybernetics, 2022, 52(8): 8239 - 8245.
12	LIU Q, HUANG Z, YIN Y, et al. EKT: Exercise-aware knowledge tracing for student performance prediction[J]. IEEE Transactions on Knowledge and Data Engineering, 2019, 33(1): 100-115.
13	TORRE J, DOUGLAS J A. Higher-order latent trait models for cognitive diagnosis[J]. Psychometrika, 2004, 69(3): 333-353.
14	DE L, SONG H. Simultaneous estimation of overall and domain abilities: a higher-order IRT model approach[J]. Applied Psychological Measurement, 2009, 33(8): 620-639.
15	LIU Q, WU R Z, CHEN E H, et al. Fuzzy cognitive diagnosis for modelling examinee performance[J]. ACM Transactions on Intelligent Systems and Technology, 2018, 9(4): 1-26.
16	WANG F, LIU Q, CHEN E, et al. Neural cognitive diagnosis for intelligent education systems[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York: AAAI, 2020: 6153-6161.
17	王炼红,刘畅,周熊,等. 基于学习者认知反应模型的认知诊断方法, CN202110122198.0[P]. 2021-05-07.
18	GELFAND A E, HILLS S E, RACINE-POON A. Illustration of Bayesian inference in normal data models using Gibbs sampling[J]. Journal of the American Statistical Association, 1990, 85(412): 972-985.
19	WEN H, DING G, LIU C, et al. Matrix factorization meets cosine similarity: addressing sparsity problem in collaborative filtering recommender system[C]//The 16th Asia-Pacific Web Conference. Cham: Springer, 2014: 306-317.
20	BAG S, KUMAR S K, TIWARI M K. An efficient recommendation generation using relevant Jaccard similarity[J]. Information Sciences, 2019, 483: 53-64.
21	SUBAKAN C, RAVANELLI M, CORNELL S, et al. Attention is all you need in speech separation[C]//2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Toronto: IEEE, 2021: 21-25.
22	SALAKHUTDINOV R, MNIH A. Probabilistic matrix factorization[C]//Advances in Neural Information Processing Systems 20(NIPS 2007). Vancouver: ACM, 2007: 1257-1264.

[1]	康海燕, 冀源蕊. 基于本地化差分隐私的时序位置发布方案研究[J]. 电子学报, 2022, 50(9): 2222-2232.
[2]	刘群, 谭洪胜, 张优敏, 王国胤. 基于元路径的动态异质网络表示学习[J]. 电子学报, 2022, 50(8): 1830-1839.
[3]	张志昌, 于沛霖, 庞雅丽, 朱林, 曾扬扬. SMGN：用于对话状态跟踪的状态记忆图网络[J]. 电子学报, 2022, 50(8): 1851-1858.
[4]	赵俊男, 佘青山, 孟明, 陈云. 基于多流空间注意力图卷积SRU网络的骨架动作识别[J]. 电子学报, 2022, 50(7): 1579-1585.
[5]	王延达, 陈炜通, 皮德常, 岳琳. 一种自适应记忆神经网络多跳读取与覆盖度机制结合的药物推荐模型[J]. 电子学报, 2022, 50(4): 943-953.
[6]	张浩, 胡昌华, 杜党波, 裴洪, 张建勋. 多状态影响下基于Bi‑LSTM网络的锂电池剩余寿命预测方法[J]. 电子学报, 2022, 50(3): 619-624.
[7]	张凌明, 赵悦, 李鹏程, 刘洋, 高陈强. 基于局部注意力机制的三维牙齿模型分割网络[J]. 电子学报, 2022, 50(3): 681-690.
[8]	伍邦谷, 张苏林, 石红, 朱鹏飞, 王旗龙, 胡清华. 基于多分支结构的不确定性局部通道注意力机制[J]. 电子学报, 2022, 50(2): 374-382.
[9]	魏博文, 全红艳. 基于语义与形态特征融合的语义分割网络[J]. 电子学报, 2022, 50(11): 2688-2697.
[10]	周登文, 李文斌, 李金新, 黄志勇. 一种轻量级的多尺度通道注意图像超分辨率重建网络[J]. 电子学报, 2022, 50(10): 2336-2346.
[11]	王波, 黄冕, 刘利军, 黄青松, 单文琦. 基于多层聚焦Inception-V3卷积网络的细粒度图像分类[J]. 电子学报, 2022, 50(1): 72-78.
[12]	赵琰, 赵凌君, 匡纲要. 基于注意力机制特征融合网络的SAR图像飞机目标快速检测[J]. 电子学报, 2021, 49(9): 1665-1674.
[13]	周东明, 张灿龙, 李志欣, 王智文. 基于多层级视觉融合的图像描述模型[J]. 电子学报, 2021, 49(7): 1286-1290.
[14]	李志欣, 孙亚茹, 唐素勤, 张灿龙, 马慧芳. 双路注意力引导图卷积网络的关系抽取[J]. 电子学报, 2021, 49(2): 315-323.
[15]	江泽涛, 钱艺, 伍旭, 张少钦. 一种基于ARD‑GAN的低照度图像增强方法[J]. 电子学报, 2021, 49(11): 2160-2165.