TY - GEN
T1 - MMDF-ESI
T2 - 16th International Conference on Brain Informatics, BI 2023
AU - Jiao, Meng
AU - Yang, Shihao
AU - Wang, Boyu
AU - Xian, Xiaochen
AU - Semenov, Yevgeniy R.
AU - Wan, Guihong
AU - Liu, Feng
N1 - Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.
PY - 2023
Y1 - 2023
N2 - Electrophysiological Source Imaging (ESI) aims to reconstruct the underlying electric brain sources based on Electroencephalography (EEG) or Magnetoencephalography (MEG) measurements of brain activities. Due to the ill-posed nature of the ESI problem, faithfully recovering the latent brain sources has been a significant challenge. Classical algorithms have primarily focused on the design of regularization terms according to predefined priors derived from neurophysiological assumptions. Deep learning frameworks attempt to learn the mapping between brain source signals and scalp EEG/MEG measurements in a data-driven manner, and have demonstrated improved performance compared to the classical methods. Given that EEG and MEG can be complementary for measuring the tangential and radial electrical signals of the cortex, combining both modalities is believed to be advantageous in improving the reconstruction performance of ESI. However, the fusion of these two modalities for the ESI problem has not been fully explored in existing deep learning frameworks. In this paper, we propose a Multi-Modal Deep Fusion (MMDF) framework for solving the ESI inverse problem, termed as MMDF-ESI. Our framework integrates both EEG and MEG in a deep learning framework with a specially designed squeeze-and-excitation module. This integration of EEG and MEG is conducted at an early phase in the deep learning framework rather than on the final decision level fusion. Our experimental results show that (1) the localization accuracy of MMDF-ESI consistently outperforms that of using a single modality; (2) MMDF-ESI exhibits excellent stability, characterized by significantly smaller error variance in source reconstruction compared to benchmark methods, particularly for sources with larger extended activation areas and under low signal-to-noise ratio (SNR) conditions; (3) the evaluation on a real EEG/MEG dataset demonstrates that MMDF-ESI enables a more concentrated reconstruction, effectively recovering an extended area of underlying source activation.
AB - Electrophysiological Source Imaging (ESI) aims to reconstruct the underlying electric brain sources based on Electroencephalography (EEG) or Magnetoencephalography (MEG) measurements of brain activities. Due to the ill-posed nature of the ESI problem, faithfully recovering the latent brain sources has been a significant challenge. Classical algorithms have primarily focused on the design of regularization terms according to predefined priors derived from neurophysiological assumptions. Deep learning frameworks attempt to learn the mapping between brain source signals and scalp EEG/MEG measurements in a data-driven manner, and have demonstrated improved performance compared to the classical methods. Given that EEG and MEG can be complementary for measuring the tangential and radial electrical signals of the cortex, combining both modalities is believed to be advantageous in improving the reconstruction performance of ESI. However, the fusion of these two modalities for the ESI problem has not been fully explored in existing deep learning frameworks. In this paper, we propose a Multi-Modal Deep Fusion (MMDF) framework for solving the ESI inverse problem, termed as MMDF-ESI. Our framework integrates both EEG and MEG in a deep learning framework with a specially designed squeeze-and-excitation module. This integration of EEG and MEG is conducted at an early phase in the deep learning framework rather than on the final decision level fusion. Our experimental results show that (1) the localization accuracy of MMDF-ESI consistently outperforms that of using a single modality; (2) MMDF-ESI exhibits excellent stability, characterized by significantly smaller error variance in source reconstruction compared to benchmark methods, particularly for sources with larger extended activation areas and under low signal-to-noise ratio (SNR) conditions; (3) the evaluation on a real EEG/MEG dataset demonstrates that MMDF-ESI enables a more concentrated reconstruction, effectively recovering an extended area of underlying source activation.
KW - Brain Source Imaging
KW - Brain Source Localization
KW - EEG and MEG Fusion
KW - EEG/MEG Inverse Problem
KW - Multi-Modal Deep Learning
UR - http://www.scopus.com/inward/record.url?scp=85172417802&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85172417802&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-43075-6_24
DO - 10.1007/978-3-031-43075-6_24
M3 - Conference contribution
AN - SCOPUS:85172417802
SN - 9783031430749
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 273
EP - 285
BT - Brain Informatics - 16th International Conference, BI 2023, Proceedings
A2 - Liu, Feng
A2 - Wang, Hongjun
A2 - Zhang, Yu
A2 - Kuai, Hongzhi
A2 - Stephen, Emily P.
Y2 - 1 August 2023 through 3 August 2023
ER -