Imputation of missing values in time series with lagged correlations

Shah Atiqur Rahman; Yuxiao Huang; Jan Claassen; Samantha Kleinberg

doi:10.1109/ICDMW.2014.110

Imputation of missing values in time series with lagged correlations

Shah Atiqur Rahman
, Yuxiao Huang
, Jan Claassen
, Samantha Kleinberg

Stevens Institute of Technology
Columbia University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

10 Scopus citations

Abstract

Missing values are a common problem in real world data and are particularly prevalent in biomedical time series, where a patient's medical record may be split across multiple institutions or a device may briefly fail. These data are not missing completely at random, so ignoring the missing values can lead to bias and error during data mining. However, current methods for imputing missing values have yet to account for the fact that variables are correlated and that those relationships exist across time. To address this, we propose an imputation method (FLk-NN) that incorporates time lagged correlations both within and across variables by combining two imputation methods, based on an extension to k-NN and the Fourier transform. This enables imputation of missing values even when all data at a time point is missing and when there are different types of missingness both within and across variables. In comparison to other approaches on two biological datasets (simulated glucose in Type 1 diabetes and multi-modality neurological ICU monitoring) the proposed method has the highest imputation accuracy. This was true for up to half the data being missing and when consecutive missing values are a significant fraction of the overall time series length.

Original language	English
Title of host publication	Proceedings - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014
Editors	Zhi-Hua Zhou, Wei Wang, Ravi Kumar, Hannu Toivonen, Jian Pei, Joshua Zhexue Huang, Xindong Wu
Pages	753-762
Number of pages	10
Edition	January
ISBN (Electronic)	9781479942749
DOIs	https://doi.org/10.1109/ICDMW.2014.110
State	Published - 26 Jan 2015
Event	14th IEEE International Conference on Data Mining Workshops, ICDMW 2014 - Shenzhen, China Duration: 14 Dec 2014 → …

Publication series

Name	IEEE International Conference on Data Mining Workshops, ICDMW
Number	January
Volume	2015-January
ISSN (Print)	2375-9232
ISSN (Electronic)	2375-9259

Conference

Conference	14th IEEE International Conference on Data Mining Workshops, ICDMW 2014
Country/Territory	China
City	Shenzhen
Period	14/12/14 → …

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Keywords

Fourier imputation
correlated data with time-lag
extended k-NN imputation
missing data

Access to Document

10.1109/ICDMW.2014.110

Cite this

Rahman, S. A., Huang, Y., Claassen, J., & Kleinberg, S. (2015). Imputation of missing values in time series with lagged correlations. In Z.-H. Zhou, W. Wang, R. Kumar, H. Toivonen, J. Pei, J. Zhexue Huang, & X. Wu (Eds.), Proceedings - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014 (January ed., pp. 753-762). Article 7022671 (IEEE International Conference on Data Mining Workshops, ICDMW; Vol. 2015-January, No. January). https://doi.org/10.1109/ICDMW.2014.110

Rahman, Shah Atiqur ; Huang, Yuxiao ; Claassen, Jan et al. / Imputation of missing values in time series with lagged correlations. Proceedings - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014. editor / Zhi-Hua Zhou ; Wei Wang ; Ravi Kumar ; Hannu Toivonen ; Jian Pei ; Joshua Zhexue Huang ; Xindong Wu. January. ed. 2015. pp. 753-762 (IEEE International Conference on Data Mining Workshops, ICDMW; January).

@inproceedings{58c81bcffb25477fa2576945e857c257,

title = "Imputation of missing values in time series with lagged correlations",

abstract = "Missing values are a common problem in real world data and are particularly prevalent in biomedical time series, where a patient's medical record may be split across multiple institutions or a device may briefly fail. These data are not missing completely at random, so ignoring the missing values can lead to bias and error during data mining. However, current methods for imputing missing values have yet to account for the fact that variables are correlated and that those relationships exist across time. To address this, we propose an imputation method (FLk-NN) that incorporates time lagged correlations both within and across variables by combining two imputation methods, based on an extension to k-NN and the Fourier transform. This enables imputation of missing values even when all data at a time point is missing and when there are different types of missingness both within and across variables. In comparison to other approaches on two biological datasets (simulated glucose in Type 1 diabetes and multi-modality neurological ICU monitoring) the proposed method has the highest imputation accuracy. This was true for up to half the data being missing and when consecutive missing values are a significant fraction of the overall time series length.",

keywords = "Fourier imputation, correlated data with time-lag, extended k-NN imputation, missing data",

author = "Rahman, \{Shah Atiqur\} and Yuxiao Huang and Jan Claassen and Samantha Kleinberg",

note = "Publisher Copyright: {\textcopyright} 2014 IEEE.; 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014 ; Conference date: 14-12-2014",

year = "2015",

month = jan,

day = "26",

doi = "10.1109/ICDMW.2014.110",

language = "English",

series = "IEEE International Conference on Data Mining Workshops, ICDMW",

number = "January",

pages = "753--762",

editor = "Zhi-Hua Zhou and Wei Wang and Ravi Kumar and Hannu Toivonen and Jian Pei and \{Zhexue Huang\}, Joshua and Xindong Wu",

booktitle = "Proceedings - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014",

edition = "January",

}

Rahman, SA, Huang, Y, Claassen, J & Kleinberg, S 2015, Imputation of missing values in time series with lagged correlations. in Z-H Zhou, W Wang, R Kumar, H Toivonen, J Pei, J Zhexue Huang & X Wu (eds), Proceedings - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014. January edn, 7022671, IEEE International Conference on Data Mining Workshops, ICDMW, no. January, vol. 2015-January, pp. 753-762, 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014, Shenzhen, China, 14/12/14. https://doi.org/10.1109/ICDMW.2014.110

Imputation of missing values in time series with lagged correlations. / Rahman, Shah Atiqur; Huang, Yuxiao; Claassen, Jan et al.
Proceedings - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014. ed. / Zhi-Hua Zhou; Wei Wang; Ravi Kumar; Hannu Toivonen; Jian Pei; Joshua Zhexue Huang; Xindong Wu. January. ed. 2015. p. 753-762 7022671 (IEEE International Conference on Data Mining Workshops, ICDMW; Vol. 2015-January, No. January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Imputation of missing values in time series with lagged correlations

AU - Rahman, Shah Atiqur

AU - Huang, Yuxiao

AU - Claassen, Jan

AU - Kleinberg, Samantha

PY - 2015/1/26

Y1 - 2015/1/26

N2 - Missing values are a common problem in real world data and are particularly prevalent in biomedical time series, where a patient's medical record may be split across multiple institutions or a device may briefly fail. These data are not missing completely at random, so ignoring the missing values can lead to bias and error during data mining. However, current methods for imputing missing values have yet to account for the fact that variables are correlated and that those relationships exist across time. To address this, we propose an imputation method (FLk-NN) that incorporates time lagged correlations both within and across variables by combining two imputation methods, based on an extension to k-NN and the Fourier transform. This enables imputation of missing values even when all data at a time point is missing and when there are different types of missingness both within and across variables. In comparison to other approaches on two biological datasets (simulated glucose in Type 1 diabetes and multi-modality neurological ICU monitoring) the proposed method has the highest imputation accuracy. This was true for up to half the data being missing and when consecutive missing values are a significant fraction of the overall time series length.

AB - Missing values are a common problem in real world data and are particularly prevalent in biomedical time series, where a patient's medical record may be split across multiple institutions or a device may briefly fail. These data are not missing completely at random, so ignoring the missing values can lead to bias and error during data mining. However, current methods for imputing missing values have yet to account for the fact that variables are correlated and that those relationships exist across time. To address this, we propose an imputation method (FLk-NN) that incorporates time lagged correlations both within and across variables by combining two imputation methods, based on an extension to k-NN and the Fourier transform. This enables imputation of missing values even when all data at a time point is missing and when there are different types of missingness both within and across variables. In comparison to other approaches on two biological datasets (simulated glucose in Type 1 diabetes and multi-modality neurological ICU monitoring) the proposed method has the highest imputation accuracy. This was true for up to half the data being missing and when consecutive missing values are a significant fraction of the overall time series length.

KW - Fourier imputation

KW - correlated data with time-lag

KW - extended k-NN imputation

KW - missing data

UR - https://www.scopus.com/pages/publications/84936851652

UR - https://www.scopus.com/pages/publications/84936851652#tab=citedBy

U2 - 10.1109/ICDMW.2014.110

DO - 10.1109/ICDMW.2014.110

M3 - Conference contribution

AN - SCOPUS:84936851652

T3 - IEEE International Conference on Data Mining Workshops, ICDMW

SP - 753

EP - 762

BT - Proceedings - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014

A2 - Zhou, Zhi-Hua

A2 - Wang, Wei

A2 - Kumar, Ravi

A2 - Toivonen, Hannu

A2 - Pei, Jian

A2 - Zhexue Huang, Joshua

A2 - Wu, Xindong

T2 - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014

Y2 - 14 December 2014

ER -

Rahman SA, Huang Y, Claassen J, Kleinberg S. Imputation of missing values in time series with lagged correlations. In Zhou ZH, Wang W, Kumar R, Toivonen H, Pei J, Zhexue Huang J, Wu X, editors, Proceedings - 14th IEEE International Conference on Data Mining Workshops, ICDMW 2014. January ed. 2015. p. 753-762. 7022671. (IEEE International Conference on Data Mining Workshops, ICDMW; January). doi: 10.1109/ICDMW.2014.110

Imputation of missing values in time series with lagged correlations

Abstract

Publication series

Conference

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this