TY - JOUR
T1 - Regression analysis
T2 - likelihood, error and entropy
AU - Grechuk, Bogdan
AU - Zabarankin, Michael
N1 - Publisher Copyright:
© 2018, Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society.
PY - 2019/3/1
Y1 - 2019/3/1
N2 - In a regression with independent and identically distributed normal residuals, the log-likelihood function yields an empirical form of the L 2 -norm, whereas the normal distribution can be obtained as a solution of differential entropy maximization subject to a constraint on the L 2 -norm of a random variable. The L 1 -norm and the double exponential (Laplace) distribution are related in a similar way. These are examples of an “inter-regenerative” relationship. In fact, L 2 -norm and L 1 -norm are just particular cases of general error measures introduced by Rockafellar et al. (Finance Stoch 10(1):51–74, 2006) on a space of random variables. General error measures are not necessarily symmetric with respect to ups and downs of a random variable, which is a desired property in finance applications where gains and losses should be treated differently. This work identifies a set of all error measures, denoted by E, and a set of all probability density functions (PDFs) that form “inter-regenerative” relationships (through log-likelihood and entropy maximization). It also shows that M-estimators, which arise in robust regression but, in general, are not error measures, form “inter-regenerative” relationships with all PDFs. In fact, the set of M-estimators, which are error measures, coincides with E. On the other hand, M-estimators are a particular case of L-estimators that also arise in robust regression. A set of L-estimators which are error measures is identified—it contains E and the so-called trimmed L p -norms.
AB - In a regression with independent and identically distributed normal residuals, the log-likelihood function yields an empirical form of the L 2 -norm, whereas the normal distribution can be obtained as a solution of differential entropy maximization subject to a constraint on the L 2 -norm of a random variable. The L 1 -norm and the double exponential (Laplace) distribution are related in a similar way. These are examples of an “inter-regenerative” relationship. In fact, L 2 -norm and L 1 -norm are just particular cases of general error measures introduced by Rockafellar et al. (Finance Stoch 10(1):51–74, 2006) on a space of random variables. General error measures are not necessarily symmetric with respect to ups and downs of a random variable, which is a desired property in finance applications where gains and losses should be treated differently. This work identifies a set of all error measures, denoted by E, and a set of all probability density functions (PDFs) that form “inter-regenerative” relationships (through log-likelihood and entropy maximization). It also shows that M-estimators, which arise in robust regression but, in general, are not error measures, form “inter-regenerative” relationships with all PDFs. In fact, the set of M-estimators, which are error measures, coincides with E. On the other hand, M-estimators are a particular case of L-estimators that also arise in robust regression. A set of L-estimators which are error measures is identified—it contains E and the so-called trimmed L p -norms.
KW - Entropy
KW - Error measure
KW - L-estimator
KW - Likelihood
KW - M-estimator
KW - Regression
UR - http://www.scopus.com/inward/record.url?scp=85044377976&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85044377976&partnerID=8YFLogxK
U2 - 10.1007/s10107-018-1256-6
DO - 10.1007/s10107-018-1256-6
M3 - Article
AN - SCOPUS:85044377976
SN - 0025-5610
VL - 174
SP - 145
EP - 166
JO - Mathematical Programming
JF - Mathematical Programming
IS - 1-2
ER -