TY - GEN
T1 - The Lasso and Ridge Regression Yield Biased Estimates of Imbalanced Binary Features
AU - Larsson, Johan
AU - Wallin, Jonas
PY - 2024
Y1 - 2024
N2 - Many regularized methods, such as the lasso and ridge regression, are sensitive to the scales of the features in the data. As a consequence, it has become standard practice to normalize(center and scale) features such that they share the same scale. For continuous data, the most common strategy is standardization: centering and scaling each feature by its mean andand standard deviation, respectively. For binary data, especially when it is high-dimensional and sparse, the most common strategy, however, is to not scale at all. In this paper, we showthat this choice has dramatic effects for the estimated model in the case when the binary features are imbalanced and that these effects, moreover, depend on the type regularization(lasso or ridge) used. In particular, we demonstrate the size of a feature’s corresponding coefficient in the lasso is directly related to its class imbalance and that this effect dependson the normalization used. We suggest possible remedies for this problem and also discuss the case when data is mixed, that is, contains both continuous and binary features.
AB - Many regularized methods, such as the lasso and ridge regression, are sensitive to the scales of the features in the data. As a consequence, it has become standard practice to normalize(center and scale) features such that they share the same scale. For continuous data, the most common strategy is standardization: centering and scaling each feature by its mean andand standard deviation, respectively. For binary data, especially when it is high-dimensional and sparse, the most common strategy, however, is to not scale at all. In this paper, we showthat this choice has dramatic effects for the estimated model in the case when the binary features are imbalanced and that these effects, moreover, depend on the type regularization(lasso or ridge) used. In particular, we demonstrate the size of a feature’s corresponding coefficient in the lasso is directly related to its class imbalance and that this effect dependson the normalization used. We suggest possible remedies for this problem and also discuss the case when data is mixed, that is, contains both continuous and binary features.
M3 - Miscellaneous
ER -