Penalized likelihood and model averaging

Title: Penalized likelihood and model averaging Authors: Zhou, Jianhong (周建红) Abstract: Penalized likelihood and model averaging are alternatives to model selection. The former has the attractive feature of model selection, and its parameter estimation can be achieved by a single minimization problem with computational cost growing polynomially with the sample size. The latter compromises across all the competing models so that it can incorporate model uncertainty into the estimation process. Model averaging also can avoid selecting very poor models, which in turn holds the promise of reducing estimation risks. Penalized likelihood and model averaging have advantages over model selection in some aspects, and they have been developed in parallel. This prompts the question whether there exists any relationship between penalized likelihood and model averaging. However, to the best of my knowledge, there is a paucity of literature focused on this question. An creditable exception is made by Ullah et al. (2013). They propose a new generalized ridge estimator which is one of the penalized likelihoods, and prove that this estimator is algebraically identical to the model average estimator. In this thesis, we choose the Tobit model and Poisson regression model to study penalized likelihood, model averaging and the relationship between them. The Tobit model is now a standard approach to model censored dependent variables. Based on the seminal contribution of Tobin (1958), many extensions of the original Tobit model have been developed, and numerous applications of these models have appeared in economics since the 1970s. The Poisson regression model is a widely used econometric and statistical tool for studying the relationship between a Poisson-type response variable and a set of explanatory variables. We arrange this thesis in the following manner: Chapter 1 presents the rationale and introduction. We also introduce the Tobit model, the Poisson regression model, and the methodology of the performance measures. Chapter 2 surveys the relevant literature on model selection, penalized likelihood, and model averaging. Additionally, under the statistical framework, we discuss the relationship between model selection and penalized likelihood, and the relationship between pretest and the WALS (weighted-average least squares) estimator. We also present model averaging, i.e., Bayesian model averaging and frequestist model averaging under the same statistical framework. Chapter 3 considers the Tobit I model. First, we propose one-step sparse estimates, and their adjusted implication is also suggested. We conduct a simulation study to investigate the performance of one-step sparse estimates with LASSO (least absolute shrinkage and selection operator) and ALASSO (adaptive LASSO) penalties. For simplicity, we call them LASSO estimator and ALASSO estimator. The results demonstrate that, compared with LASSO estimators, ALASSO estimators usually have advantages on getting more accurate results in terms of mean square error, and providing a more accurate number of correct zeros. However, they have a disadvantage on producing a more accurate number of correct nonzeros. Second, we explore the relationships among one-step sparse estimates, model selection, and model averaging. The results by both a simulation study and empirical example, demonstrate that none of these three methods consistently performs better than the others. Model averaging tends to get more accurate results when there is a high noise level to the model; otherwise, none of these three methods has an obvious superiority. Chapter 4 considers the Tobit II model. We propose a procedure combining the Heckman two-step procedure with penalized regression for parameter estimation and variable selection. We investigate the finite samples results of this procedure with LASSO and ALASSO penalties by designing a simulation study. The conclusion is similar with that of Chapter 3. Next, we study the relationships among this procedure, model selection, and model averaging. From the results by both simulation study and empirical example, it can be seen that each method has its own advantage. Especially in cases with a high level of noise, model averaging has competitive strength over other methods. Chapter 5 develops a model averaging procedure based on an unbiased estimator of the expected Kullback-Leibler distance for the Poisson regression. Simulation studies show that, compared with other commonly used model selection and model average estimators, the proposed model average estimator performs better in certain situations, especially when the model is nonsparse. In all the cases we simulated, the proposed method never produces the worst results. Our proposed method is further applied to a real data example, demonstrating the advantage of our method. Finally, Chapter 6 summarizes this thesis and presents possible directions for future study. Notes: CityU Call Number: QA276 .Z45 2015; xviii, 173 pages : illustrations 30 cm; Thesis (Ph.D.)--City University of Hong Kong, 2015.; Includes bibliographical references (pages 153-166)