Likelihood-Based Statistical Inference 1
1.1 Statistical evidence ................... .... 2
1.1.1 Response and its statistical model . .......... 3
1.1.2 Sample space, parameter space, and model function . 3
1.1.3 Interest functions .......... .......... . 5
1.2 Statistical inference ................... .... 8
1.2.1 Evidential statements . ................. 9
1.2.2 Uncertainties of statements . .............. 9
1.3 Likelihood concepts and law of likelihood . .......... 10
1.3.1 Likelihood, score, and observed information functions 10
1.3.2 Law of likelihood and relative likelihood function . . . 15
1.4 Likelihood-based methods ................... . 17
1.4.1 Likelihood region . .. .... ............ .. 19
1.4.2 Uncertainty of likelihood region . ............ 20
1.5 Profile likelihood-based confidence intervals . ......... 22
1.5.1 Profile likelihood function . ............... 23
1.5.2 Profile likelihood region and its uncertainty ...... 26
1.5.3 Profile likelihood-based confidence interval ....... 28
1.5.4 Calculation of profile likelihood-based confidence inter-
vals ................... . ...... . 31
1.5.5 Comparison with the delta method . .......... 34
1.6 Likelihood ratio tests ................... ... 37
1.6.1 Model restricted by hypothesis . ............ 38
1.6.2 Likelihood of the restricted model . .......... 39
1.6.3 General likelihood ratio test statistic (LRT statistic) . 41
1.6.4 Likelihood ratio test and its observed significance level 42
1.7 Maximum likelihood estimate . ................ 45
1.7.1 Maximum likelihood estimate (MLE) . ......... 45
1.7.2 Asymptotic distribution of MLE . ........... 47
1.8 Model selection .............. ........ . 47
1.9 Bibliographic notes ....... ............... . 49
2 Generalized Regression Model 51
2.1 Examples of regression data .................. 51
2.2 Definition of generalized regression model . ........ . . 69
2.2.1 Response ................ ......... 710
2.2.2 Distributions of the components of response ...... 70
2.2.3 Regression function and regression parameter .... . 70
2.2.4 Regressors and model matrix (matrices) ....... . 71
2.2.5 Example ................... ..... . 72
2.3 Special cases of GRM ...................... 73
2.3.1 Assumptions on parts of GRM . ............ 73
2.3.2 Various special GRMs ......... . ...... . 74
2.4 Likelihood inference ..... .................. 75
2.5 MLE with iterative reweighted least squares ......... 76
2.6 Model checking ................... ....... 78
2.7 Bibliographic notes ................ ....... 79
3 General Linear Model 81
3.1 Definition of the general linear model . ............ 81
3.2 Estimate of regression coefficients . .............. 87
3.2.1 Least squares estimate (LSE) . ............. 87
3.2.2 Maximum likelihood estimate (MLE) ......... . .. 90
3.3 Test of linear hypotheses ................... . . 92
3.4 Confidence regions and intervals . ............... 95
3.4.1 Joint confidence regions for finite sets of linear combi-
nations . .................. . .. . . 95
3.4.2 Separate confidence intervals for linear combinations . 97
3.5 Model checking ................... . . . . . 100
3.6 Bibliographic notes ................... .... 103
4 Nonlinear Regression Model 107
4.1 Definition of nonlinear regression model . ........... 107
4.2 Estimate of regression parameters . .............. 109
4.2.1 Least squares estimate (LSE) of regression parameters 109
4.2.2 Maximum likelihood estimate (MLE) ......... . 112
4.3 Approximate distribution of LRT statistic . .......... 114
4.4 Profile likelihood-based confidence region . .......... 115
4.5 Profile likelihood-based confidence interval . ......... 115
4.6 LRT for a hypothesis on finite set of functions . ....... 121
4.7 Model checking ............................ . ... 123
4.8 Bibliographic notes .................. .... . 124
5 Generalized Linear Model 127
5.1 Definition of generalized linear model . ............ 127
5.1.1 Distribution, linear predictor, and link function . . .. 127
5.1.2 Examples of distributions generating generalized linear
models ............ ........ 129
5.2 MLE of regression coefficients . ................ 136
5.2.1 MLE ................... ....... . 136
5.2.2 Newton-Raphson and Fisher-scoring . ......... 138
5.3 Bibliographic notes ................... ... 140
6 Binomial and Logistic Regression Model 141
6.1 Data ............................... 141
6.2 Binomial distribution ................... ... 144
6.3 Link functions ........ .................. 146
6.3.1 Unparametrized link functions . ............ 146
6.3.2 Parametrized link functions . .............. 149
6.4 Likelihood inference ..... . .................. 151
6.4.1 Likelihood function of binomial data . ......... 151
6.4.2 Estimates of parameters . ................ 152
6.4.3 Likelihood ratio statistic or deviance function .... . 154
6.4.4 Distribution of deviance . ................ 154
6.4.5 Model checking ................... ... 156
6.5 Logistic regression model ................... . 157
6.6 Models with other link functions . ............... 163
6.7 Nonlinear binomial regression model . ............. 165
6.8 Bibliographic notes ................... ... 168
7 Poisson Regression Model 169
7.1 Data ................. ..... ....... 169
7.2 Poisson distribution ....................... 170
7.3 Link functions . ......................... 172
7.3.1 Unparametrized link functions . ............ 172
7.3.2 Parametrized link functions . .............. 175
7.4 Likelihood inference . ...................... 176
7.4.1 Likelihood function of Poisson data ........... 176
7.4.2 Estimates of parameters . ................ 177
7.4.3 Likelihood ratio statistic or deviance function .... . 179
7.4.4 Distribution of deviance . ................ 179
7.4.5 Model checking ................... ... 181
7.5 Log-linear model ......................... 182
7.6 Bibliographic notes . ........ ........ ..... 187
8 Multinornial Regression Model 189
8.1 Data ............................... 189
8.2 Multinomial distribution ............... ..... 191
8.3 Likelihood function ........... ........... . 191
8.4 Logistic multinoinial regression model ........ ..... 193
8.5 Proportional odds regression model . ............. 195
8.6 Bibliographic notes ................... ... 199
9 Other Generalized Linear Regressions Models 201
9.1 Negative binomial regression model . ............. 201
9.1.1 Data ....... ....... .... . ........ . 201
9.1.2 Negative binomial distribution . ............ 203
9.1.3 Likelihood inference . .................. 204
9.1.4 Negative binomial logistic regression model ...... 208
9.2 Gamma regression model . ............... ... 211
9.2.1 Data ................... ....... . 211
9.2.2 Gamma distribution ...... ............. 211
9.2.3 Link function ................... .. . 213
9.2.4 Likelihood inference . .................. 214
9.2.5 Model checking ................... ... 221
10 Other Generalized Regression Models 225
10.1 Weighted general linear model . ................ 225
10.1.1 Model ......... ..... ............ 225
10.1.2 Weighted linear regression model as GRM ....... 226
10.2 Weighted nonlinear regression model . ............. 229
10.2.1 Model........ .................. 229
10.2.2 Weighted nonlinear regression model as GRM ..... 230
10.3 Quality design or Taguchi model .......... ...... 231
10.4 Lifetime regression model ................... . 237
10.5 Cox regression model ........... ........... 240
10.6 Bibliographic notes ................... ... 248
A Datasets 251
B Notation Used for Statistical Models 271