Papadopoulos Lab

Sparse Inference

Richard Aubrey White — Thu, 13 Apr 2023 22:00:00 GMT

Introduction

Machine learning and statistical modeling are two important tools in data science that are used to make predictions and infer relationships between variables, respectively. Models for prediction aim to estimate the relationship between inputs and outputs in order to make accurate predictions about new, unseen data. In contrast, models for inference aim to understand the underlying relationships between variables in the data, often in the form of identifying causal relationships.

In this blog post, we will explore using models for inference using a simulated dataset, and we will apply penalized regression to perform feature selection on a binary outcome. Penalized regression is particularly useful in situations where the number of predictors (i.e. independent variables) is much larger than the sample size.

We will investigate the frequentist solution of using a two-stage solution with LASSO regression via glmnet and then using the selectiveInference package to perform post inference and adjust for the bias introduced by the selection process. We will also investigate a Bayesian solution that approximates a LASSO regression via a Laplace prior.

Simulating a Dataset

We will simulate a dataset with n = 5000 people and p = 50 variables, where only three of the 50 variables will have an association with the binary outcome, and they will have odds ratios of 4, 3, and 2. The remaining variables will have no association with the outcome.

library(data.table)
library(magrittr)
library(ggplot2)

set.seed(123)
n <- 5000
p <- 50
x <- matrix(rnorm(n * p), nrow = n)
beta <- c(log(4), log(3), log(2), rep(0, 47))
prob <- plogis(x %*% beta)
y <- rbinom(n, 1, prob)

data <- data.frame(cbind(y,x))
colnames(data) <- c("y", paste0("x",1:ncol(x)))

x <- model.matrix(y ~ ., data = data)[,-1]
y <- data$y

LASSO Regression

We will now fit a LASSO regression model using the glmnet package in R. LASSO is a popular method for feature selection in high-dimensional data, where the number of predictors p is much larger than the number of observations n.

# get standard deviation of X, because we will need to standardize/scale it outside of glmnet
sds <- apply(x, 2, sd)

# standardize x
x_scaled <- scale(x,TRUE,TRUE)

# run glmnet
cfit <- glmnet::cv.glmnet(x_scaled,y,standardize=FALSE, family="binomial")
coef(cfit)

51 x 1 sparse Matrix of class "dgCMatrix"
                     s1
(Intercept) 0.008303975
x1          1.081745656
x2          0.821477348
x3          0.452041341
x4          .          
x5          .          
x6          .          
x7          .          
x8          .          
x9          .          
x10         .          
x11         .          
x12         .          
x13         .          
x14         .          
x15         .          
x16         .          
x17         .          
x18         .          
x19         .          
x20         .          
x21         .          
x22         .          
x23         .          
x24         .          
x25         .          
x26         .          
x27         .          
x28         .          
x29         .          
x30         .          
x31         .          
x32         .          
x33         .          
x34         .          
x35         .          
x36         .          
x37         .          
x38         .          
x39         .          
x40         .          
x41         .          
x42         .          
x43         .          
x44         .          
x45         .          
x46         .          
x47         .          
x48         .          
x49         .          
x50         .

No confidence intervals

Note that LASSO regression does not provide any confidence intervals or p-values, only coeficient estimates.

Inference

LASSO regression is a popular method for variable selection in high-dimensional datasets. It shrinks some coefficients to zero, allowing us to select only a subset of variables that have the strongest association with the outcome. However, LASSO does not provide confidence intervals or p-values for the selected variables.

The reason for this is that LASSO performs variable selection by penalizing the likelihood function, not by explicitly testing the significance of each variable. Therefore, we cannot use traditional methods to compute confidence intervals or p-values. Instead, we need to use methods that are specifically designed for post-selection inference.

One such method is provided in the R package selectiveInference. The function fixedLassoInf provides confidence intervals and p-values for LASSO selected variables by accounting for the fact that variable selection was performed. It does this by using a two-stage procedure. In the first stage, LASSO selects a subset of variables. In the second stage, selectiveInference performs inference on the selected variables, adjusting for the selection procedure.

Remember to account for decisions taken at all stages of your modelling process.

It is important to use selectiveInference rather than naively using simpler (but incorrect) methods that do not take into account the two-stage process.

A simpler (but incorrect) method involves first selecting variables with LASSO and then fitting a traditional logistic regression on the selected variables. However, this can lead to biased estimates because the LASSO selection process ignores the uncertainty in the variable selection. Therefore, the second-stage regression will not account for the fact that the variable selection was performed, leading to over-optimistic estimates of the significance of the selected variables.

By using selectiveInference, we can properly account for the selection process and obtain unbiased estimates of the significance of the selected variables.

# compute fixed lambda p-values and selection intervals
out <- selectiveInference::fixedLassoInf(
  x = x_scaled,
  y = y,
  beta = coef(cfit),
  lambda = cfit$lambda.1se,
  alpha = 0.05,
  family = "binomial"
)

retval <- data.frame(
  var = names(out$vars),
  Odds_Ratio = exp(out$coef0/sds[out$vars]),
  LowConf = exp(out$ci[,1]/sds[out$vars]),
  UpperConf = exp(out$ci[,2]/sds[out$vars]),
  pval = out$pv
)
row.names(retval) <- NULL
retval$var[retval$pval < 0.05] <- paste0("*", retval$var[retval$pval < 0.05])
names(retval) <- c(
  "Variable",
  "Odds ratio",
  "Conf int 5%",
  "Conf int 95%",
  "Pvalue"
)
retval

  Variable Odds ratio Conf int 5% Conf int 95% Pvalue
1      *x1   3.761571    3.471404     4.073621      0
2      *x2   2.793509    2.594666     3.007784      0
3      *x3   1.848946    1.727840     1.979062      0

Bayesian Logistic Regression using `rstanarm`

Another way to perform inference on a logistic regression model with feature selection is through Bayesian methods. In particular, we can use the rstanarm R package to fit a Bayesian logistic regression model with a Laplace prior.

Laplace prior

The Laplace prior is used to promote sparsity by assigning a probability distribution to the coefficients that puts more probability mass around zero. It is equivalent to LASSO regression (Tibshirani 1996).

options(mc.cores = parallel::detectCores())

fit <- rstanarm::stan_glm(
  formula = y ~ .,
  data = data,
  family = binomial(),
  prior = rstanarm::laplace(),
  chains = 4,
  iter = 5000,
  refresh=0
)

retval <- data.frame(
  var = names(coef(fit)),
  Odds_Ratio = round(exp(coef(fit)),3),
  round(exp(rstanarm::posterior_interval(fit, prob = 0.9)),3),
  pvalue_equivalent = round(bayestestR::pd_to_p(bayestestR::p_direction(fit)$pd),2)
)
row.names(retval) <- NULL
retval$var[retval$pvalue_equivalent < 0.05] <- paste0("*", retval$var[retval$pvalue_equivalent < 0.05])
names(retval) <- c(
  "Variable",
  "Odds ratio",
  "Cred int 5%",
  "Cred int 95%",
  "Pvalue equivalent"
)
retval

      Variable Odds ratio Cred int 5% Cred int 95% Pvalue equivalent
1  (Intercept)      1.009       0.949        1.073              0.81
2          *x1      4.049       3.744        4.385              0.00
3          *x2      2.968       2.767        3.193              0.00
4          *x3      1.932       1.815        2.058              0.00
5           x4      0.953       0.899        1.011              0.18
6           x5      1.018       0.961        1.079              0.62
7           x6      0.999       0.942        1.060              0.98
8           x7      0.984       0.927        1.044              0.66
9           x8      0.974       0.918        1.034              0.46
10          x9      0.961       0.904        1.019              0.26
11         x10      0.998       0.941        1.058              0.96
12         x11      1.042       0.983        1.104              0.26
13         x12      0.962       0.908        1.020              0.29
14         x13      1.039       0.979        1.103              0.29
15         x14      1.004       0.944        1.066              0.91
16         x15      0.998       0.940        1.059              0.96
17         x16      0.960       0.905        1.018              0.25
18         x17      1.031       0.972        1.095              0.39
19         x18      0.958       0.901        1.018              0.25
20         x19      0.993       0.936        1.054              0.84
21         x20      1.007       0.949        1.069              0.86
22         x21      0.984       0.930        1.042              0.65
23         x22      1.042       0.984        1.106              0.25
24         x23      0.958       0.902        1.017              0.24
25         x24      0.996       0.938        1.057              0.92
26         x25      1.040       0.980        1.105              0.28
27         x26      0.980       0.925        1.039              0.57
28         x27      1.038       0.980        1.101              0.29
29         x28      1.013       0.958        1.072              0.71
30         x29      1.024       0.964        1.085              0.52
31         x30      1.007       0.951        1.068              0.84
32        *x31      0.904       0.851        0.961              0.00
33         x32      1.026       0.968        1.085              0.47
34         x33      1.032       0.972        1.097              0.39
35         x34      1.001       0.943        1.063              0.99
36         x35      1.007       0.950        1.065              0.85
37         x36      1.049       0.989        1.113              0.18
38         x37      1.023       0.965        1.086              0.52
39         x38      1.060       1.000        1.123              0.10
40         x39      1.022       0.961        1.085              0.56
41         x40      1.030       0.969        1.093              0.41
42         x41      0.985       0.926        1.045              0.67
43         x42      1.007       0.949        1.067              0.85
44         x43      1.026       0.967        1.089              0.47
45         x44      0.947       0.892        1.005              0.13
46         x45      1.045       0.983        1.109              0.24
47         x46      1.012       0.953        1.075              0.74
48         x47      1.005       0.947        1.066              0.90
49         x48      0.954       0.899        1.011              0.18
50         x49      0.960       0.904        1.017              0.25
51         x50      0.969       0.912        1.027              0.38

P-value equivalent

Probability of Direction (PoD) and p-values are both statistical measures used in hypothesis testing Makowski et al. (2019). They are similar in that they both provide evidence for or against a null hypothesis. PoD measures the proportion of posterior draws from a Bayesian model that are in the direction of the alternative hypothesis. It provides a measure of the strength of evidence for the alternative hypothesis relative to the null hypothesis. A high PoD value indicates strong evidence in favor of the alternative hypothesis, while a low PoD value indicates weak evidence in favor of the alternative hypothesis.

Similarly, a p-value measures the probability of obtaining a test statistic as extreme as or more extreme than the observed value, assuming that the null hypothesis is true. A low p-value indicates that the observed result is unlikely to have occurred by chance alone, providing evidence against the null hypothesis.

To convert PoD to a p-value equivalent, one approach is to use the following formula:

p-value = 2 * min(PoD, 1-PoD)

This formula assumes a two-tailed test and converts the PoD to a p-value for a test of the null hypothesis that the effect size is equal to zero. The resulting p-value can be interpreted as the probability of obtaining the observed result or a more extreme result under the null hypothesis.

Conclusion

The blog article discusses the limitations of using LASSO (Least Absolute Shrinkage and Selection Operator) models for statistical inference, particularly in situations where the number of predictors (i.e. independent variables) is much larger than the sample size. In these cases, LASSO models can suffer from high variability in the estimated coefficients, which can lead to incorrect or unreliable conclusions.

One proposed solution to this problem is to use a two-stage inference approach, where LASSO is first used to select a subset of predictors, and then a separate statistical method (such as ordinary least squares) is used to estimate the coefficients for the selected predictors. However, this two-stage approach can also have limitations, such as a loss of power in the second stage and increased computational complexity.

In contrast, Bayesian statistics offer a one-stage inference approach that can provide more reliable and interpretable results in complex modeling situations. Bayesian statistics allow for the incorporation of prior knowledge and uncertainty in the model, which can help to reduce variability and improve accuracy. Bayesian methods also provide a framework for model comparison and selection, which can help to identify the most appropriate model for a given dataset.

Overall, while LASSO models can be useful in certain situations, their limitations in high-dimensional data settings highlight the advantages of Bayesian statistics for reliable and interpretable statistical inference.

References

Makowski, Dominique, Matthew S Ben-Shachar, Simon H A Chen, and Daniel Lüdecke. 2019. “Indices of Effect Existence and Significance in the Bayesian Framework.” Frontiers in Psychology 10: 2767. https://doi.org/10.3389/fpsyg.2019.02767.

Makowski, Dominique, Matthew S Ben-Shachar, and Daniel Lüdecke. 2019. “bayestestR: Describing Effects and Their Uncertainty, Existence and Significance Within the Bayesian Framework.” Journal of Open Source Software 4 (40): 1541. https://doi.org/10.21105/joss.01541.

Tibshirani, Robert. 1996. “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society. Series B (Methodological) 58 (1): 267–88. http://www.jstor.org/stable/2346178.

Sampling Bias

Richard Aubrey White — Wed, 22 Feb 2023 23:00:00 GMT

library(data.table)
library(magrittr)
library(ggplot2)

What is Sampling Bias?

Sampling bias refers to the phenomenon of a biased sample being used in a study that does not accurately represent the population being studied. This can happen in a number of ways, such as through selection bias, survivorship bias, or measurement bias. When sampling bias is present, it can lead to inaccurate results, incorrect estimates of associations between variables, and incorrect conclusions. This, in turn, can have an impact on public policy decisions, research funding, and clinical practice.

Types of Sampling Bias

There are several types of sampling bias, including:

1. Selection Bias

Selection bias occurs when the selection of study participants is not random or representative of the larger population. This can happen when certain groups are excluded or overrepresented, leading to inaccurate conclusions about the study population.

For example, if a study only recruits participants from a single geographic region, the results may not be generalizable to the larger population. Similarly, if a study only recruits individuals with a certain health condition, the results may not accurately reflect the general population.

2. Survivorship Bias

Survivorship bias occurs when only the surviving members of a population are included in a study. This can lead to inaccurate conclusions about the population, as those who did not survive may have had different characteristics or experiences.

For example, if a study only includes individuals who survived a specific disease, the results may not be generalizable to the larger population of individuals who did not survive.

3. Measurement Bias

Measurement bias occurs when the measurement instruments or techniques used in a study are inaccurate or unreliable. This can result in inaccurate data and misinterpretation of results.

For example, if a study relies on self-reported data, individuals may underreport or overreport certain behaviors, leading to inaccurate conclusions about the study population. Similarly, if a study uses different measurement techniques for different groups, the results may not be comparable and may lead to inaccurate conclusions.

Example of Sampling Bias in a Study

To better understand the impact of sampling bias on study results, let’s take a look at an example.

Suppose we want to study the relationship between smoking and lung function. We know that in our city there are 100,000 people, 20,000 of whom are smokers. To our study we recruit 5,000 smokers and 5,000 non-smokers (oversampling the smokers, a type of selection bias). We also collect data on how frequently they exercise, whether they have good genes for lung function, and whether they frequently wear hats.

We now want to overcome our selection bias when assessing the association between the outcome of lung function and the exposures of exercise, good genes, and the frequency of hat wearing.

set.seed(4)

d <- data.table(id = 1:100000)
d[, is_smoker := rbinom(.N, 1, 0.2)]
d[, probability_of_exercises_frequently := ifelse(is_smoker==T, 0.05, 0.3)]
d[, exercises_frequently := rbinom(.N, 1, probability_of_exercises_frequently)]
d[, has_good_genes := rbinom(.N, 1, 0.2)]
d[, wears_hats_frequently := rbinom(.N, 1, 0.2)]

d[, lung_function := 30 - 10 * is_smoker + 5 * exercises_frequently + 8 * has_good_genes + rnorm(.N, mean = 0, sd = 3)]

d[, probability_of_selection_uniform := 1/.N]

d[, probability_of_selection_oversample_smoker := ifelse(is_smoker==T, 5, 1)]
d[, probability_of_selection_oversample_smoker := probability_of_selection_oversample_smoker/sum(probability_of_selection_oversample_smoker)]

# We have a dataset with oversampled smokers
d_oversampled_smokers <- d[sample(1:.N, size = 5000, prob = probability_of_selection_oversample_smoker)]
(weight_smoker <- mean(d$is_smoker)/mean(d_oversampled_smokers$is_smoker))

[1] 0.3700516

(weight_non_smoker <- mean(!d$is_smoker)/mean(!d_oversampled_smokers$is_smoker))

[1] 1.747289

d_oversampled_smokers[, weights := ifelse(is_smoker==T, weight_smoker, weight_non_smoker)]

# The real associations:
# is_smoker: -10 (also associated with exercises_frequently!)
# exercises_frequently: +5 (also associated with is_smoker!)
# has_good_genes: +8 (only associated with outcome, not with other exposures)
# wears_hats_frequently: 0 (not associated with outcome nor other exposures)
summary(lm(lung_function ~ is_smoker + exercises_frequently + has_good_genes + wears_hats_frequently, data=d))


Call:
lm(formula = lung_function ~ is_smoker + exercises_frequently + 
    has_good_genes + wears_hats_frequently, data = d)

Residuals:
     Min       1Q   Median       3Q      Max 
-12.6549  -2.0112   0.0055   2.0045  12.1835 

Coefficients:
                       Estimate Std. Error  t value Pr(>|t|)    
(Intercept)            30.02288    0.01421 2112.751   <2e-16 ***
is_smoker             -10.05071    0.02427 -414.150   <2e-16 ***
exercises_frequently    4.99532    0.02250  221.992   <2e-16 ***
has_good_genes          7.98073    0.02355  338.831   <2e-16 ***
wears_hats_frequently  -0.01537    0.02360   -0.651    0.515    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.989 on 99995 degrees of freedom
Multiple R-squared:  0.7975,    Adjusted R-squared:  0.7975 
F-statistic: 9.847e+04 on 4 and 99995 DF,  p-value: < 2.2e-16

# When we run the model in the full data, excluding is_smoker, we get the following associations:
# exercises_frequently: +7.2 (biased from association with is_smoker)
# has_good_genes: +8 (not biased)
# wears_hats_frequently: 0 (not biased)
summary(lm(lung_function ~ exercises_frequently + has_good_genes + wears_hats_frequently, data=d))


Call:
lm(formula = lung_function ~ exercises_frequently + has_good_genes + 
    wears_hats_frequently, data = d)

Residuals:
    Min      1Q  Median      3Q     Max 
-21.056  -2.670   0.928   3.445  14.532 

Coefficients:
                       Estimate Std. Error  t value Pr(>|t|)    
(Intercept)           2.747e+01  2.109e-02 1302.327   <2e-16 ***
exercises_frequently  7.172e+00  3.605e-02  198.941   <2e-16 ***
has_good_genes        7.958e+00  3.881e-02  205.044   <2e-16 ***
wears_hats_frequently 4.393e-04  3.888e-02    0.011    0.991    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 4.926 on 99996 degrees of freedom
Multiple R-squared:  0.4502,    Adjusted R-squared:  0.4502 
F-statistic: 2.73e+04 on 3 and 99996 DF,  p-value: < 2.2e-16

# When we run the model in the biased data, with oversampling of smokers (that has also an association with the outcome):
# exercises_frequently: +9.8 (biased from association with is_smoker and the biased sampling)
# has_good_genes: +7.6 (not biased)
# wears_hats_frequently: +0.3 (not biased)
summary(lm(lung_function ~ exercises_frequently + has_good_genes + wears_hats_frequently, data=d_oversampled_smokers))


Call:
lm(formula = lung_function ~ exercises_frequently + has_good_genes + 
    wears_hats_frequently, data = d_oversampled_smokers)

Residuals:
     Min       1Q   Median       3Q      Max 
-14.4131  -4.3613  -0.6571   4.4586  17.3734 

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)            23.7436     0.1014 234.191   <2e-16 ***
exercises_frequently    9.8631     0.2140  46.095   <2e-16 ***
has_good_genes          7.6164     0.1967  38.726   <2e-16 ***
wears_hats_frequently   0.3363     0.1905   1.765   0.0776 .  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 5.501 on 4996 degrees of freedom
Multiple R-squared:  0.4221,    Adjusted R-squared:  0.4218 
F-statistic:  1217 on 3 and 4996 DF,  p-value: < 2.2e-16

# Run the model in the biased data, with weights:
# exercises_frequently: +7.4 (biased from association with is_smoker)
# has_good_genes: +7.6 (not biased)
# wears_hats_frequently: +0.3 (not biased)
summary(lm(lung_function ~ exercises_frequently + has_good_genes + wears_hats_frequently, data=d_oversampled_smokers, weights = weights))


Call:
lm(formula = lung_function ~ exercises_frequently + has_good_genes + 
    wears_hats_frequently, data = d_oversampled_smokers, weights = weights)

Weighted Residuals:
    Min      1Q  Median      3Q     Max 
-12.537  -4.885  -2.767   2.077  18.233 

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)           27.33919    0.09322 293.291   <2e-16 ***
exercises_frequently   7.40799    0.16055  46.140   <2e-16 ***
has_good_genes         7.60104    0.17490  43.459   <2e-16 ***
wears_hats_frequently  0.26672    0.16734   1.594    0.111    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 4.859 on 4996 degrees of freedom
Multiple R-squared:   0.45, Adjusted R-squared:  0.4497 
F-statistic:  1363 on 3 and 4996 DF,  p-value: < 2.2e-16

# Run the model in the biased data, with is_smoker:
# is_smoker: -9.9 (not biased)
# exercises_frequently: +5.3 (not biased)
# has_good_genes: +7.8 (not biased)
# wears_hats_frequently: +0.2 (not biased)
summary(lm(lung_function ~ is_smoker + exercises_frequently + has_good_genes + wears_hats_frequently, data=d_oversampled_smokers))


Call:
lm(formula = lung_function ~ is_smoker + exercises_frequently + 
    has_good_genes + wears_hats_frequently, data = d_oversampled_smokers)

Residuals:
     Min       1Q   Median       3Q      Max 
-10.6202  -1.9903  -0.0699   1.9855  11.1052 

Coefficients:
                      Estimate Std. Error  t value Pr(>|t|)    
(Intercept)           29.83110    0.07786  383.149   <2e-16 ***
is_smoker             -9.88043    0.08978 -110.051   <2e-16 ***
exercises_frequently   5.25052    0.12300   42.688   <2e-16 ***
has_good_genes         7.79706    0.10630   73.350   <2e-16 ***
wears_hats_frequently  0.15569    0.10296    1.512    0.131    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.973 on 4995 degrees of freedom
Multiple R-squared:  0.8313,    Adjusted R-squared:  0.8311 
F-statistic:  6152 on 4 and 4995 DF,  p-value: < 2.2e-16

Conclusion

Conclusion: Biased datasets can be corrected for by either:

Sample weights
Including the sampling variables as covariates in the regression model

Papadopoulos Lab

Sparse Inference

Introduction

Simulating a Dataset

LASSO Regression

No confidence intervals

Inference

Bayesian Logistic Regression using rstanarm

Laplace prior

P-value equivalent

Conclusion

References

Sampling Bias

What is Sampling Bias?

Types of Sampling Bias

1. Selection Bias

2. Survivorship Bias

3. Measurement Bias

Example of Sampling Bias in a Study

Conclusion

Bayesian Logistic Regression using `rstanarm`