---
title: "Parameter and Model Standardization"
output: 
  rmarkdown::html_vignette:
    toc: true
    fig_width: 10.08
    fig_height: 6
tags: [r, standardization, effect size, cohen d, standardized coefficients]
vignette: >
  %\VignetteIndexEntry{Parameter and Model Standardization}
  \usepackage[utf8]{inputenc}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  chunk_output_type: console
bibliography: bibliography.bib
---

```{r message=FALSE, warning=FALSE, include=FALSE}
library(knitr)
knitr::opts_chunk$set(comment = ">",
                      warning = FALSE,
                      message = FALSE)
options(digits = 2)
options(knitr.kable.NA = '')

pkgs <- c("effectsize", "parameters", "correlation")
if (!all(sapply(pkgs, requireNamespace, quietly = TRUE))) {
  knitr::opts_chunk$set(eval = FALSE)
}

set.seed(333)
```

<!-- centering and interactions! -->

# Introduction

Standardizing parameters (*i.e.*, coefficients) can allow for their comparison
within and between models, variables and studies. Moreover, as it returns
coefficients expressed in terms of **change of variance** (for instance,
coefficients expressed in terms of SD of the response variable), it can allow
for the usage of [effect size interpretation
guidelines](https://easystats.github.io/effectsize/articles/interpret.html),
such as Cohen's (1988) famous rules of thumb.

However, standardizing a model's parameters should *not* be automatically and
mindlessly done: for some research fields, particular variables or types of
studies (*e.g.*, replications), it sometimes makes more sense to keep, use and
interpret the original parameters, especially if they are well known or easily
understood.

Critically, **parameters standardization is not a trivial process**. Different
techniques exist, that can lead to drastically different results. Thus, it is
critical that the standardization method is explicitly documented and detailed.

<!-- **`parameters` include different techniques of parameters
standardization**, described below
[@bring1994standardize;@menard2004six;@gelman2008scaling;@schielzeth2010simple;@menard2011standards].
-->

## Standardizing Parameters of Simple Models

### Standardized Associations

```{r}
library(effectsize)

m <- lm(rating ~ complaints, data = attitude)

standardize_parameters(m)
```

Standardizing the coefficient of this *simple* linear regression gives a value
of `0.87`, but did you know that for a simple regression this is actually the
**same as a correlation**? Thus, you can eventually apply some (*in*)famous
interpretation guidelines (e.g., Cohen's rules of thumb).

```{r}
correlation::correlation(attitude, select = c("rating", "complaints"))
```

### Standardized Differences

How does it work in the case of differences, when **factors** are entered and
differences between a given level and a reference level? You might have heard
that it is similar to a **Cohen's *d***. Well, let's see.

```{r include=FALSE}
mtcars <- datasets::mtcars
```

```{r}
# Select portion of data containing the two levels of interest
mtcars$am <- factor(mtcars$am, labels = c("Manual", "Automatic"))

m <- lm(mpg ~ am, data = mtcars)
standardize_parameters(m)
```

This linear model suggests that the *standardized* difference between *Manual*
(the reference level - the model's intercept) and *Automatic* is of 1.20
standard deviation of `mpg` (because the response variable was standardized,
right?). Let's compute the **Cohen's *d*** between these two levels:

```{r}
cohens_d(mpg ~ am, data = mtcars) 
```

***It is larger!*** Why? How? Both differences should be expressed in units of
SD! But which SDs? Different SDs!

When looking at the difference between groups as a **slope**, the standardized
parameter is the difference between the means in $SD_{mpg}$. That is, the
*slope* between `Manual` and `Automatic` is a change of 1.20 $SD_{mpg}$s.

However, when looking a the difference as a **distance between two populations**, Cohen's d is the distance between the means in units of [**pooled SDs**](https://easystats.github.io/effectsize/reference/sd_pooled.html). That
is, the *distance* between `Manual` and `Automatic` is of 1.48 SDs of *each of
the groups* (here assumed to be equal).

In this simple model, the pooled SD is the residual SD, so we can also estimate
Cohen's *d* as:

```{r}
coef(m)[2] / sigma(m)
```

And we can also get an approximation of Cohen's *d* by converting the
$t$-statistic from the regression model via `t_to_d()`:

```{r}
parameters::model_parameters(m)

t_to_d(4.11, df_error = 30)
```

It is also interesting to note that using the `smart` method (explained in
detail below) when standardizing parameters will give you indices equivalent to
**Glass' *delta***, which is a standardized difference expressed in terms of SD
of the reference group.

```{r}
m <- lm(mpg ~ am, data = mtcars)

standardize_parameters(m, method = "smart")

glass_delta(mpg ~ am, data = mtcars)
```

***... So note that some standardized differences are different than others!
:)***

## Standardizing Parameters of Linear Models

As mentioned above, standardization of parameters can also be used to compare
among parameters within the same model. Essentially, what prevents us from
normally being able to compare among different parameters is that their
underlying variables are on different scales.[^But also as noted above, this is
not always an issue. For example, when the variables scale is important for the
interpretation of results, standardization might in fact hinder interpretation!]

For example, in the following example, we use a liner regression model to
predict a worker's salary (in Shmekels) from their age (years), seniority
(years), overtime (`xtra_hours`) and how many compliments they give their boss
(`n_comps`).

Let us explore the different parameter standardization methods provided by
`effectsize`.

### Standardized Slopes are Not (Always) Correlations

We saw that in simple linear models, the standardized slope is equal to the
correlation between the outcome and predictor - does this hold for **multiple
regression** as well? As in each effect in a regression model is "adjusted" for
the other ones, we might expect coefficients to be somewhat alike to **partial
correlations**. Let's first start by computing the partial correlation between
numeric predictors and the outcome.

```{r}
data("hardlyworking", package = "effectsize")

head(hardlyworking)

correlation::correlation(
  hardlyworking,
  select = "salary",
  select2 = c("xtra_hours", "n_comps", "age", "seniority"),
  partial = TRUE # get partial correlations
) 
```

Let's compare these to the standardized slopes:

```{r}
mod <- lm(salary ~ xtra_hours + n_comps + age + seniority,
          data = hardlyworking)

standardize_parameters(mod)
```

They are quite different! It seems then that ***standardized slopes in multiple
linear regressions are not the same a correlations or partial correlations***
:(

However, not all hope is lost yet - we can still try and recover the partial
correlations from our model, in another way: by converting the *t*-statistics
(and their degrees of freedom, *df*) into a partial correlation coefficient
*r*.

```{r}
params <- parameters::model_parameters(mod)

t_to_r(params$t[-1], df_error = params$df_error[-1])
```

Wow, the retrieved correlations coefficients from the regression model are
**exactly** the same as the partial correlations we estimated above! So these
"*r*" effect sizes can also be used.

### Methods of Standardizing Parameters

Let's convert `age` into a 3-level factor:

```{r}
hardlyworking$age_g <- cut(hardlyworking$age,
                           breaks = c(25,30,35,45))

mod <- lm(salary ~ xtra_hours + n_comps + age_g + seniority,
          data = hardlyworking)

parameters::model_parameters(mod)
```

It seems like the best or most important predictor is `n_comps` as it has the
coefficient. However, it is hard to compare among predictors, as they are on
different scales. To address this issue, we must have all the predictors on the
same scale - usually in the arbitrary unit of *standard deviations*.

#### **`"refit"`**: Re-fitting the model with standardized data

**This method is based on a complete model re-fit with a standardized version of
data**. Hence, this method is equal to standardizing the variables *before*
fitting the model. It is the "purest" and the most accurate [@neter1989applied],
but it is also the most computationally costly and long (especially for heavy
models such as Bayesian models, or complex mixed models). This method is
particularly recommended for models that include interactions or transformations
(e.g., exponentiation, log, polynomial or spline terms).

```{r}
standardize_parameters(mod, method = "refit")
```

`standardize_parameters` also has a `robust` argument (default to `FALSE`),
which enables a **robust standardization of the data**, *i.e.*, based on the
**median** and **MAD** instead of the **mean** and **SD**:

```{r}
standardize_parameters(mod, method = "refit", robust = TRUE)
```

Note that since `age_g` is a factor, it is not numerically standardized, and so
it standardized parameter is still not directly comparable to those of numeric
variables. To address this, we can set `two_sd = TRUE`, thereby scaling
parameters on 2 SDs (or MADs) of the predictors [@gelman2008scaling].

```{r}
standardize_parameters(mod, method = "refit", two_sd = TRUE)
```

`effectsize` also comes with a helper function that returns the re-fit model,
without summarizing it, which can then be used as the original model would:

```{r}
mod_z <- standardize(mod, two_sd = FALSE, robust = FALSE)
mod_z

parameters::model_parameters(mod_z)
```

#### **`"posthoc"`**: Refit without refitting

Post-hoc standardization of the parameters aims at emulating the results
obtained by `"refit"` without refitting the model. The coefficients are divided
by the standard deviation (or MAD if `robust`) of the outcome (which becomes
their expression 'unit'). Then, the coefficients related to numeric variables
are additionally multiplied by the standard deviation (or MAD if `robust`) of
the related terms, so that they correspond to changes of 1 SD of the predictor
(e.g., "A change in 1 SD of *x* is related to a change of 0.24 of the SD of
*y*). This does not apply to binary variables or factors, so the coefficients
are still related to changes in levels. This method is not accurate and tend to
give aberrant results when interactions are specified.

```{r}
standardize_parameters(mod, method = "posthoc")
```

#### **`"smart"`**: Standardization of Model's parameters with Adjustment, Reconnaissance and Transformation

> Experimental

Similar to `method = "posthoc"` in that it does not involve model refitting. The
difference is that the SD of the response is computed on the relevant section of
the data. For instance, if a factor with 3 levels A (the intercept), B and C is
entered as a predictor, the effect corresponding to B vs. A will be scaled by
the variance of the response at the intercept only. As a results, the
coefficients for effects of factors are similar to a Glass' *delta*.

```{r}
standardize_parameters(mod, method = "smart")
```

#### **`"basic"`**: Raw scaling of the model frame

This method is similar to `method = "posthoc"`, but treats all variables as
continuous: it scales the coefficient by the standard deviation of model's
matrix' parameter of factors levels (transformed to integers) or binary
predictors. Although it can be argued that this might be inappropriate for these
cases, this method allows for easier importance judgment across all predictor
type (numeric, factor, interactions...). It is also the type of standardization
implemented by default in other software packages (also `lm.beta::lm.beta()`),
and, such as can be used for reproducibility and replication purposes.

```{r}
standardize_parameters(mod, method = "basic")
```

### Standardizing Parameters In Mixed Models

Linear mixed models (LMM/HLM/MLM) offer an additional conundrum to
standardization - how does one even calculate the SDs of the various predictors?
Or of the response - is it the deviations within each group? Or perhaps between
them?

The solution: standardize according to level of the predictor
[@hoffman2015longitudinal, page 342]! Level 1 parameters are standardized
according to variance *within* groups, while level 2 parameters are standardized
according to variance *between* groups. The resulting standardized coefficient
are also called *pseudo*-standardized coefficients.[^Note that like method
`"basic"`, these are based on the model matrix.]

```{r, eval=knitr::opts_chunk$get("eval") && require(lme4) && require(lmerTest), warning=FALSE}
m <- lme4::lmer(Reaction ~ Days + (Days|Subject), data = lme4::sleepstudy)

standardize_parameters(m, method = "pseudo", ci_method = "satterthwaite")

# compare to:
standardize_parameters(m, method = "basic", ci_method = "satterthwaite")
```

### Standardizing Parameters In Generalized Linear Models

Unlike linear (/mixed) models, in generalized linear (/mixed) models (GLMs)
there is *less* of a need for standardization. Why? Because in many GLMs the
estimated coefficients are themselves measures of effect size, such as
*odds-ratios* (OR) in logistic regression, or *incidence rate ratios* (IRR) in
Poisson regressions. This is because in such model the outcome is **not** on an
arbitrary scale - that is, the meaning of rates and probabilities are changed by
arbitrary linear transformations.

But still, some standardization is sometimes needed, for the predictors.
Luckily, `standardize_parameters()` (and `standardize()`) are smart enough to
know when GLMs are passed so as to only standardize according to the
predictors:

```{r}
mod_b <- glm(am ~ mpg + factor(cyl),
             data = mtcars,
             family = binomial())

standardize_parameters(mod_b, method = "refit", two_sd = TRUE)
# standardize_parameters(mod_b, method = "posthoc", two_sd = TRUE)
# standardize_parameters(mod_b, method = "basic")
```

These can then be converted to OR (with `exp()`) and discussed as the "*change
in Odds as a function of a change in one SD of x*".

```{r}
std <- standardize_parameters(mod_b, method = "refit", two_sd = TRUE)
exp(std$Std_Coefficient)
```

Or we can directly ask for the coefficients to be exponentiated:

```{r}
standardize_parameters(mod_b, method = "refit", two_sd = TRUE, exponentiate = TRUE)
```

## Cohen's *f*

Cohen's $f$ (of [ANOVA fame](https://easystats.github.io/effectsize/articles/anovaES.html)) can be used as a measure of effect size in the context of sequential multiple regression (i.e., [**nested models**](https://easystats.github.io/performance/reference/test_performance.html)).
That is, when comparing two models, we can examine the ratio between the
increase in $R^2$ and the unexplained variance:

$$
f^{2}={R_{AB}^{2}-R_{A}^{2} \over 1-R_{AB}^{2}}
$$

```{r}
m1 <- lm(salary ~ xtra_hours, data = hardlyworking)
m2 <- lm(salary ~ xtra_hours + n_comps + seniority, data = hardlyworking)

cohens_f_squared(m1, model2 = m2)
```

<!-- ## Methods Comparison -->

<!-- We will use the "refit" method as the baseline. We will then compute the
differences between these standardized parameters and the ones provided by the
other functions. The **bigger the (absolute) number, the worse it is**. -->

<!-- > **SPOILER ALERT: the standardization implemented in `effectsize` is the
most accurate and the most flexible.** -->

<!-- ### Convenience function -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- library(effectsize) --> <!--
library(lm.beta) --> <!-- library(MuMIn) -->

<!-- comparison <- function(model, robust=FALSE){ --> <!-- out <-
standardize_parameters(model, method="refit", robust=robust)[1:2] -->

<!-- out$posthoc <- tryCatch({ --> <!-- out[, 2] - standardize_parameters(model,
method="posthoc", robust=robust)[, 2] --> <!-- }, error =
function(error_condition) { --> <!-- "Error" --> <!-- }) --> <!-- out$basic <-
tryCatch({ --> <!-- out[, 2] - standardize_parameters(model, method="basic",
robust=robust)[, 2] --> <!-- }, error = function(error_condition) { --> <!--
"Error" --> <!-- }) -->

<!-- out$lm.beta <- tryCatch({ --> <!-- out[, 2] -
lm.beta::lm.beta(model)$standardized.coefficients --> <!-- }, error =
function(error_condition) { --> <!-- "Error" --> <!-- }, warning =
function(warning_condition) { --> <!-- "Error" --> <!-- }) -->

<!-- out$MuMIn <- tryCatch({ --> <!-- out[, 2] - MuMIn::std.coef(model,
partial.sd=FALSE)[, 1] --> <!-- }, error = function(error_condition) { --> <!--
"Error" --> <!-- }) -->

<!-- ### Data -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- data <- iris --> <!--
data$Group_Sepal.Width <- as.factor(ifelse(data$Sepal.Width > 3, "High", "Low"))
--> <!-- data$Binary_Sepal.Width <- as.factor(ifelse(data$Sepal.Width > 3, 1,
0)) -->

<!-- summary(data) --> <!-- ``` -->

<!-- ### Models with only numeric predictors --> <!-- HERE -->

<!-- #### Linear Model -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- model <- lm(Sepal.Length ~
Petal.Width + Sepal.Width, data=data) --> <!-- comparison(model) --> <!-- ```
-->

<!-- #### Linear Mixed Model -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- library(lme4) -->

<!-- model <- lme4::lmer(Sepal.Length ~ Petal.Width + Sepal.Width + (1|Species),
--> <!-- data=data) --> <!-- comparison(model) --> <!-- ``` -->

<!-- #### Bayesian Models -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- library(rstanarm) -->

<!-- model <- stan_glm(Sepal.Length ~ Petal.Width + Sepal.Width, data=data) -->
<!-- comparison(model) --> <!-- ``` -->

<!-- For these simple models, **all methods return results equal to the "refit"
method** (although the other packages fail). -->

<!-- #### Transformation -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- model <- lm(Sepal.Length ~
poly(Petal.Width, 2) + poly(Sepal.Width, 2), data=data) --> <!--
comparison(model) --> <!-- ``` -->

<!-- When transformation are involved (e.g., polynomial transformations), **the
basic method becomes very unreliable**. -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- model <- lm(Sepal.Length ~
Petal.Width + Group_Sepal.Width, data=data) --> <!-- comparison(model) --> <!--
``` -->

<!-- #### Logistic Model -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- model <-
glm(Binary_Sepal.Width ~ Petal.Width + Species, data=data, family="binomial")
--> <!-- comparison(model) --> <!-- ``` -->

<!-- #### Linear Mixed Model -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- library(lme4) -->

<!-- model <- lme4::lmer(Sepal.Length ~ Petal.Length + Group_Sepal.Width +
(1|Species), data=data) --> <!-- comparison(model) --> <!-- ``` -->

<!-- #### Bayesian Models -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- library(rstanarm) -->

<!-- model <- stan_lmer(Sepal.Length ~ Petal.Width + Group_Sepal.Width +
(1|Species), --> <!-- data=data) --> <!-- comparison(model) --> <!-- ``` -->

<!-- When factors are involved, the basic method (that standardizes the numeric
transformation of factors) give again different results. --> <!-- HERE -->

<!-- ### Models with interactions -->

<!-- Long story short, coeffcient obtained via **posthoc** standardization
(without refitting the model) go berserk when interactions are involved.
However, **this is "normal"**: a regression model estimates coefficient between
two variables when the other predictors are at 0 (are *fixed* at 0, that people
interpret as *"adjusted for"*). When a standardized data is passed (in the
*refit* method), the effects and interactions are estimated at the **means** of
the other predictors (because 0 is the mean for a standardized variable).
Whereas in posthoc standardization, this coefficient correspond to something
different (because the 0 corresponds to something different in standardzed and
non-standardized data). In other words, when it comes to interaction, passing
standardized data results in a different model, which coefficient have an
intrinsically different meaning from unstandardized data. And as [for
now](https://github.com/easystats/effectsize/issues/6/), we are unable to
retrieve one from another. -->

<!-- #### Between continuous -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- model <- lm(Sepal.Length ~
Petal.Width * Sepal.Width, data=data) --> <!-- comparison(model) --> <!-- ```
-->

<!-- #### Between factors -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- model <- lm(Sepal.Length ~
Species * Group_Sepal.Width, data=data) --> <!-- comparison(model) --> <!-- ```
-->

<!-- #### Between factors and continuous -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- model <- lm(Sepal.Length ~
Petal.Width * Group_Sepal.Width, data=data) --> <!-- comparison(model) --> <!--
``` -->

<!-- ```{r message=FALSE, warning=FALSE} --> <!-- model <- lm(Sepal.Length ~
Group_Sepal.Width * Petal.Width, data=data) --> <!-- comparison(model) --> <!--
``` -->

<!-- ## Conclusion -->

<!-- Use `refit` if possible, but if no interactions, can use `posthoc` or
`smart`. -->

# References

