Core Concepts

Multiple Linear Regression (MLR) extends simple linear regression to model the linear relationship between one dependent variable and multiple independent variables.
The model aims to predict the dependent variable or understand the individual contribution of each independent variable while controlling for others.
Ordinary Least Squares (OLS) is the standard method for estimating the model's coefficients (intercept and slopes) by minimizing the sum of squared errors.
Significance tests (F-test and t-tests) are crucial for assessing the overall model validity and the statistical importance of individual predictors.
Residual analysis is performed to validate the underlying assumptions of the linear regression model.
R-squared measures the proportion of variance in the dependent variable explained by the model, while multicollinearity assesses problematic correlations among independent variables.

Key Terms Definitions

Multiple Linear Regression (MLR): A statistical technique used to model the linear relationship between a single dependent (response) variable and two or more independent (predictor) variables.
Regression Coefficients ( $\beta_0, \beta_1, ..., \beta_p$ ): Population parameters representing the intercept ( $\beta_0$ ) and the change in the mean of the dependent variable for a one-unit change in a specific independent variable ( $\beta_i$ ), holding all other independent variables constant.
Estimated Coefficients ( $b_0, b_1, ..., b_p$ ): Sample-based estimates of the population regression coefficients, calculated using methods like OLS.
Sum of Squares Error (SSE): The sum of the squared differences between the observed values ( $y_i$ ) and the values predicted by the regression model ( $\hat{y}_i$ ). OLS minimizes this value.
Coefficient of Determination ( $R^2$ ): A statistic indicating the proportion of the total variability in the dependent variable that is accounted for by the set of independent variables in the regression model.
Multicollinearity: A condition where two or more independent variables in an MLR model are highly linearly related, potentially causing issues with coefficient estimation and interpretation.
Variance Inflation Factor (VIF): A measure used to quantify the severity of multicollinearity in an MLR model for a specific independent variable.

Basics of Multiple Linear Regression

Basics - Definition

An extension of simple linear regression used to establish and quantify a linear relationship between a dependent variable ( $y$ ) and two or more independent variables ( $x_1, x_2, ..., x_p$ ).

Basics - Key Insights

Allows for the analysis of more complex relationships where multiple factors influence an outcome.
The coefficient ( $\beta_i$ or its estimate $b_i$ ) for an independent variable $x_i$ represents the expected change in $y$ for a one-unit increase in $x_i$ , assuming all other independent variables in the model are held constant.
The goal is often prediction of $y$ or understanding the independent effect of each $x$ variable.

Basics - Formula

Population Model: $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_p x_p + \epsilon$
Estimated Model (based on sample data): $\hat{y} = b_0 + b_1 x_1 + b_2 x_2 + ... + b_p x_p$ $\overset{y}{^} = b_{0} + b_{1} x_{1} + b_{2} x_{2} + ... + b_{p} x_{p}$ Where:
- $y$ : Observed dependent variable value.
- $\hat{y}$ : Predicted dependent variable value.
- $x_i$ : Value of the i-th independent variable.
- $\beta_0, \beta_i$ : Population intercept and slope coefficients.
- $b_0, b_i$ : Sample estimates of intercept and slope coefficients.
- $\epsilon$ : Random error term.
- $p$ : Number of independent variables.

Ordinary Least Squares (OLS)

Ordinary Least Squares (OLS) - Definition

The primary method used to estimate the regression coefficients ( $b_0, b_1, ..., b_p$ ) in multiple linear regression.

Ordinary Least Squares (OLS) - Key Insights

OLS finds the coefficient values that minimize the sum of the squared differences between observed dependent variable values ( $y_i$ ) and the values predicted by the model ( $\hat{y}_i$ ). This minimized sum is the SSE.
While formulas exist (often involving matrix algebra), computation is typically performed using statistical software (e.g., Excel, R, Python). Interpretation of software output is key.

Ordinary Least Squares (OLS) - Formula

Objective: Minimize $SSE = \sum(y_i - \hat{y}_i)^2 = \sum(y_i - (b_0 + b_1 x_{1i} + ... + b_p x_{pi}))^2$

Test of Significance

Overall Test of Significance (F-test)

Overall Test of Significance (F-test) - Definition

A statistical test to determine if there is a significant linear relationship between the dependent variable and at least one of the independent variables included in the model. It assesses the overall utility of the model.

Overall Test of Significance (F-test) - Key Insights

Tests the null hypothesis that all slope coefficients are simultaneously zero against the alternative that at least one is non-zero.
- $H_0: \beta_1 = \beta_2 = ... = \beta_p = 0$
- $H_1$ : At least one $\beta_i \neq 0$ (for $i = 1, ..., p$ )
Rejection of $H_0$ (typically based on a small p-value associated with the F-statistic) indicates that the model, as a whole, explains a statistically significant amount of variation in the dependent variable.

Test of Significance for Individual Coefficients (t-test)

Test of Significance for Individual Coefficients (t-test) - Definition

A statistical test performed for each independent variable's coefficient ( $b_i$ ) to determine if that specific variable has a statistically significant linear relationship with the dependent variable, controlling for the presence of the other independent variables in the model.

Test of Significance for Individual Coefficients (t-test) - Key Insights

Tests the null hypothesis that a specific population coefficient ( $\beta_i$ $β_{i}$ ) is zero against the alternative that it is not zero.
- $H_0: \beta_i = 0$
- $H_1: \beta_i \neq 0$
Rejection of $H_0$ for a specific $b_i$ suggests that the corresponding variable $x_i$ makes a statistically significant contribution to predicting $y$ , even when other predictors are accounted for.
The significance of the intercept ( $b_0$ ) is often less critical unless the point where all $x$ variables are zero is meaningful in the context.

Residual Analysis

Residual Analysis - Definition

The examination of the differences (residuals) between the observed values ( $y_i$ ) and the predicted values ( $\hat{y}_i$ ) from the regression model to check if the model's assumptions are reasonably satisfied.

Residual Analysis - Key Insights

Assumptions checked include: linearity, independence of errors, constant variance of errors (homoscedasticity), and normality of errors.
Residual plots (e.g., residuals vs. predicted values, residuals vs. each independent variable) are used to visually inspect for patterns. Ideally, plots should show random scatter around zero. Non-random patterns (e.g., curves, funnels) suggest assumption violations.
Standardized residuals are often used to identify potential outliers (observations with unusually large errors). A common rule of thumb is that standardized residuals outside $\pm 2$ warrant investigation, and those outside $\pm 3$ are likely outliers.
Influential observations are points that, if removed, would significantly change the estimated regression coefficients.

Residual Analysis - Formula

Residual: $e_i = y_i - \hat{y}_i$

Summary Statistics

Coefficient of Determination (R-squared)

Coefficient of Determination (R-squared) - Definition

In MLR, $R^2$ (also called the multiple coefficient of determination) measures the proportion of the total variance in the dependent variable ( $y$ ) that is explained by the linear combination of all independent variables ( $x_1, ..., x_p$ ) included in the model.

Coefficient of Determination (R-squared) - Key Insights

Ranges from 0 (no variance explained) to 1 (all variance explained).
Provides a measure of the model's goodness-of-fit. A higher $R^2$ generally indicates a better fit, but should be interpreted in context.
Example: An $R^2$ of 0.67 means that 67% of the variability observed in the dependent variable is accounted for by the independent variables in the model.
Note: $R^2$ never decreases when adding more variables, even if they are irrelevant. Adjusted $R^2$ is often preferred for comparing models with different numbers of predictors.

Coefficient of Determination (R-squared) - Formula

$R^2 = \frac{SSR}{SST} = 1 - \frac{SSE}{SST}$ $R^{2} = \frac{SSR}{SST} = 1 - \frac{SSE}{SST}$ Where:
- $SST = \sum(y_i - \bar{y})^2$ (Total Sum of Squares: total variability in $y$ )
- $SSR = \sum(\hat{y}_i - \bar{y})^2$ (Regression Sum of Squares: variability explained by the model)
- $SSE = \sum(y_i - \hat{y}_i)^2$ (Error Sum of Squares: unexplained variability)
- $SST = SSR + SSE$

Multicollinearity

Multicollinearity - Definition

A phenomenon in multiple linear regression where two or more independent variables are highly correlated with each other.

Multicollinearity - Key Insights

High multicollinearity does not violate OLS assumptions but makes it difficult to disentangle the individual effects of the correlated predictors on the dependent variable.
Consequences can include:
- Unstable and unreliable estimates of individual regression coefficients (large standard errors).
- Difficulty in interpreting coefficients as the unique effect of one variable.
- Coefficients might have unexpected signs or magnitudes.
Detection methods include:
- Examining the correlation matrix of independent variables (rule of thumb: correlations > 0.7 or 0.8 might indicate issues).
- Calculating the Variance Inflation Factor (VIF) for each predictor. A common rule of thumb is that VIF > 10 suggests problematic multicollinearity.

Multicollinearity - Formula

$VIF_j = \frac{1}{1 - R_j^2}$ Where $R_j^2$ is the R-squared value obtained by regressing the j-th independent variable ( $x_j$ ) on all other independent variables in the model.

Conclusion

Multiple Linear Regression (MLR) provides a powerful framework for modeling how multiple factors linearly influence an outcome variable. Coefficients are estimated using Ordinary Least Squares (OLS), quantifying the effect of each predictor while controlling for others. The overall model adequacy is checked using the F-test, while individual predictor significance is assessed via t-tests. Evaluating model fit with R-squared and validating assumptions through residual analysis are critical steps, alongside diagnosing potential issues like multicollinearity using VIF to ensure reliable interpretation and prediction.

Multiple Linear Regression I