Core Concepts

Statistical analysis often begins with understanding the linear relationship between two variables using covariance and correlation.
Simple Linear Regression (SLR) models this relationship, predicting a dependent variable from a single independent variable.
The Ordinary Least Squares (OLS) method is the standard approach for estimating the parameters (intercept and slope) of the linear regression model.
Tests of significance determine if the identified linear relationship is statistically meaningful (i.e., not due to random chance).
Residual analysis is crucial for validating the underlying assumptions of the regression model.
Summary statistics provide initial data insights and evaluate the overall model fit (e.g., R-square).

Definitions of Key Terms

Covariance: A measure indicating the direction of the linear relationship between two variables; it shows how they change together.
Correlation Coefficient: A standardized measure (ranging from -1 to +1) indicating both the strength and direction of the linear relationship between two variables, unaffected by their units of measurement.
Simple Linear Regression (SLR): A statistical method modeling the relationship between one independent variable ( $x$ ) and one dependent variable ( $y$ ) using a linear equation.
Ordinary Least Squares (OLS): An estimation technique used in regression to find the line that minimizes the sum of the squared differences between observed and predicted values.
Residuals (or Error Term $\epsilon$ ): The differences between the observed values of the dependent variable ( $y_i$ ) and the values predicted by the regression model ( $\hat{y}_i$ ). They represent unexplained variation.
R-square (Coefficient of Determination): A summary statistic representing the proportion of the variance in the dependent variable that is predictable from the independent variable using the regression model.

Covariance and Correlation

Covariance and Correlation - Definition

Measures used to quantify the extent and direction of a linear association between two quantitative variables.

Covariance and Correlation - Key Insights

Covariance indicates the direction (positive or negative) of the linear relationship.
Correlation provides a standardized measure (-1 to +1) of both the strength and direction of the linear relationship.
The Pearson product moment correlation coefficient is a common measure for sample correlation.
Comparison - Covariance: Value depends on the units of measurement of the variables. A positive value indicates variables tend to increase together; a negative value indicates one tends to increase as the other decreases.
Comparison - Correlation: Unit-free measure, making it easier to compare the strength of linear relationships across different pairs of variables.

Covariance and Correlation - Formula

Sample Correlation Coefficient (R): $R(x, y) = \frac{S(x, y)}{s_x \cdot s_y}$ $R (x, y) = \frac{S ( x , y )}{s _{x} \cdot s _{y}}$ Where:
- $S(x, y)$ is the sample covariance between $x$ and $y$ .
- $s_x$ is the sample standard deviation of $x$ .
- $s_y$ is the sample standard deviation of $y$ .

Simple Linear Regression (SLR)

SLR - Definition

A statistical technique that assumes a linear relationship between a single independent variable ( $x$ ) and a dependent variable ( $y$ ) to model and predict outcomes.

SLR - Key Insights

The goal is to find the best-fitting straight line through the data points.
The model includes an intercept ( $\beta_0$ ), a slope ( $\beta_1$ ), and a random error term ( $\epsilon$ ).
The process involves data preparation (splitting, descriptive stats), model definition, parameter estimation (typically via OLS), model validation (diagnostics), and application (prediction).
The expected value of $y$ for a given $x$ , $E(y)$ , represents the average value of $y$ on the true regression line.

SLR - Formula

Regression Model: $y = \beta_0 + \beta_1 x + \epsilon$
Expected Value of y: $E(y) = \beta_0 + \beta_1 x$ $E (y) = β_{0} + β_{1} x$ Where:
- $y$ is the dependent variable.
- $x$ is the independent variable.
- $\beta_0$ is the true population intercept.
- $\beta_1$ is the true population slope.
- $\epsilon$ is the random error term.

Ordinary Least Squares (OLS)

OLS - Definition

A method used to estimate the unknown parameters (intercept $b_0$ and slope $b_1$ as estimates of $\beta_0$ and $\beta_1$ ) in a linear regression model.

OLS - Key Insights

OLS works by minimizing the sum of the squared residuals (the vertical distances between observed data points and the fitted regression line).
It provides the best linear unbiased estimates (BLUE) for the regression coefficients under the standard regression assumptions.

OLS - Formula

Estimated Slope ( $b_1$ ): $b_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$
Estimated Intercept ( $b_0$ $b_{0}$ ): $b_0 = \bar{y} - b_1 \bar{x}$ $b_{0} = \overset{y}{ˉ} - b_{1} \overset{x}{ˉ}$ Where:
- $x_i$ , $y_i$ are individual data points.
- $\bar{x}$ , $\bar{y}$ are the sample means of $x$ and $y$ .
- $\sum$ denotes summation over all data points.

Test of Significance (in Regression)

Test of Significance (in Regression) - Definition

Statistical procedures used to determine whether the observed linear relationship between the independent and dependent variable in the sample data is strong enough to conclude that a relationship exists in the population.

Test of Significance (in Regression) - Key Insights

Commonly involves testing the null hypothesis that the population slope coefficient ( $\beta_1$ ) is zero ( $H_0: \beta_1 = 0$ ).
Uses test statistics (like t-statistic or F-statistic), their corresponding distributions (t-distribution, F-distribution), p-values, and confidence intervals.
Rejecting the null hypothesis suggests statistically significant evidence of a linear relationship.

(Note: Specific formulas for t-statistic or F-statistic are context-dependent but generally compare the estimated coefficient to its standard error or compare model variance to residual variance).

Residual Analysis

Residual Analysis - Definition

The process of examining the model's residuals (the difference between observed and predicted values) to check the validity of regression assumptions and identify potential problems like outliers or influential points.

Residual Analysis - Key Insights

Key assumptions about the error term ( $\epsilon$ $ϵ$ ) checked via residuals:
1. Mean is zero.
2. Constant variance (Homoscedasticity).
3. Independence of errors.
4. Normal distribution.
Residual plots (residuals vs. predicted values or independent variable) are primary tools. An ideal plot shows random scatter around zero (horizontal band) with no discernible pattern.
Patterns (e.g., funnel shape, curve) suggest violations of assumptions, potentially indicating the linear model is inappropriate or needs modification.
Outliers (points far from the general trend) require investigation.

Residual Analysis - Formula

Residual calculation: $\epsilon_i = y_i - \hat{y}_i$ $ϵ_{i} = y_{i} - \overset{y}{^}_{i}$ Where:
- $\epsilon_i$ is the residual for the i-th observation.
- $y_i$ is the observed value of the dependent variable.
- $\hat{y}_i$ is the predicted value from the regression line ( $b_0 + b_1 x_i$ ).

Summary Statistics (in Regression Context)

Summary Statistics - Definition

Numerical values that summarize and describe the characteristics of a dataset or the properties of a regression model.

Summary Statistics - Key Insights

Descriptive Statistics: Used before modeling (e.g., mean, median, standard deviation, count) to understand data variability and center. Visualizations like box plots and scatter plots aid this understanding.
Regression Output Statistics: Used to evaluate model fit and parameter estimates.
- R-square (Coefficient of Determination): Measures the proportion of variation in the dependent variable explained by the model. A higher value indicates a better fit relative to the total variation.
- Standard Error of the Estimate: Measures the typical deviation of observed values from the regression line.
- SST (Total Sum of Squares): Measures total variability in the dependent variable around its mean.
- SSR (Regression Sum of Squares): Measures variability explained by the regression model.
- SSE (Error Sum of Squares): Measures unexplained variability (sum of squared residuals).

Summary Statistics - Examples

Descriptive: Mean, Median, Standard Deviation, Number of observations (N).
Regression Evaluation: R-square, Standard Error of the estimate.
An R-square value of 0.37 indicates that 37% of the total variation in the dependent variable ( $y$ ) is explained by the linear relationship with the independent variable ( $x$ ).

Summary Statistics - Formula

Coefficient of Determination ( $R^2$ ): $R^2 = \frac{SSR}{SST}$
Relationship between Sums of Squares: $SST = SSR + SSE$

Conclusion

These statistical concepts provide a foundational toolkit for analyzing bivariate relationships. Covariance and correlation offer initial insights into the linear association between two variables. Simple Linear Regression, using OLS for parameter estimation, formalizes this relationship into a predictive model. Subsequently, tests of significance validate the statistical relevance of the model's findings, while residual analysis ensures the model's underlying assumptions are met, safeguarding the reliability of conclusions. Summary statistics, particularly R-square, quantify the model's explanatory power, providing a concise measure of fit.

Association between Random Variables and Simple Linear Regression