Core Concepts

• Multiple Linear Regression (MLR) can incorporate qualitative (categorical) independent variables through the use of dummy variables.
• A categorical variable with $k$ levels requires $k-1$ dummy variables for inclusion in the model, with one level serving as the baseline category for comparison.
• The coefficients of dummy variables represent the average difference in the dependent variable between the represented category and the baseline category, holding other predictors constant.
• Interaction terms, created by multiplying two or more independent variables, allow the model to capture conditional relationships, where the effect of one predictor on the dependent variable depends on the value of another predictor.
• Interpreting models with interaction terms requires considering the coefficients of the main effects and the interaction term together.

Key Terms Definitions

• Categorical Variable: A variable that can take on one of a limited, and usually fixed, number of possible values, assigning each individual or other unit of observation to a particular group or nominal category.
• Dummy Variable: A numerical variable (typically binary 0 or 1) used in regression analysis to represent subgroups of the sample (categories of a qualitative variable).
• Baseline Category (Reference Category): The category of a qualitative variable for which all associated dummy variables are set to zero. Coefficients of other categories are interpreted relative to this baseline.
• Interaction Variable (Interaction Term): A variable created by multiplying two or more independent variables. It represents the combined effect or interaction between those variables on the dependent variable.

Categorical Variables in MLR

Categorical Variables in MLR - Definition

The extension of Multiple Linear Regression to include independent variables that are qualitative or categorical in nature.

Categorical Variables in MLR - Key Insights

• Categorical variables must be numerically coded for regression analysis, typically via dummy variables.
• For a categorical variable with $k$ distinct levels (categories), $k-1$ binary (0/1) dummy variables are created.
• One level is implicitly chosen as the baseline/reference category (coded as 0 for all $k-1$ dummy variables).
• The regression intercept ($b_0$) represents the expected value of the dependent variable for the baseline category when all other numerical predictors are zero.
• The coefficient ($b_i$) associated with a specific dummy variable represents the estimated average difference in the dependent variable between the category represented by that dummy variable and the baseline category, ceteris paribus (holding all other variables constant).

Categorical Variables in MLR - Examples

• Two Categories (Brand: Lenovo vs. Dell): Create one dummy variable, Is_Lenovo.
  • Is_Lenovo = 1 if Brand is Lenovo.
  • Is_Lenovo = 0 if Brand is Dell (Baseline).
  • The coefficient for Is_Lenovo estimates the average difference in the dependent variable (e.g., satisfaction) between Lenovo and Dell.
• Three Categories (Brand: Lenovo, Dell, Asus): Create two dummy variables, e.g., Is_Lenovo and Is_Dell, with Asus as the baseline.
  • Asus: Is_Lenovo = 0, Is_Dell = 0 (Baseline)
  • Lenovo: Is_Lenovo = 1, Is_Dell = 0
  • Dell: Is_Lenovo = 0, Is_Dell = 1
  • The coefficient for Is_Lenovo estimates the average difference between Lenovo and Asus.
  • The coefficient for Is_Dell estimates the average difference between Dell and Asus.
• Location (Premium vs. Non-Premium): Create one dummy variable, Is_Premium.
  • Is_Premium = 1 if location is premium.
  • Is_Premium = 0 if location is non-premium (Baseline).
  • In a model predicting flat price based on area and location (Price = $b_0 + b_1 \cdot Area + b_2 \cdot Is\_Premium$), $b_2$ estimates the average price difference between premium and non-premium flats of the same area.

Categorical Variables in MLR - Formula

• Model with one categorical variable (k=3 levels, baseline=level 1) and one numerical variable ($x_1$): $\hat{y} = b_0 + b_1 x_1 + b_2 \cdot Dummy_2 + b_3 \cdot Dummy_3$ Where:
  • $Dummy_2 = 1$ if category 2, 0 otherwise.
  • $Dummy_3 = 1$ if category 3, 0 otherwise.
  • Category 1 is the baseline ($Dummy_2=0$, $Dummy_3=0$).
  • $b_2$ = Estimated difference in $\hat{y}$ between category 2 and baseline (category 1).
  • $b_3$ = Estimated difference in $\hat{y}$ between category 3 and baseline (category 1).

Interaction Variables in MLR

Interaction Variables in MLR - Definition

Variables created by multiplying two or more independent variables (numerical $\times$ numerical, categorical $\times$ categorical, or numerical $\times$ categorical) to model how their combined presence affects the dependent variable differently than their individual effects alone.

Interaction Variables in MLR - Key Insights

• Interaction terms allow the effect of one independent variable ($X_1$) on the dependent variable ($Y$) to change depending on the level or value of another independent variable ($X_2$).
• If an interaction term is significant, the effects of the individual variables involved in the interaction cannot be interpreted in isolation. The effect of $X_1$ is now dependent on $X_2$'s value, and vice versa.
• Categorical variables involved in interactions can act as moderators, changing the relationship between another predictor and the outcome.

Interaction Variables in MLR - Examples

• Numerical $\times$ Categorical (Work Experience $\times$ Gender):
  • Model: Salary = $b_0 + b_1\cdot Experience + b_2 \cdot Is\_Female + b_3 \cdot (Experience \times Is\_Female)$
  • Here, Is_Female is a dummy variable (1 if Female, 0 if Male - baseline).
  • The term $b_3 \cdot (Experience \times Is\_Female)$ captures whether the impact of an additional year of experience ($b_1 + b_3 \cdot Is\_Female$) on salary differs between females ($b_1 + b_3$) and males ($b_1$). A significant $b_3$ indicates an interaction effect.
• Applications:
  • E-commerce: Interaction between time of day and past purchase history influencing ad click-through rates.
  • Medicine: Interaction between a specific gene variant (categorical) and lifestyle factor (e.g., smoking, numerical/categorical) affecting disease risk or drug response.

Interaction Variables in MLR - Formula

• Model with two numerical variables ($x_1, x_2$) and their interaction: $\hat{y} = b_0 + b_1 x_1 + b_2 x_2 + b_3 (x_1 \cdot x_2)$
  • The effect of a one-unit change in $x_1$ on $\hat{y}$ is $b_1 + b_3 x_2$. This effect depends on the value of $x_2$.
• Model with one numerical ($x_1$) and one categorical (Dummy_1, baseline=0) and their interaction: $\hat{y} = b_0 + b_1 x_1 + b_2 \cdot Dummy_1 + b_3 (x_1 \cdot Dummy_1)$
  • Effect of $x_1$ when Dummy_1=0 (baseline category): $b_1$
  • Effect of $x_1$ when Dummy_1=1 (other category): $b_1 + b_3$
  • $b_3$ represents the difference in the effect of $x_1$ between the two categories.

Conclusion

Incorporating categorical variables via dummy coding and interaction terms significantly enhances the flexibility and realism of Multiple Linear Regression models. These techniques allow analysts to quantify the impact of qualitative factors and to model complex scenarios where the effect of one predictor is conditional upon another. Mastering the creation and interpretation of dummy and interaction variables enables more nuanced data analysis, leading to better-informed business strategies and decisions.