Extra Features on GBM vs. Logistic Regression

Requirements Clarification & Assessment

1. Understanding the Models:

   • GBM (Gradient Boosting Machine): An ensemble learning method that builds models in a stage-wise fashion and generalizes them by allowing optimization of an arbitrary differentiable loss function. It is well-suited for capturing complex patterns and interactions between features.
   • Logistic Regression: A linear model used for binary classification problems. It assumes a linear relationship between input features and the log-odds of the output.

2. Objective:

   • Evaluate the impact of adding an additional feature on the performance of GBM vs. Logistic Regression.

3. Key Factors to Consider:

   • Feature Importance: How relevant the new feature is to the target variable.
   • Correlation: Whether the new feature is correlated with existing features.
   • Data Size: The number of observations relative to the number of features.
   • Model Complexity: The complexity of the model and its susceptibility to overfitting.

4. Potential Issues:

   • Curse of Dimensionality: Adding features without sufficient data can lead to overfitting, especially in high-dimensional spaces.
   • Overfitting vs. Underfitting: Balancing model complexity and generalization capability.