learn-python/contrib/machine-learning/random-forest.md

Random Forest

Random Forest is a versatile machine learning algorithm capable of performing both regression and classification tasks. It is an ensemble method that operates by constructing a multitude of decision trees during training and outputting the average prediction of the individual trees (for regression) or the mode of the classes (for classification).

Introduction

Random Forest is an ensemble learning method used for classification and regression tasks. It is built from multiple decision trees and combines their outputs to improve the model's accuracy and control over-fitting.

How Random Forest Works

1. Bootstrap Sampling:

• Random subsets of the training dataset are created with replacement. Each subset is used to train an individual tree.

2. Decision Trees:

• Multiple decision trees are trained on these subsets.

3. Feature Selection:

• At each split in the decision tree, a random selection of features is chosen. This randomness helps create diverse trees.

4. Voting/Averaging:

For classification, the mode of the classes predicted by individual trees is taken (majority vote). For regression, the average of the outputs of the individual trees is taken.

Detailed Working Mechanism

Step 1: Bootstrap Sampling:

Each tree is trained on a random sample of the original data, drawn with replacement (bootstrap sample). This means some data points may appear multiple times in a sample while others may not appear at all.

Step 2: Tree Construction:

Each node in the tree is split using the best split among a random subset of the features. This process adds an additional layer of randomness, contributing to the robustness of the model.

Step 3: Aggregation:

For classification tasks, the final prediction is based on the majority vote from all the trees. For regression tasks, the final prediction is the average of all the tree predictions.

Advantages and Disadvantages

Advantages

• Robustness: Reduces overfitting and generalizes well due to the law of large numbers.
• Accuracy: Often provides high accuracy because of the ensemble method.
• Versatility: Can be used for both classification and regression tasks.
• Handles Missing Values: Can handle missing data better than many other algorithms.
• Feature Importance: Provides estimates of feature importance, which can be valuable for understanding the model.

Disadvantages

• Complexity: More complex than individual decision trees, making interpretation difficult.
• Computational Cost: Requires more computational resources due to multiple trees.
• Training Time: Can be slow to train compared to simpler models, especially with large datasets.

Hyperparameters

Key Hyperparameters

• n_estimators: The number of trees in the forest.
• max_features: The number of features to consider when looking for the best split.
• max_depth: The maximum depth of the tree.
• min_samples_split: The minimum number of samples required to split an internal node.
• min_samples_leaf: The minimum number of samples required to be at a leaf node.
• bootstrap: Whether bootstrap samples are used when building trees. If False, the whole dataset is used to build each tree.

Tuning Hyperparameters

Hyperparameter tuning can significantly improve the performance of a Random Forest model. Common techniques include Grid Search and Random Search.

Code Examples

Classification Example

Below is a simple example of using Random Forest for a classification task with the Iris dataset.

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report


# Load dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize Random Forest model
clf = RandomForestClassifier(n_estimators=100, random_state=42)

# Train the model
clf.fit(X_train, y_train)

# Make predictions
y_pred = clf.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy * 100:.2f}%")
print("Classification Report:\n", classification_report(y_test, y_pred))

Feature Importance

Random Forest provides a way to measure the importance of each feature in making predictions.

import matplotlib.pyplot as plt

# Get feature importances
importances = clf.feature_importances_
indices = np.argsort(importances)[::-1]

# Print feature ranking
print("Feature ranking:")
for f in range(X.shape[1]):
    print(f"{f + 1}. Feature {indices[f]} ({importances[indices[f]]})")

# Plot the feature importances
plt.figure()
plt.title("Feature importances")
plt.bar(range(X.shape[1]), importances[indices], align='center')
plt.xticks(range(X.shape[1]), indices)
plt.xlim([-1, X.shape[1]])
plt.show()

Hyperparameter Tuning

Using Grid Search for hyperparameter tuning.

from sklearn.model_selection import GridSearchCV

# Define the parameter grid
param_grid = {
    'n_estimators': [100, 200, 300],
    'max_features': ['auto', 'sqrt', 'log2'],
    'max_depth': [4, 6, 8, 10, 12],
    'criterion': ['gini', 'entropy']
}

# Initialize the Grid Search model
grid_search = GridSearchCV(estimator=clf, param_grid=param_grid, cv=3, n_jobs=-1, verbose=2)

# Fit the model
grid_search.fit(X_train, y_train)

# Print the best parameters
print("Best parameters found: ", grid_search.best_params_)

Regression Example

Below is a simple example of using Random Forest for a regression task with the Boston housing dataset.

import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# Load dataset
boston = load_boston()
X, y = boston.data, boston.target

# Split dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize Random Forest model
regr = RandomForestRegressor(n_estimators=100, random_state=42)

# Train the model
regr.fit(X_train, y_train)

# Make predictions
y_pred = regr.predict(X_test)

# Evaluate the model
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f"Mean Squared Error: {mse:.2f}")
print(f"R^2 Score: {r2:.2f}")

Conclusion

Random Forest is a powerful and flexible machine learning algorithm that can handle both classification and regression tasks. Its ability to create an ensemble of decision trees leads to robust and accurate models. However, it is important to be mindful of the computational cost associated with training multiple trees.

References

Scikit-learn Random Forest Documentation Wikipedia: Random Forest Machine Learning Mastery: Introduction to Random Forest Kaggle: Random Forest Guide Towards Data Science: Understanding Random Forests