Robustness to Target Noise

Diagnostics

Procedure

This procedure evaluates the robustness of a regression model by analyzing its performance variance when Gaussian noise is added to the target variable. This helps determine whether the model is sensitive to small perturbations in the target data, which could indicate overfitting or fragility.

Define Parameters for Gaussian Noise
- What to do: Choose the parameters for the Gaussian noise to be added to the target variable.
  - Select a mean (e.g., 0) and standard deviation (e.g., a fraction of the target variable’s standard deviation) for the Gaussian noise.
  - Ensure the noise level is small enough to simulate realistic perturbations while effectively testing robustness.
Add Gaussian Noise to the Target Variable
- What to do: Create multiple noisy versions of the target variable in the test dataset.
  - Apply Gaussian noise to the target values using the defined parameters.
  - Generate at least 5-10 variations of the test dataset, each with a different noise instance, to cover a range of perturbations.
Evaluate the Model on Noisy Datasets
- What to do: Test the model on each test dataset with noisy target variables.
  - Use the same test set features and performance metric as the original evaluation.
  - Keep the model and all configurations constant across evaluations to isolate the effect of noise in the target variable.
Record Performance Metrics
- What to do: Capture the performance metrics for the model on each noisy dataset.
  - Organize the results in a structured format, such as a table or spreadsheet.
  - Include metrics such as R-squared, Mean Absolute Error (MAE), or Root Mean Squared Error (RMSE), depending on the problem context.
Calculate Variance in Performance
- What to do: Compute the variance (or standard deviation) of the performance metrics across all noisy datasets.
  - Use statistical tools or libraries (e.g., Pandas, NumPy) to calculate the variance.
  - Ensure the calculations are consistent with the chosen performance metric.
Interpret Variance Results
- What to do: Analyze the variance in model performance.
  - Low variance indicates the model is robust to small perturbations in the target variable.
  - High variance suggests the model is sensitive to changes, which could signal overfitting or a lack of generalization.
Report the Findings
- What to do: Summarize the results and their implications.
  - Present the computed variance alongside the individual performance metrics for each noisy dataset.
  - Highlight any significant performance drops and provide recommendations, such as refining the model, collecting more diverse data, or improving regularization.

Code Example

This Python function evaluates whether a regression model demonstrates robustness by analyzing the variance in performance metrics when Gaussian noise is added to the target variable during evaluation.

import numpy as np

def robustness_with_target_noise_test(X, y, model, metric, noise_levels, n_runs):
    """
    Evaluate a regression model's robustness by analyzing performance variance when Gaussian noise is added to the target variable.

    Parameters:
        X (np.ndarray): Features of the test dataset.
        y (np.ndarray): Target variable of the test dataset.
        model: Pre-trained machine learning model to evaluate.
        metric (callable): Performance metric function (e.g., mean_squared_error).
        noise_levels (list): List of standard deviations for Gaussian noise to test.
        n_runs (int): Number of noisy datasets to generate for each noise level.

    Returns:
        dict: Dictionary containing noise level, variance, mean performance, and interpretation for each noise level.
    """
    results = []

    for noise_std in noise_levels:
        performance_scores = []

        for _ in range(n_runs):
            # Add Gaussian noise to the target variable
            noisy_y = y + np.random.normal(0, noise_std, y.shape)

            # Predict and evaluate the model
            y_pred = model.predict(X)
            score = metric(noisy_y, y_pred)
            performance_scores.append(score)

        # Calculate variance and mean performance
        variance = np.var(performance_scores)
        mean_performance = np.mean(performance_scores)

        # Interpret the results
        if variance < 0.01:
            interpretation = "High robustness: very low performance variance at this noise level."
        elif variance < 0.05:
            interpretation = "Moderate robustness: acceptable performance variance at this noise level."
        else:
            interpretation = "Low robustness: high sensitivity to noise at this level."

        results.append({
            "Noise Level (Std Dev)": noise_std,
            "Variance": variance,
            "Mean Performance": mean_performance,
            "Interpretation": interpretation,
        })

    return results

# Demo Usage
if __name__ == "__main__":
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.datasets import make_regression
    from sklearn.metrics import mean_squared_error

    # Generate synthetic dataset
    X, y = make_regression(
        n_samples=500, n_features=10, noise=0.1, random_state=42
    )

    # Split into train and test sets
    from sklearn.model_selection import train_test_split
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    # Train the model
    model = RandomForestRegressor(n_estimators=10, random_state=42)
    model.fit(X_train, y_train)

    # Perform robustness test with Gaussian noise on the target variable
    results = robustness_with_target_noise_test(X_test, y_test, model,
        mean_squared_error, [0.0, 0.1, 0.5, 1.0, 2.0], 30)

    # Print results
    print("Robustness Test Results with Gaussian Noise on Target Variable:")
    for res in results:
        for key, value in res.items():
            print(f"{key}: {value}")
        print()

Example Output

Robustness Test Results with Gaussian Noise on Target Variable:
Noise Level (Std Dev): 0.0
Variance: 3.308722450212111e-24
Mean Performance: 5300.216962401416
Interpretation: High robustness: very low performance variance at this noise level.

Noise Level (Std Dev): 0.1
Variance: 2.8648902415964153
Mean Performance: 5300.120893138741
Interpretation: Low robustness: high sensitivity to noise at this level.

Noise Level (Std Dev): 0.5
Variance: 44.663879097002884
Mean Performance: 5303.400355960043
Interpretation: Low robustness: high sensitivity to noise at this level.

Noise Level (Std Dev): 1.0
Variance: 192.11738194847163
Mean Performance: 5303.341526708394
Interpretation: Low robustness: high sensitivity to noise at this level.

Noise Level (Std Dev): 2.0
Variance: 732.5701087060801
Mean Performance: 5303.484340691761
Interpretation: Low robustness: high sensitivity to noise at this level.

Robustness to Input Noise Robustness to Label Noise