Sweep.best_run() method does not return the best run

[shraddha-a]

Hi,

I’m using Weights & Biases Sweeps to run and track hyperparameter optimization for an XGBoost model.

I’ve defined the sweep configuration like so,

sweep_configs = {
    "method": "random",
    "metric": {"name": "Mean_RMSE_score", "goal": "minimize"},
    "parameters": {
        'max_depth': {"values": [4, 9, 14, 19, 29, 34, 44, 49, 54, 59]},
        'learning_rate': {"values": [0.05, 0.0697, 0.0973, 0.1357, 0.1893, 0.2641, 0.3684, 0.5139, 0.7169, 1.0]},
        'objective': {"values": ['reg:squarederror']}, 
        'reg_alpha': {"values": [0.001, 0.0027, 0.0046,0.0077]},
        'reg_lambda': {"values": [0.0010, 0.0046, 0.0129, 0.0215, 0.0359, 0.0599, 0.1]},
        'min_child_weight': {"values": [0.1802, 0.3246, 0.5848, 1.0536, 3.42, 6.1616, 11.1009, 20.0]},
        'random_state': {"values": [46]},
    }
}

For each iteration in the random search process, I perform K-fold cross-validation and log the mean RMSE score to the run using

wandb.log({"Mean_RMSE_score": score})

I then use sweep.best_run() to fetch the run (& hyperparam config) with the lowest mean RMSE score. However, the best_run() method does not return the run with the lowest RMSE score. In fact, it is returning a run, that has the 5th lowest RMSE score. Below is a screenshot of some runs in one of my sweeps.

As a result, I’m not getting the best set of hyperparameters to build the optimum model.

Here is the code where I access the sweep by its sweep ID and use best_run()

def get_best_run(sweep_id):
  api = wandb.Api()
  sweep = api.sweep(f"{SETTINGS['WANDB_ENTITY']}/{SETTINGS['WANDB_PROJECT']}/{sweep_id}")
  
  best_run = sweep.best_run()
  run_id = best_run.id

  with  init_wandb_run(name = "best_hpo_run",
                              group = "train",
                              job_type = "hpo",
                              run_id = run_id,
                              resume = "must"
                              ) as run:

        best_config = dict(run.config)

return best_config

What could be going wrong here?

[luis_bergua]

Hi @shraddha-a, I just checked with the following code and the run is being returned properly. Would you mind executing it and seeing if it works for you? Also, what is your wandb --version?

import wandb
import numpy as np
import random
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import xgboost as xgb

# Define sweep config
sweep_configuration = {
    "method": "random",
    "metric": {"name": "Mean_RMSE_score", "goal": "minimize"},
    "parameters": {
        'max_depth': {"values": [4, 9, 14, 19, 29, 34, 44, 49, 54, 59]},
        'learning_rate': {"values": [0.05, 0.0697, 0.0973, 0.1357, 0.1893, 0.2641, 0.3684, 0.5139, 0.7169, 1.0]},
        'objective': {"values": ['reg:squarederror']}, 
        'reg_alpha': {"values": [0.001, 0.0027, 0.0046,0.0077]},
        'reg_lambda': {"values": [0.0010, 0.0046, 0.0129, 0.0215, 0.0359, 0.0599, 0.1]},
        'min_child_weight': {"values": [0.1802, 0.3246, 0.5848, 1.0536, 3.42, 6.1616, 11.1009, 20.0]},
        'random_state': {"values": [46]},
    }
}

sweep_id = wandb.sweep(sweep=sweep_configuration, project="sweep_best_run_test")

def main():
    run = wandb.init()
    config = run.config

    # Generate random data
    X = np.random.rand(1000, 10)
    y = np.random.rand(1000)

    # Split the data
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=config.random_state)

    # Define the model
    model = xgb.XGBRegressor(max_depth=config.max_depth, learning_rate=config.learning_rate, 
                             objective=config.objective, reg_alpha=config.reg_alpha, 
                             reg_lambda=config.reg_lambda, min_child_weight=config.min_child_weight, 
                             random_state=config.random_state)

    # Train the model
    model.fit(X_train, y_train)

    # Make predictions
    y_pred = model.predict(X_test)

    # Calculate RMSE
    rmse = np.sqrt(mean_squared_error(y_test, y_pred))

    # Log the RMSE
    wandb.log({"Mean_RMSE_score": rmse})

wandb.agent(sweep_id, function=main, count=20)

[luis_bergua]

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

[shraddha-a]

Hi @luis_bergua1 ,

Thanks for your response. I wrote my own function which was manually fetching all runs in a sweep and picking the best one with the lowest Mean_RMSE_score.

I’ve not had a chance to look at your solution, but I might try it later.

You can close this support ticket.

Thanks once again!