Change N/A config fields after run was completed

[tim-kuipers]

I am running a couple of sweeps and in between sweeps I’ve added a new type of augmentation for which I’ve added a config parameter do_augmentation.

In the workspace I’ve added a parameter importance panel, but now it lists the most important parameter as do_augmentation.value_NaN

I would like to fill in the do_augmentation field for all the runs I’ve performed before I’ve added this setting to some specific value (False, or zero, depends).

How would I do that?
How would I see the accurate importance and correlation between the model performance and the hyperparameter which got added midway in between some sweeps?

[artsiom]

Hi tim!

Thank you for writing in! When you say you add a new variable to your sweep, do you attach it in the middle of a sweep to your sweep config?

Could you send me some of the minimal reproduction code of what you are doing on your side?

[tim-kuipers]

What I mean is that i run a sweep, interrupt or finish it, then add a new variable (eg do_aug) to the sweep config, and then run another sweep.
The results of the first sweep then get a new column for do_aug with the value set to NaN. How can I change those values to False?

[artsiom]

Hi Tim! Could you send me a link to your workspace where you are seeing this behavior on your side? When you run another sweep, does it also have Nan values, or just the first one?

[artsiom]

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

[artsiom]

Hi Tim, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

[tim-kuipers]

Here you go. I’ve followed the reproduction steps for you so you don’t have to.
Can you take my request seriously now?

tim-kuipers

[tim-kuipers]

Here’s the code:

import wandb
import time

sweep_config = {
    'method': 'random'
    }
parameters_dict = {
    'optimizer': {
        'values': ['adam', 'sgd']
        },
    'fc_layer_size': {
        'values': [128, 256, 512]
        },
    'dropout': {
          'values': [0.3, 0.4, 0.5]
        },
    }

sweep_config['parameters'] = parameters_dict

sweep_id = wandb.sweep(sweep_config, project="pytorch-sweeps-demo", entity='tim-kuipers')

def train(config=None):
    # Initialize a new wandb run
    with wandb.init(config=config):
        # If called by wandb.agent, as below,
        # this config will be set by Sweep Controller
        config = wandb.config
        print("body of train function")
        time.sleep(1)

wandb.agent(sweep_id, train, count=5)
wandb.finish()

The first couple of runs the dropout variable didn’t exist yet.

[tim-kuipers]

I’ve made very easy to follow reproduction steps. Did you try them? What’s the status on this? Has it been reopened or not?

[mohammadbakir]

Hi @tim-kuipers , stepping in for Artsiom to assist. Thank you for providing the code snippet and link to your workspace.

I ran through multiple tests and did not encounter the behavior described , The first couple of runs the dropout variable didn’t exist yet..

Sweep 1 - Ran to completion with the default config params provided above. All runs reflect the configuration provided.

Sweep 2 - Initiated new sweep with updated config to include do_aug with True/False values. Ran to completion and all runs reflect the config provided.

Sweep 3 - Initialized sweep with default config above, interrupted the sweep, attempted to update the config locally to include, include do_aug with True/False and resume sweep using a sweep agent, wandb agent mohammadbakir/timk-sweep-test/801u9jlh. Wandb will ignore this because modifying a sweep config is not supported. The following will be printed to the terminal, WARNING Changes to your wandb environment variables will be ignored because your wandb session has already started. For more information on how to modify your settings with wandb.

Additionally, any new sweep that includes config params that do not exist in old sweeps has no impact on the graphs of the old sweeps. This new sweep includes do_aug_2

yet sweep 1 does not and it’s sweep graph is not changed

Could you provide a direct link of an example where you see this per your comment

Lastly, you cannot directly add/update a config value in the UI for a specific run. This can be done through the api,

api = wandb.Api()
run = api.run("entity/project/run-id>")
run.config["<config>"] = <val>
run.update()

However, updating a run config value during an ongoing sweep or after its completion will not affect the parameter importance. Parameter importance is determined based on the run results produced from the configuration values used to initialize the sweep

[tim-kuipers]

Thank you very much for looking into this.

I think we are looking at it from different screens. What I mean is the parameter importance plot inside the project workspace, rather than the sweep workspace.

Updated code:

import wandb
import time
import random as rnd

sweep_config = {
    'method': 'grid'
    }
parameters_dict = {
    'optimizer': {
        'values': ['adam']
        },
    # 'do_aug': {
        # 'values': [False,True]
        # },
    }

sweep_config['parameters'] = parameters_dict

sweep_id = wandb.sweep(sweep_config, project="pytorch-sweeps-demo", entity='tim-kuipers')

def train(config=None):
    # Initialize a new wandb run
    with wandb.init(config=config):
        # If called by wandb.agent, as below,
        # this config will be set by Sweep Controller
        config = wandb.config
        print("body of train function")
        time.sleep(1)
        wandb.log({"loss": rnd.random()})

wandb.agent(sweep_id, train, count=5)
wandb.finish()

Just run to completion, and then a second run with do_aug uncommented.

Gives this: Weights & Biases

Problem is that I want to map do_aug.value_NaN to do_aug.value_False so that I can get accurate parameter importance statistics.

By your last comment I conclude that this is currently impossible. If so, please regard this issue as a feature request or bug report depending how you look at it.

[mohammadbakir]

Thanks for the additional insight @tim-kuipers . I will mark this as a feature request and keep you updated once there’s been movement on the ticket. Cheers!