Why is min and max causing errors when logging gradients for biases in model?

Why is this error happening when wandb is logging the grad.data field?

  File "/home/miranda9/automl-meta-learning/automl-proj-src/experiments/meta_learning/main_metalearning.py", line 360, in <module>
    main(args)
  File "/home/miranda9/automl-meta-learning/automl-proj-src/experiments/meta_learning/main_metalearning.py", line 333, in main
    meta_train_fixed_iterations_full_epoch_possible(args)
  File "/home/miranda9/automl-meta-learning/automl-proj-src/meta_learning/training/meta_training.py", line 216, in meta_train_fixed_iterations_full_epoch_possible
    log_train_val_stats(args, args.it, train_loss, train_acc, valid=meta_eval, bar=bar_it,
  File "/home/miranda9/automl-meta-learning/automl-proj-src/meta_learning/training/meta_training.py", line 129, in log_train_val_stats
    val_loss, val_acc = valid(args, save_val_ckpt=save_val_ckpt)
  File "/home/miranda9/automl-meta-learning/automl-proj-src/meta_learning/training/meta_training.py", line 274, in meta_eval
    eval_loss, eval_acc = args.meta_learner(spt_x, spt_y, qry_x, qry_y)
  File "/home/miranda9/miniconda3/envs/metalearning_cpu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/miranda9/automl-meta-learning/automl-proj-src/meta_learning/meta_learners/maml_meta_learner.py", line 159, in forward
    (qry_loss_t / meta_batch_size).backward()  # note this is more memory efficient (as it removes intermediate data that used to be needed since backward has already been called)
  File "/home/miranda9/miniconda3/envs/metalearning_cpu/lib/python3.9/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/miranda9/miniconda3/envs/metalearning_cpu/lib/python3.9/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(
  File "/home/miranda9/miniconda3/envs/metalearning_cpu/lib/python3.9/site-packages/wandb/wandb_torch.py", line 285, in <lambda>
    handle = var.register_hook(lambda grad: _callback(grad, log_track))
  File "/home/miranda9/miniconda3/envs/metalearning_cpu/lib/python3.9/site-packages/wandb/wandb_torch.py", line 283, in _callback
    self.log_tensor_stats(grad.data, name)
  File "/home/miranda9/miniconda3/envs/metalearning_cpu/lib/python3.9/site-packages/wandb/wandb_torch.py", line 235, in log_tensor_stats
    tensor = flat.histc(bins=self._num_bins, min=tmin, max=tmax)
RuntimeError: max must be larger than min

I am not doing anything myself so I am unsure how I can fix this…

Hmm, this appears to be a new bug. We fixed a similar RunTimeError a while back, so it’s surprising to me that this is happening. For now, turning off gradient logging in watch (set wandb.watch(log=None)) will prevent this from triggering.

I suggest you raise this issue with W&B Support via the small gray bubble in the bottom-right of the screen on https://wandb.ai. If you don’t see a bubble, turn off any ad-blockers or email support@wandb.com.