Bug in WandB's Keras callback when specifying args

Hi, I am using WandB’s Keras callback in autokeras (which uses keras-tuner behind the scenes along with other modules). However, if I specify any args things immediately grind down to a halt.

For reproduction, using the official example would be adequate.

The problem for me is here,

WandbCB = WandbCallback(
    monitor="val_loss", verbose=0, mode="min",
    log_weights=(True), save_model=(True),
    validation_data=validation,
    predictions=5, generator=validation, input_type='images', output_type='images',
    log_evaluation=(True), validation_steps=None, class_colors=None,
)

model.fit(x=training, validation_data=validation, batch_size=BATCH_SIZE, shuffle=True, callbacks=[WandCB, EStop])

However, if I specify WandB callback without any args,

model.fit(x=training, validation_data=validation, batch_size=BATCH_SIZE, shuffle=True, callbacks=[WandbCallback(), EStop])

it works very well.

This is the error I am getting in the former case,

2021-12-31 19:18:38.582746: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-12-31 19:18:38.583261: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199995000 Hz
Traceback (most recent call last):
  File "/usr/local/envs/ak_env/lib/python3.7/site-packages/keras_tuner/engine/tuner.py", line 287, in _deepcopy_callbacks
    callbacks = copy.deepcopy(callbacks)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 216, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle _thread.RLock objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "nas.py", line 125, in <module>
    log_evaluation=True, validation_steps=None, class_colors=None), EStop])
  File "/usr/local/envs/ak_env/lib/python3.7/site-packages/autokeras/auto_model.py", line 291, in fit
    **kwargs
  File "/usr/local/envs/ak_env/lib/python3.7/site-packages/autokeras/engine/tuner.py", line 175, in search
    new_callbacks = self._deepcopy_callbacks(callbacks)
  File "/usr/local/envs/ak_env/lib/python3.7/site-packages/keras_tuner/engine/tuner.py", line 293, in _deepcopy_callbacks
    "It is not possible to do `copy.deepcopy(%s)`" % (callbacks,)
ValueError: All callbacks used during a search should be deep-copyable (since they are reused across trials). It is not possible to do `copy.deepcopy([<wandb.integration.keras.keras.WandbCallback object at 0x7f399310b690>, <tensorflow.python.keras.callbacks.EarlyStopping object at 0x7f39918b1710>])`

wandb: Waiting for W&B process to finish, PID 6373... (failed 1). Press ctrl-c to abort syncing.

Does anyone have any idea?

EDIT- This is the EarlyStopping snippet

EStop = tf.keras.callbacks.EarlyStopping(
    monitor='MAPEMetric', min_delta=2, patience=3, verbose=0, mode="min", baseline=100, #baseline is 100
    restore_best_weights=True)

The difference is perhaps the metrics being monitored in both - MAPEMetric is a simple custom TF metric that computes the “Mean Absolute Percentage Error”;

def MAPEMetric(target, output):
        return tf.math.reduce_mean(tf.math.abs((output - target) / output)) * 100   #100 for %

Hello Neel,

Unfortunately I was not able to reproduce your issue. Could you possibly send a minimal reproduction in a colab?

Thanks,

Nate

Hi Neel,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Thanks,

Nate