Hi, I am using WandB’s Keras callback in autokeras (which uses keras-tuner behind the scenes along with other modules). However, if I specify any args things immediately grind down to a halt.
For reproduction, using the official example would be adequate.
The problem for me is here,
WandbCB = WandbCallback(
monitor="val_loss", verbose=0, mode="min",
log_weights=(True), save_model=(True),
validation_data=validation,
predictions=5, generator=validation, input_type='images', output_type='images',
log_evaluation=(True), validation_steps=None, class_colors=None,
)
model.fit(x=training, validation_data=validation, batch_size=BATCH_SIZE, shuffle=True, callbacks=[WandCB, EStop])
However, if I specify WandB callback without any args,
model.fit(x=training, validation_data=validation, batch_size=BATCH_SIZE, shuffle=True, callbacks=[WandbCallback(), EStop])
it works very well.
This is the error I am getting in the former case,
2021-12-31 19:18:38.582746: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-12-31 19:18:38.583261: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199995000 Hz
Traceback (most recent call last):
File "/usr/local/envs/ak_env/lib/python3.7/site-packages/keras_tuner/engine/tuner.py", line 287, in _deepcopy_callbacks
callbacks = copy.deepcopy(callbacks)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 216, in _deepcopy_list
append(deepcopy(a, memo))
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 281, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 241, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/envs/ak_env/lib/python3.7/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.RLock objects
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "nas.py", line 125, in <module>
log_evaluation=True, validation_steps=None, class_colors=None), EStop])
File "/usr/local/envs/ak_env/lib/python3.7/site-packages/autokeras/auto_model.py", line 291, in fit
**kwargs
File "/usr/local/envs/ak_env/lib/python3.7/site-packages/autokeras/engine/tuner.py", line 175, in search
new_callbacks = self._deepcopy_callbacks(callbacks)
File "/usr/local/envs/ak_env/lib/python3.7/site-packages/keras_tuner/engine/tuner.py", line 293, in _deepcopy_callbacks
"It is not possible to do `copy.deepcopy(%s)`" % (callbacks,)
ValueError: All callbacks used during a search should be deep-copyable (since they are reused across trials). It is not possible to do `copy.deepcopy([<wandb.integration.keras.keras.WandbCallback object at 0x7f399310b690>, <tensorflow.python.keras.callbacks.EarlyStopping object at 0x7f39918b1710>])`
wandb: Waiting for W&B process to finish, PID 6373... (failed 1). Press ctrl-c to abort syncing.
Does anyone have any idea?
EDIT- This is the EarlyStopping snippet
EStop = tf.keras.callbacks.EarlyStopping(
monitor='MAPEMetric', min_delta=2, patience=3, verbose=0, mode="min", baseline=100, #baseline is 100
restore_best_weights=True)
The difference is perhaps the metrics being monitored in both - MAPEMetric is a simple custom TF metric that computes the “Mean Absolute Percentage Error”;
def MAPEMetric(target, output):
return tf.math.reduce_mean(tf.math.abs((output - target) / output)) * 100 #100 for %