Training crashes on 2nd epoch with ValueError: not enough image data

W&B Help
1

I am trying to train a model but on the second epoch training crashes with the following error that is raised inside the wandb site-package:

----> 1 history, model = train_model()

Cell In [11], line 54, in train_model()
     51 early_stopping = EarlyStopping(monitor="val_accuracy", patience=3)
     53 checkpoint = ModelCheckpoint("my_tiny_model", save_weights_only=True)
---> 54 history = model.fit(train_dataset,
     55         epochs=config.epochs, 
     56         validation_data=val_dataset, 
     57         callbacks=[tensorboard_callback, wandb_callback, checkpoint, early_stopping])
     59 wandb.finish()
     60 return history, model

File d:\Miniconda\envs\tiny_cnn\lib\site-packages\wandb\integration\keras\keras.py:174, in patch_tf_keras.<locals>.new_v2(*args, **kwargs)
    172     for cbk in cbks:
    173         set_wandb_attrs(cbk, val_data)
--> 174 return old_v2(*args, **kwargs)

File d:\Miniconda\envs\tiny_cnn\lib\site-packages\wandb\integration\keras\keras.py:174, in patch_tf_keras.<locals>.new_v2(*args, **kwargs)
    172     for cbk in cbks:
    173         set_wandb_attrs(cbk, val_data)
--> 174 return old_v2(*args, **kwargs)
...
--> 798     raise ValueError("not enough image data")
    799 if s[1] != 0:
    800     raise ValueError("cannot decode image data")

ValueError: not enough image data

Any idea on what needs to be fixed? Thanks for your support.

2

Hi Susanne,

Thanks for writing in! Could you share a snippet of the code that you are using? Also, could you try running this code without the wandb_callback and check if the same error raises? Thanks!

Best,
Luis

3

Hi Susanne,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Luis

4

Due to several code fixes I am no longer able to reproduce the original error. Right now my training crashes with an error:

“”"
wandb.termwarn( “No validation_data set, pass a generator to the callback.”
“”"

If I run the code without the wandb callback everything runs fine. My suspicion is that this is due to the fact that I use tf.data to stream my data and not an ImageGenerator.

Code with disabled callback is available here: https://github.com/subrockmann/tiny_cnn/blob/master/Experiment%20Workbench.ipynb

5

Hi Susanne,

Thanks for your answer! I cannot access the code since I am not able to find tiny_cnn in your profile, could you share if it is private or send me a code snippet please? Thanks!

Best,
Luis

6

Hi Susanne,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Luis

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.