Parameter sweep small dataset

Hi,

I am training a neural network and I have to do hyperparameter optimization.
For good results of my model, I need to train with large datasets and the training becomes slow, e.g. 1 hour per epoch.
I want to do a parameter sweep, but I am wondering if I could do a sweep with a smaller dataset such that it becomes faster and I can do more sweep runs.
Is this a good idea in general?

Thank you in advance for your answer!

Hi @p-scheepers , if you update your training function to include a smaller subset of the data, then yes you can sweep over this function. This approach has its advantages and drawbacks, so it’s crucial to consider them carefully.

Advantages

  1. Faster Iterations: Reducing the size of the dataset can significantly decrease training time per epoch, allowing for quicker iterations over different hyperparameter configurations.
  2. Resource Efficiency: It saves computational resources, which is beneficial if you have limited access to hardware or are trying to optimize costs.
  3. Early Insights: You can gain early insights into which hyperparameters are more promising, helping to narrow down the search space for more intensive training later on.

Drawbacks

  1. Reduced Generalization: A smaller dataset may not represent the full complexity and variability of the entire dataset, leading to models that perform well on the smaller set but poorly on the full dataset.
  2. Overfitting Risks: Training on a smaller dataset increases the risk of overfitting to that specific subset of data, especially if the dataset is not carefully curated to maintain a representative sample.
  3. Optimization Bias: The optimal set of hyperparameters found using a smaller dataset might not be optimal for the full dataset. This mismatch can lead to suboptimal performance when the model is eventually trained on the entire dataset.

Hi @p-scheepers, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!