Hi,
I am training a neural network and I have to do hyperparameter optimization.
For good results of my model, I need to train with large datasets and the training becomes slow, e.g. 1 hour per epoch.
I want to do a parameter sweep, but I am wondering if I could do a sweep with a smaller dataset such that it becomes faster and I can do more sweep runs.
Is this a good idea in general?
Thank you in advance for your answer!
Hi @p-scheepers , if you update your training function to include a smaller subset of the data, then yes you can sweep over this function. This approach has its advantages and drawbacks, so it’s crucial to consider them carefully.
Advantages
- Faster Iterations: Reducing the size of the dataset can significantly decrease training time per epoch, allowing for quicker iterations over different hyperparameter configurations.
- Resource Efficiency: It saves computational resources, which is beneficial if you have limited access to hardware or are trying to optimize costs.
- Early Insights: You can gain early insights into which hyperparameters are more promising, helping to narrow down the search space for more intensive training later on.
Drawbacks
- Reduced Generalization: A smaller dataset may not represent the full complexity and variability of the entire dataset, leading to models that perform well on the smaller set but poorly on the full dataset.
- Overfitting Risks: Training on a smaller dataset increases the risk of overfitting to that specific subset of data, especially if the dataset is not carefully curated to maintain a representative sample.
- Optimization Bias: The optimal set of hyperparameters found using a smaller dataset might not be optimal for the full dataset. This mismatch can lead to suboptimal performance when the model is eventually trained on the entire dataset.
Hi @p-scheepers, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!