Hi . I want to do a study in pytorch with wandb sweep. I want to try activation function optimization algorithm and lr value with various combinations. but I have no idea how to do this. I did it with keras but not in pytorch. can you help me
Hi @yasarniyaz ,
We have an example Colab Notebook detailing how to run a sweep in PyTorch. You can access it here.
thank you my friend. I looked at this resource, but it doesn’t show how to change my activation functions here, and frankly, this resource seems very confusing to me. do you have any other suggestions?
I would say you would have to do some manipulation in your code to get this going. Something like:
if config.activation == 'relu': self.activation = nn.ReLU elif config.activation == 'tanh': self.activation = nn.tanh
__init__ function. You can then use the
self.activation object to set up your activations in your model based on the sweep config.