Pytorch sweep help

[yasarniyaz]

Hi . I want to do a study in pytorch with wandb sweep. I want to try activation function optimization algorithm and lr value with various combinations. but I have no idea how to do this. I did it with keras but not in pytorch. can you help me

[ramit_goolry]

Hi @yasarniyaz ,

We have an example Colab Notebook detailing how to run a sweep in PyTorch. You can access it here.

Thanks,
Ramit

[yasarniyaz]

thank you my friend. I looked at this resource, but it doesn’t show how to change my activation functions here, and frankly, this resource seems very confusing to me. do you have any other suggestions?

[ramit_goolry]

I would say you would have to do some manipulation in your code to get this going. Something like:

if config.activation == 'relu':
  self.activation = nn.ReLU
elif config.activation == 'tanh':
  self.activation = nn.tanh

in your __init__ function. You can then use the self.activation object to set up your activations in your model based on the sweep config.

Thanks,
Ramit