closing this after having couple of b ack and forths:
install nvidia container toolkit
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
and put “gpus”:“all” option in que configuration
1 Like