Using Seq2rel function with cuda

[jjyhh1223]

Here's how I pass the device argument for function Seq2rel

from seq2rel import Seq2Rel
from seq2rel.common import util
model = 'model.tar.gz'
kwargs = {'cuda_device': 1}
seq2rel = Seq2Rel(model, **kwargs)

and got this error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-15-d951d9bfd1c5>](https://localhost:8080/#) in <cell line: 2>()
      1 kwargs = {'cuda_device': 1}
----> 2 seq2rel = Seq2Rel(model, **kwargs)

12 frames
[/content/drive/MyDrive/seq2rel/seq2rel/seq2rel.py](https://localhost:8080/#) in __init__(self, pretrained_model_name_or_path, **kwargs)
     86         if "overrides" in kwargs:
     87             overrides.update(kwargs.pop("overrides"))
---> 88         archive = load_archive(pretrained_model_name_or_path, overrides=overrides, **kwargs)
     89         self._predictor = Predictor.from_archive(archive, predictor_name="seq2seq")
     90 

[/usr/local/lib/python3.9/dist-packages/allennlp/models/archival.py](https://localhost:8080/#) in load_archive(archive_file, cuda_device, overrides, weights_file)
    233             config.duplicate(), serialization_dir
    234         )
--> 235         model = _load_model(config.duplicate(), weights_path, serialization_dir, cuda_device)
    236 
    237         # Load meta.

[/usr/local/lib/python3.9/dist-packages/allennlp/models/archival.py](https://localhost:8080/#) in _load_model(config, weights_path, serialization_dir, cuda_device)
    277 
    278 def _load_model(config, weights_path, serialization_dir, cuda_device):
--> 279     return Model.load(
    280         config,
    281         weights_file=weights_path,

[/usr/local/lib/python3.9/dist-packages/allennlp/models/model.py](https://localhost:8080/#) in load(cls, config, serialization_dir, weights_file, cuda_device)
    436             # get_model_class method, that recurses whenever it finds a from_archive model type.
    437             model_class = Model
--> 438         return model_class._load(config, serialization_dir, weights_file, cuda_device)
    439 
    440     def extend_embedder_vocab(self, embedding_sources_mapping: Dict[str, str] = None) -> None:

[/usr/local/lib/python3.9/dist-packages/allennlp/models/model.py](https://localhost:8080/#) in _load(cls, config, serialization_dir, weights_file, cuda_device)
    341         # in sync with the weights
    342         if cuda_device >= 0:
--> 343             model.cuda(cuda_device)
    344         else:
    345             model.cpu()

[/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in cuda(self, device)
    686             Module: self
    687         """
--> 688         return self._apply(lambda t: t.cuda(device))
    689 
    690     def xpu(self: T, device: Optional[Union[int, device]] = None) -> T:

[/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    576     def _apply(self, fn):
    577         for module in self.children():
--> 578             module._apply(fn)
    579 
    580         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    576     def _apply(self, fn):
    577         for module in self.children():
--> 578             module._apply(fn)
    579 
    580         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    576     def _apply(self, fn):
    577         for module in self.children():
--> 578             module._apply(fn)
    579 
    580         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    576     def _apply(self, fn):
    577         for module in self.children():
--> 578             module._apply(fn)
    579 
    580         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    576     def _apply(self, fn):
    577         for module in self.children():
--> 578             module._apply(fn)
    579 
    580         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    599             # `with torch.no_grad():`
    600             with torch.no_grad():
--> 601                 param_applied = fn(param)
    602             should_use_set_data = compute_should_use_set_data(param, param_applied)
    603             if should_use_set_data:

[/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in <lambda>(t)
    686             Module: self
    687         """
--> 688         return self._apply(lambda t: t.cuda(device))
    689 
    690     def xpu(self: T, device: Optional[Union[int, device]] = None) -> T:

RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

What is the correct way of using Seq2rel with cuda? Thank you!

[JohnGiorgi]

seq2rel is just pytorch, really, so this suggests a problem with your environment. Does 'cuda_device': 1 exist? (can you see it if you call it nvidia-smi?). Any chance you meant 'cuda_device': 0?

[jjyhh1223]

I tried doing the following,

from seq2rel import Seq2Rel
from seq2rel.common import util
model = 'model.tar.gz'
kwargs = {'cuda_device': 0}
seq2rel = Seq2Rel(model, **kwargs)

It worked, but I'm not sure if it's loaded on cuda because running examples seems very slowly (much slower than validating in training). Other than this, is there any ways to quickly running inference using Seq2rel?

[JohnGiorgi]

To see if the GPU is being utilized, you can run nvidia-smi on the same machine this code is running, and see if GPU utilization is >0%.

In general, the speed of inference will depend on quite a few arguments that you can control with the config or kwargs. max_length and beam_size have the largest effect, so I would try reducing either to see if you get a speed boost without harming performance. use_amp will really help if your GPU supports mixed-precision (and can actually slow things down if it doesn't). The larger the batch_size the better, for inference use the largest size that doesn't cause OOM errors.

The Seq2Rel interface is meant to just be a convenience thing. If you are trying to evaluate on a bunch of examples it might make more sense to use allennlp evaluate. You can see how to use that command here. The advice above on how to set the parameters for fastest inference is the same.