RuntimeError: expected scalar type Float but found BFloat16

[yusskkt]

what i input :
ns-train gaussctrl --load-checkpoint unedited_models/bear/splatfacto/2024-12-20_113611/nerfstudio_models/step-000029999.ckpt --experiment-name bear --output-dir outputs --pipeline.datamanager.data data/bear --pipeline.edit_prompt "a photo of a polar bear in the forest" --pipeline.reverse_prompt "a photo of a bear statue in the forest" --pipeline.guidance_scale 5 --pipeline.chunk_size 3 --pipeline.langsam_obj 'bear' --viewer.quit-on-train-completion True

Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████| 7/7 [00:01<00:00, 5.32it/s]
Attempting to load checkpoint from: unedited_models/bear/splatfacto/2024-12-20_113611/nerfstudio_models/step-000029999.ckpt
Done loading Nerfstudio checkpoint from
unedited_models/bear/splatfacto/2024-12-20_113611/nerfstudio_models/step-000029999.ckpt
Rendering view 0
Traceback (most recent call last):
File "/root/data1/miniconda3/envs/gsctrl/bin/ns-train", line 8, in
sys.exit(entrypoint())
^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint
main(
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/scripts/train.py", line 247, in main
launch(
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/scripts/train.py", line 189, in launch
main_func(local_rank=0, world_size=world_size, config=config)
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/scripts/train.py", line 99, in train_loop
trainer.setup()
File "/root/data1/code/gaussctrl/gaussctrl/gc_trainer.py", line 76, in setup
self.pipeline.render_reverse()
File "/root/data1/code/gaussctrl/gaussctrl/gc_pipeline.py", line 134, in render_reverse
rendered_image = self._model.get_outputs_for_camera(current_cam)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/code/gaussctrl/gaussctrl/gc_model.py", line 232, in get_outputs_for_camera
outs = self.get_outputs(camera.to(self.device))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/code/gaussctrl/gaussctrl/gc_model.py", line 140, in get_outputs
self.xys, depths, self.radii, conics, num_tiles_hit, cov3d = project_gaussians( # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/gsplat/project_gaussians.py", line 59, in project_gaussians
return _ProjectGaussians.apply(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/torch/autograd/function.py", line 574, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/gsplat/project_gaussians.py", line 109, in forward
) = _C.project_gaussians_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/gsplat/cuda/init.py", line 9, in call_cuda
return getattr(_C, name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expected scalar type Float but found BFloat16

How can I solve this problem? Thanks a lot!

[jingwu2121]

Hi, please try manually set the variable type to float16 in

File "/root/data1/code/gaussctrl/gaussctrl/gc_pipeline.py", line 134, in render_reverse
rendered_image = self._model.get_outputs_for_camera(current_cam)

like this

current_cam.to(torch.float16())

[yusskkt]

Thank you very much for your reply! I changed the variable to float16 and tried to run it again, and then an error occurred.
change：
current_cam = self.datamanager.cameras[cam_idx].to(self.device).to(torch.float16)

  if current_cam.metadata is None:
      current_cam.metadata = {}
  current_cam.metadata["cam_idx"] = cam_idx
  rendered_image = self._model.get_outputs_for_camera(current_cam)

error：
Attempting to load checkpoint from: unedited_models/bear/splatfacto/2024-12-20_113611/nerfstudio_models/step-000029999.ckpt
Done loading Nerfstudio checkpoint from
unedited_models/bear/splatfacto/2024-12-20_113611/nerfstudio_models/step-000029999.ckpt
Rendering view 0
Traceback (most recent call last):
File "/root/data1/miniconda3/envs/gsctrl/bin/ns-train", line 8, in
sys.exit(entrypoint())
^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint
main(
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/scripts/train.py", line 247, in main
launch(
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/scripts/train.py", line 189, in launch
main_func(local_rank=0, world_size=world_size, config=config)
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/scripts/train.py", line 99, in train_loop
trainer.setup()
File "/root/data1/code/gaussctrl/gaussctrl/gc_trainer.py", line 76, in setup
self.pipeline.render_reverse()
File "/root/data1/code/gaussctrl/gaussctrl/gc_pipeline.py", line 132, in render_reverse
current_cam = self.datamanager.cameras[cam_idx].to(self.device).to(torch.float16)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/utils/tensor_dataclass.py", line 265, in to
return self._apply_fn_to_fields(lambda x: x.to(device))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/utils/tensor_dataclass.py", line 310, in _apply_fn_to_fields
return dataclasses.replace(self_dc, **new_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/dataclasses.py", line 1503, in replace
return obj.class(**changes)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/cameras/cameras.py", line 152, in init
self.height = self._init_get_height_width(height, self.cy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/data1/miniconda3/envs/gsctrl/lib/python3.11/site-packages/nerfstudio/cameras/cameras.py", line 245, in _init_get_height_width
assert not torch.is_floating_point(h_w), f"height and width tensor must be of type int, not: {h_w.dtype}"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: height and width tensor must be of type int, not: torch.float16