CUDA: add set

[jeemzz147]

Make sure to read the contributing guidelines before submitting a PR

[jeemzz147]

Part of #14909

[jeemzz147]

Hi, @JohannesGaessler Could you please review the changes when you have a chance? Thank you in advance!

[JohannesGaessler]

One of my current goals is to consolidate and deduplicate the code around copying data in the CUDA backend. As such I think rather than adding new kernels here it would be better to re-use the existing code. If the operation is not inlace you can use cudaMemsetAsync to set dst with the contents of src0. Afterwards you can use ggml_cpy_flt_cuda in cpy.cu to do the copy. That kernel does not have an argument for the offset but it's not needed as you can simply apply the offset in host code.

[jeemzz147]

@JohannesGaessler

I have already used cudaMemcpyAsync to copy the contents from src0 to dst.

• Regarding the implementation of ggml_cpy_flt_cuda, the functionality of the set kernel and ggml_cpy_flt_cuda may be inconsistent.

  • ggml_cpy_flt_cuda

  • set_f32_cuda

  • The element size of src1 may be less than in dst.

  • The set kernel doesn’t rely on any implicit shape in dst itself;

  • The dst of the set may be a slice, and when writing from src1 to dst, so it may not be aware of dst->ne. Instead, dst->nb1, nb2, and nb3 are provided by the parameters.

  • Could it be that the data of src1 within dst are not contiguous?

If I’m wrong, please correct me. Thanks for your help!

[jeemzz147]

Hi @am17an, thanks again for your previous review. Since @JohannesGaessler hasn’t had a chance to respond for a few weeks, would it be possible to ask another maintainer or contributor to review this as well? I’d really appreciate any additional feedback to help move this forward.

[JohannesGaessler]

Sorry, I forgot about this PR. The code in cpy.cuh can do the same as the one in this PR (and more), you just need to call it with the right parameters.

The element size of src1 may be less than in dst.

Set both types to float.

The set kernel doesn’t rely on any implicit shape in dst itself;

Use the same shape twice.