[Bug] Did Dspy totally deprecate HFClientTGI or is the documentation incorrect?

[Aaditya-Bhatia]

What happened?

I am following the documentation to connect to TGI, and get the following error:

AttributeError: module 'dspy' has no attribute 'HFClientTGI'

I couldnt find any implementation of HFClientTGI, and am wondering how connect to a local model. Essentially I finetuned a model saved at a checkpoint directory, and want to infer it via dspy's prompt compilation. Not sure which documentation to trust?

Steps to reproduce

tgi_llama2 = dspy.HFClientTGI(model=model_path, port=8080, url="http://localhost")

DSPy version

2.6.5

[thangld201]

Seems likely. Can you just use lm = dspy.LM(..); dspy.configure(lm=lm) ? I recall TGI does support openai format compatible with dspy.LM.

[okhat]

Yes it's gone. See the landing page dspy.ai for how to use local models!

[Aaditya-Bhatia]

to serve models from a disk (which is the case for me since I finetuned a model), in TGI you have to provide the model id as folder location.
e.g.,

sudo docker run --gpus all --runtime=nvidia --shm-size 1g -p 8080:80 -v my_model_checkpoint:/data ghcr.io/huggingface/text-generation-inference:3.1.0 --model-id /data   # here /data is the model id. 

Now, DSPY doesnt like this /data as the model id. For instance, the code

import dspy
lm = dspy.LM(
    model="/data",
    api_base="http://127.0.0.1:8080/v1/",  # TGI API endpoint for your fine-tuned model
    model_type="chat"
)
dspy.configure(lm=lm)

will throw an error: BadRequestError: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=\data

I'm in a deadlock, not being able to bypass TGI's requirements of putting a local folder as model_id. So I am thinking of moving to SGLang, as explained in the landing page. Please let me know if you guys have any suggestions, or general help using TGI. Thanks!
Also, what made you guys kill the HFClientTGI Class anyway? Would it make sense to remove the outdated documentation? It took about a day to install TGI server (since it has a hefty build process), only to know that the dspy documentation was incorrect, and its not supported :(

[okhat]

TGI is absolutely supported! Just launch the server and use dspy.LM as the client.