Include the token usage for every conversation and workspace

[aponcedeleonch]

Closes: #418

This PR does introduces the changes necessary to track the used tokens per request, process them, and to return them in the API.

Specific changes:

• Make sure we process all the stream and record to DB at the very end
• Include the flag "stream_options": {"include_usage": True}, so the providers respond with the tokens
• Added the necessary processing for the API
• Modified the initial API models to display correctly the tokens and its price

Notable:

• Ollama and Llama CPP tokens cannot be obtained with the current code. It's not critical since these are local providers and shouldn't represent any cost in tokens
  • Ollama: If we switch our requests to not use the Python client but directly HTTP requests we would be able to obtain the tokens
  • Llama CPP: I couldn't find a way to get the tokens when streaming the response.
• Copilot is not returning the used tokens. Also not critical since Copilot charges a fixed price and not per token

Sample response

curl -Ss 'http://localhost:8989/api/v1/workspaces/tokens/token-usage' | jq
{
  "tokens_by_model": {
    "hosted_vllm/unsloth/Qwen2.5-Coder-32B-Instruct": {
      "provider_type": "vllm",
      "model": "hosted_vllm/unsloth/Qwen2.5-Coder-32B-Instruct",
      "token_usage": {
        "input_tokens": 30,
        "output_tokens": 10,
        "input_cost": 0.0,
        "output_cost": 0.0
      }
    },
    "gpt-4o-mini": {
      "provider_type": "openai",
      "model": "gpt-4o-mini",
      "token_usage": {
        "input_tokens": 8,
        "output_tokens": 10,
        "input_cost": 0.0000012,
        "output_cost": 0.000006
      }
    },
    "claude-3-5-sonnet-latest": {
      "provider_type": "anthropic",
      "model": "claude-3-5-sonnet-latest",
      "token_usage": {
        "input_tokens": 471,
        "output_tokens": 35,
        "input_cost": 0.001413,
        "output_cost": 0.000525
      }
    }
  },
  "token_usage": {
    "input_tokens": 509,
    "output_tokens": 55,
    "input_cost": 0.0014142,
    "output_cost": 0.000531
  }
}