🦙 Alternative Models

Anthropic’s Claude 3

To use Anthropic models, provide the ANTHROPIC_API_KEY environment variable instead of OPENAI_API_KEY, and set the model claude-3-opus-20240229 in the .mentat_config.json file:

# in ~/.mentat/.env
ANTHROPIC_API_KEY=sk-*************

# In ~/.mentat/.mentat_config.json
{ "model": "claude-3-opus-20240229" }

OpenAI models on Azure

To use the Azure API, provide the AZURE_OPENAI_ENDPOINT (https://<your-instance-name>.openai.azure.com/) and AZURE_OPENAI_KEY environment variables instead of OPENAI_API_KEY.

In addition, Mentat uses the gpt-4-1106-preview model by default. When using Azure, you will have to set the model as described in Configuration to the name you gave your Azure model.

Warning

Due to changes in the OpenAI Python SDK, you can no longer use OPENAI_API_BASE to access the Azure API with Mentat.

Using Other Models

Mentat uses the OpenAI SDK to retrieve chat completions. This means that setting the OPENAI_API_BASE environment variable is enough to use any model that has the same response schema as OpenAI. To use models with different response schemas, we recommend setting up a litellm proxy as described here and pointing OPENAI_API_BASE to the proxy. For example:

pip install 'litellm[proxy]'
litellm --model huggingface/bigcode/starcoder --drop_params
# Should see: Uvicorn running on http://0.0.0.0:8000

# In ~/.mentat/.env
OPENAI_API_BASE=http://localhost:8000
# or
export OPENAI_API_BASE=http://localhost:8000

🦙 Local Models

This works the same as in the previous section but you must install ollama first. Replace mixtral with whichever model you want to use.

ollama pull mixtral
ollama serve
# Should see: listening on 127.0.0.1:11434

Next run the litellm proxy. In another terminal run:

pip install 'litellm[proxy]'
litellm --model ollama/mixtral --api_base http://localhost:11434 --drop_params
# Should see: Uvicorn running on http://0.0.0.0:8000

Finally set the OPENAI_API_BASE in the terminal before running mentat.

Note

When using a litellm proxy, the model set in Mentat’s config will not affect the model being run. To change the model, rerun the litellm proxy with a different model. As mentat thinks it is talking to gpt-4-1106-preview you may want to set maximum_context.

Warning

Be sure to include the –drop_params argument when running the litellm proxy! Mentat uses some arguments (such as response_format) that may not be available in alternative models.