Langroid Supported LLMs and Providers

Langroid supports a wide range of Language Model providers through its OpenAIGPTConfig class.

OpenAIGPTConfig is not just for OpenAI models!

The OpenAIGPTConfig class is a generic configuration class that can be used to configure any LLM provider that is OpenAI API-compatible. This includes both local and remote models.

You would typically set up the OpenAIGPTConfig class with the chat_model parameter, which specifies the model you want to use, and other parameters such as max_output_tokens, temperature, etc (see the OpenAIGPTConfig class and its parent class LLModelConfig for full parameter details):

import langroid.language_models as lm
llm_config = lm.OpenAIGPTConfig(
    chat_model="<model-name>", # possibly includes a <provider-name> prefix
    api_key="api-key", # optional, prefer setting in environment variables
    # ... other params such as max_tokens, temperature, etc.
)

Below are chat_model examples for each supported provider. For more details see the guides on setting up Langroid with local and non-OpenAI LLMs. Once you set up the OpenAIGPTConfig, you can then directly interact with the LLM, or set up an Agent with this LLM, and use it by itself, or in a multi-agent setup, as shown in the Langroid quick tour

Although we support specifying the api_key directly in the config (not recommended for security reasons), more typically you would set the api_key in your environment variables. Below is a table showing for each provider, an example chat_model setting, and which environment variable to set for the API key.

Provider	chat_model Example	API Key Environment Variable
OpenAI	gpt-4o	OPENAI_API_KEY
Groq	groq/llama3.3-70b-versatile	GROQ_API_KEY
Cerebras	cerebras/llama-3.3-70b	CEREBRAS_API_KEY
Gemini	gemini/gemini-2.0-flash	GEMINI_API_KEY
DeepSeek	deepseek/deepseek-reasoner	DEEPSEEK_API_KEY
GLHF	glhf/hf:Qwen/Qwen2.5-Coder-32B-Instruct	GLHF_API_KEY
OpenRouter	openrouter/deepseek/deepseek-r1-distill-llama-70b:free	OPENROUTER_API_KEY
Ollama	ollama/qwen2.5	OLLAMA_API_KEY (usually ollama)
VLLM	vllm/mistral-7b-instruct	VLLM_API_KEY
LlamaCPP	llamacpp/localhost:8080	LLAMA_API_KEY
Generic Local	local/localhost:8000/v1	No specific env var required
LiteLLM	litellm/anthropic/claude-3-7-sonnet	Depends on provider
	litellm/mistral-small	Depends on provider
HF Template	local/localhost:8000/v1//mistral-instruct-v0.2	Depends on provider
	litellm/ollama/mistral//hf

HuggingFace Chat Template Formatting

For models requiring specific prompt formatting:

import langroid.language_models as lm

# Specify formatter directly
llm_config = lm.OpenAIGPTConfig(
    chat_model="local/localhost:8000/v1//mistral-instruct-v0.2",
    formatter="mistral-instruct-v0.2"
)

# Using HF formatter auto-detection
llm_config = lm.OpenAIGPTConfig(
    chat_model="litellm/ollama/mistral//hf",
)