client_cache
langroid/language_models/client_cache.py
Client caching/singleton pattern for LLM clients to prevent connection pool exhaustion.
get_openai_client(api_key, base_url=None, organization=None, timeout=120.0, default_headers=None)
¶
Get or create a singleton OpenAI client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
OpenAI API key |
required |
base_url
|
Optional[str]
|
Optional base URL for API |
None
|
organization
|
Optional[str]
|
Optional organization ID |
None
|
timeout
|
Union[float, Timeout]
|
Request timeout |
120.0
|
default_headers
|
Optional[Dict[str, str]]
|
Optional default headers |
None
|
Returns:
Type | Description |
---|---|
OpenAI
|
OpenAI client instance |
Source code in langroid/language_models/client_cache.py
get_async_openai_client(api_key, base_url=None, organization=None, timeout=120.0, default_headers=None)
¶
Get or create a singleton AsyncOpenAI client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
OpenAI API key |
required |
base_url
|
Optional[str]
|
Optional base URL for API |
None
|
organization
|
Optional[str]
|
Optional organization ID |
None
|
timeout
|
Union[float, Timeout]
|
Request timeout |
120.0
|
default_headers
|
Optional[Dict[str, str]]
|
Optional default headers |
None
|
Returns:
Type | Description |
---|---|
AsyncOpenAI
|
AsyncOpenAI client instance |
Source code in langroid/language_models/client_cache.py
get_groq_client(api_key)
¶
Get or create a singleton Groq client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
Groq API key |
required |
Returns:
Type | Description |
---|---|
Groq
|
Groq client instance |
Source code in langroid/language_models/client_cache.py
get_async_groq_client(api_key)
¶
Get or create a singleton AsyncGroq client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
Groq API key |
required |
Returns:
Type | Description |
---|---|
AsyncGroq
|
AsyncGroq client instance |
Source code in langroid/language_models/client_cache.py
get_cerebras_client(api_key)
¶
Get or create a singleton Cerebras client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
Cerebras API key |
required |
Returns:
Type | Description |
---|---|
Cerebras
|
Cerebras client instance |
Source code in langroid/language_models/client_cache.py
get_async_cerebras_client(api_key)
¶
Get or create a singleton AsyncCerebras client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
Cerebras API key |
required |
Returns:
Type | Description |
---|---|
AsyncCerebras
|
AsyncCerebras client instance |