client_cache
langroid/language_models/client_cache.py
Client caching/singleton pattern for LLM clients to prevent connection pool exhaustion.
get_openai_client(api_key, base_url=None, organization=None, timeout=120.0, default_headers=None, http_client=None, http_client_config=None)
¶
Get or create a singleton OpenAI client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
OpenAI API key |
required |
base_url
|
Optional[str]
|
Optional base URL for API |
None
|
organization
|
Optional[str]
|
Optional organization ID |
None
|
timeout
|
Union[float, Timeout]
|
Request timeout |
120.0
|
default_headers
|
Optional[Dict[str, str]]
|
Optional default headers |
None
|
http_client
|
Optional[Any]
|
Optional httpx.Client instance |
None
|
http_client_config
|
Optional[Dict[str, Any]]
|
Optional config dict for creating httpx.Client |
None
|
Returns:
Type | Description |
---|---|
OpenAI
|
OpenAI client instance |
Source code in langroid/language_models/client_cache.py
get_async_openai_client(api_key, base_url=None, organization=None, timeout=120.0, default_headers=None, http_client=None, http_client_config=None)
¶
Get or create a singleton AsyncOpenAI client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
OpenAI API key |
required |
base_url
|
Optional[str]
|
Optional base URL for API |
None
|
organization
|
Optional[str]
|
Optional organization ID |
None
|
timeout
|
Union[float, Timeout]
|
Request timeout |
120.0
|
default_headers
|
Optional[Dict[str, str]]
|
Optional default headers |
None
|
http_client
|
Optional[Any]
|
Optional httpx.AsyncClient instance |
None
|
http_client_config
|
Optional[Dict[str, Any]]
|
Optional config dict for creating httpx.AsyncClient |
None
|
Returns:
Type | Description |
---|---|
AsyncOpenAI
|
AsyncOpenAI client instance |
Source code in langroid/language_models/client_cache.py
126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 |
|
get_groq_client(api_key)
¶
Get or create a singleton Groq client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
Groq API key |
required |
Returns:
Type | Description |
---|---|
Groq
|
Groq client instance |
Source code in langroid/language_models/client_cache.py
get_async_groq_client(api_key)
¶
Get or create a singleton AsyncGroq client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
Groq API key |
required |
Returns:
Type | Description |
---|---|
AsyncGroq
|
AsyncGroq client instance |
Source code in langroid/language_models/client_cache.py
get_cerebras_client(api_key)
¶
Get or create a singleton Cerebras client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
Cerebras API key |
required |
Returns:
Type | Description |
---|---|
Cerebras
|
Cerebras client instance |
Source code in langroid/language_models/client_cache.py
get_async_cerebras_client(api_key)
¶
Get or create a singleton AsyncCerebras client with the given configuration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
api_key
|
str
|
Cerebras API key |
required |
Returns:
Type | Description |
---|---|
AsyncCerebras
|
AsyncCerebras client instance |