parser
BaseParsingConfig
¶
Bases: BaseSettings
Base class for document parsing configurations.
GeminiConfig
¶
Bases: BaseSettings
Configuration for Gemini-based parsing.
MarkerConfig
¶
Bases: BaseSettings
Configuration for Markitdown-based parsing.
PdfParsingConfig
¶
Bases: BaseParsingConfig
enable_configs(values)
¶
Ensure correct config is set based on library selection.
Source code in langroid/parsing/parser.py
Parser(config)
¶
Source code in langroid/parsing/parser.py
add_window_ids(chunks)
¶
Chunks may belong to multiple docs, but for each doc, they appear consecutively. Add window_ids in metadata
Source code in langroid/parsing/parser.py
chunk_tokens(text)
¶
Split a text into chunks of ~CHUNK_SIZE tokens, based on punctuation and newline boundaries. Adapted from https://github.com/openai/chatgpt-retrieval-plugin/blob/main/services/chunks.py
Parameters:
Name | Type | Description | Default |
---|---|---|---|
text
|
str
|
The text to split into chunks. |
required |
Returns:
Type | Description |
---|---|
List[str]
|
A list of text chunks, each of which is a string of tokens |
List[str]
|
roughly self.config.chunk_size tokens long. |
Source code in langroid/parsing/parser.py
273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 |
|