parsing
PdfParsingConfig
¶
Bases: BaseParsingConfig
enable_configs(values)
¶
Ensure correct config is set based on library selection.
Source code in langroid/parsing/parser.py
Parser(config)
¶
Source code in langroid/parsing/parser.py
add_window_ids(chunks)
¶
Chunks may belong to multiple docs, but for each doc, they appear consecutively. Add window_ids in metadata
Source code in langroid/parsing/parser.py
chunk_tokens(text)
¶
Split a text into chunks of ~CHUNK_SIZE tokens, based on punctuation and newline boundaries. Adapted from https://github.com/openai/chatgpt-retrieval-plugin/blob/main/services/chunks.py
Parameters:
Name | Type | Description | Default |
---|---|---|---|
text
|
str
|
The text to split into chunks. |
required |
Returns:
Type | Description |
---|---|
List[str]
|
A list of text chunks, each of which is a string of tokens |
List[str]
|
roughly self.config.chunk_size tokens long. |
Source code in langroid/parsing/parser.py
299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 |
|