Bug Report: column_indexing_batch_size Not Taking Effect with OpenAI-Compatible Embedding Model #2031
Description
Bug Report: column_indexing_batch_size Setting in config.yaml Not Taking Effect for OpenAI-Compatible Embedding Model
Environment:
- WrenAI version: [canner/wren-ai-service:0.27.14]
- Embedding model: OpenAI-compatible model with a maximum batch size limit of 10.
- Configuration: Set
column_indexing_batch_size: 10inconfig.yaml.
Description:
When using an OpenAI-compatible Embedding model that enforces a maximum batch size of 10, the column_indexing_batch_size parameter set to 10 in config.yaml does not appear to take effect. During operation, the container logs the following error, indicating that the batch size being sent exceeds the model's limit of 10, despite the explicit configuration:
litellm.llms.openai.common_utils.OpenAIError: Error code: 400 - {'error': {'message': '<400> InternalError.Algo.InvalidParameter: Value error, batch size is invalid, it should not be larger than 10.: input.contents', 'type': 'InvalidParameter', 'param': None, 'code': 'InvalidParameter'}, 'id': '2354f194-0475-45e1-ada8-95757467f96a', 'request_id': '2354f194-0475-45e1-ada8-95757467f96a'}
This suggests that the batch size configuration is either being ignored or overridden internally, leading to invalid requests to the embedding model.
Steps to Reproduce:
- Configure
column_indexing_batch_size: 10inconfig.yaml. - Use an OpenAI-compatible Embedding model with a max batch size of 10.
- Run the indexing process or relevant operation.
- Observe the error in container logs.
Expected Behavior:
The batch size should respect the configured value of 10 and not exceed the model's limit, preventing the 400 error.
Actual Behavior:
The error occurs as if the batch size is larger than 10, ignoring the config setting.
Additional Notes:
- This issue persists even with the exact max batch size configured.
- Please investigate if there's an internal default or override affecting this parameter.