Skip to content

mongodb output fails with error when using MongoDB Extended JSON ($date) #3782

@mladBlum

Description

@mladBlum

I want to insert a JSON document into MongoDB that contains a date field expressed in MongoDB Extended JSON format (see https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/#mongodb-extended-json--v2-). This is valid JSON — an object with a "$date" field containing an ISO8601 string.

Example input:

{
  "thing_id": "t1",
  "timestamp": {
    "$date": "2025-10-13T09:13:52Z"
  }
}

Expected behavior:

  • Connect should forward this JSON unchanged to MongoDB. MongoDB should interpret { "$date": "2025-10-13T09:13:52Z" } as a Date and store ISODate("2025-10-13T09:13:52Z").

Actual behavior:

Instead of inserting the document, Connect crashes with the following error mesage:

rpk connect run config.yaml
INFO[2025-11-17T17:38:01+01:00] Running main config from specified file       @service=redpanda-connect benthos_version=4.70.0 path=config.yaml
INFO[2025-11-17T17:38:01+01:00] Successfully loaded Redpanda license          @service=redpanda-connect expires_at="2035-11-15T17:38:01+01:00" license_org="" license_type="open source"
INFO[2025-11-17T17:38:01+01:00] Listening for HTTP requests at: http://0.0.0.0:4195  @service=redpanda-connect
INFO[2025-11-17T17:38:01+01:00] Input type file is now active                 @service=redpanda-connect label="" path=root.input
INFO[2025-11-17T17:38:01+01:00] Launching a Redpanda Connect instance, use CTRL+C to close  @service=redpanda-connect
INFO[2025-11-17T17:38:01+01:00] Output type mongodb is now active             @service=redpanda-connect label="" path=root.output
ERRO[2025-11-17T17:38:01+01:00] Failed to send message to mongodb: failed to execute document_map: invalid JSON input; expected {  @service=redpanda-connect label="" path=root.output
INFO[2025-11-17T17:38:01+01:00] Pipeline has terminated. Shutting down the service  @service=redpanda-connect

How to reproduce:

  1. Start MongoDB:
docker run -d -p 27017:27017 --name=mongo-example1 mongo:latest
  1. Create a config.yaml file:
input:
  file:
    paths: [ ./*.json ]
    scanner:
      json_documents: {}

output:
  mongodb:
    url: mongodb://localhost
    database: admin
    collection: message
    operation: insert-one
    write_concern:
      w: majority
      j: false
      w_timeout: ""
    document_map:
      root = this
  1. Create a input.json file:
{
  "thing_id": "t1",
  "timestamp": {
    "$date": "2025-10-13T09:13:52Z"
  }
}
  1. Run
rpk connect run config.yaml

Strangely enough, the following JSON file works fine without an error. So expect that something happen if Connect tries to process $date.

{
  "$device_id": "fs1",
  "timestamp": {
    "$someThing": "2025-10-13T09:13:52Z"
  }
}

I hope this fits as an issue; I don't think it's a discussion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions