Memory Leak when using rewrite_tag (keep=false) with storage.type=filesystem on input

[art333]

Bug Report

Describe the bug
Fluent Bit memory increases linearly and unbounded when the following conditions occur together:

1. The input uses storage.type=filesystem
2. A rewrite_tag filter uses that same field (e.g., $type)
3. The rewrite_tag rule has keep=false

If any one of these conditions is removed, memory stabilizes normally.
The leak reproduces with both real logs (OpenTelemetry) and dummy input.

This issue is present in at least:
Fluent Bit 4.0.10 and 4.2.0 (Docker official images).

To Reproduce
Minimal Log Example:

{"event":"payment","amount":19.99,"currency":"EUR","meta":{"method":"card","country":"DE"}}

Minimal Reproducer (this configuration leaks memory):

service:
  storage.path: /fluent-bit/storage

pipeline:
  inputs:
    - name: dummy
      tag: v1_logs
      dummy: '{"event":"payment","amount":19.99,"currency":"EUR","meta":{"method":"card","country":"DE"}}'
      rate: 55000
      storage.type: filesystem

  filters:
    - name: rewrite_tag
      match: 'v1_logs'
      rule: $event ^(.+)$ $1 false     

  outputs:
    - name: stdout
      match: "*"

Steps to reproduce the problem

1. Save the config above as config.yaml
2. Run Fluent Bit via Docker:

mkdir -p fb-storage
docker run --rm -it \
  -v $(pwd)/config.yaml:/fluent-bit/etc/config.yaml \
  -v $(pwd)/fb-storage:/fluent-bit/storage \
  fluent/fluent-bit:4.2.0 \
  -c /fluent-bit/etc/config.yaml

3. Monitor memory (e.g., docker stats or Kubernetes metrics)
4. Memory increases continuously without plateau

Expected behavior

• Fluent Bit memory usage should stabilize after initial warm-up.
• No long-term linear RSS growth under a steady input rate.
• rewrite_tag + filesystem storage should not retain unbounded memory.

My Environment
Version used: 4.0.10, 4.2.0
Configuration: minimal reproducer included above
Environment: Docker + Kubernetes
Input type: dummy or OpenTelemetry (reproduces on both)
Server OS: Linux
Filters involved:
rewrite_tag (with keep=false)
filesystem storage on input

Additional context

• The leak only happens in this strict combination:

filesystem input
→ rewrite_tag uses existing field with keep=false
→ RSS memory grows without bound

• This affects production workloads: after several hours/days Fluent Bit pods exceed memory limits and get OOMKilled.
• This memory leak occurs only when Fluent Bit runs on containerd 1.7.27 and does not occur on containerd 2.0.0

[art333]

I updated the description as it seems that modify filter is not required to reproduce that bug.

[art333]

Inside the container, using the proc input (Name proc, mem true, proc_name fluent-bit), the process RSS (mem.VmRSS) stays stable around ~30–35 MiB.

However, Kubernetes / containerd 1.7.27 report the pod’s cgroup memory usage growing up to the full memory limit (250Mi), and the pod is eventually OOMKilled (exitCode 137).

This suggests that the leak is not in the Fluent Bit heap itself, but in how containerd 1.7.x / the kernel account memory for filesystem-backed storage and page cache under the pod’s cgroup.

On containerd 2.0.0 and Docker Engine, the same configuration keeps both process RSS and cgroup usage stable.