A worker was OOMKilled — is it a leak or a limit
The runbook the PodOOMKilled alert links to. Tell a leak from a limit that is merely too low, then find which job is holding the memory.
The two faults look identical
The kernel killed the container because it hit its memory limit. That single fact is produced by two different problems, and the fix for one makes the other worse:
- The limit is too low. The workload genuinely needs more than it was given. Raising the limit is the fix.
- The workload leaks. Memory climbs until it hits whatever ceiling exists. Raising the limit buys time proportional to the increase and changes nothing else.
Raising the limit "works" in both cases, which is why the distinction is worth two minutes before you make the change.
Some images size themselves from the limit — OpenSearch's JVM heap did on this platform — so lowering a limit is not a safe direction either. Check what the process does with the value before moving it in either direction.
1. Which shape is it
The component's memory series is on its project page and on the component's own charts.
Open the project →Look at the memory series between restarts, not across them:
- A sawtooth that climbs from the same floor each time is a limit that is too low, or a workload whose peak has grown. The floor is what the process holds at rest.
- A sawtooth whose floor rises is a leak. Each restart starts higher than the last because the thing being retained is being retained faster than it is being released.
- A flat line with one spike is a single expensive operation, not a trend. Find the request or job at that timestamp rather than changing the limit.
The alert's [logs] link is scoped to the component and to the alert's own window, so it
lands on the lines around the kill rather than on the estate.
2. Which job, which request
A worker that dies has usually died doing something specific.
The job class is on the span, and it always was. The span is named default process — the ActiveJob instrumentation defaults span_naming to :queue, so every job
on the default queue shares one name — but it carries code.namespace = EmbedNoteJob as
an attribute, next to messaging.active_job.adapter.name and the job id. Select the span
and read them; you do not need a new build to answer "which job".
What one shared name actually breaks is Pyroscope, where span_name is a series
label: every job class collapses into one series there, and no amount of reading
attributes fixes that. It is what span_naming: :job_class is for, and both Rails apps
set it now.
Do not expect the job span to hold the CPU. Samples are attributed to the INNERMOST
open span, so for a job that spends its time in the database and in HTTP calls, they land
on the SELECT/COMMIT/POST spans beneath it and not on the job. Measured on
inkwell-worker: thirteen span_name values in Pyroscope, every one a database or
SolidQueue span, and the job span not among them — while that job span is plainly there
in the trace, 118ms long, with its class on it. That is the attribution rule working, not
a missing profile.
Filter by the component and the minutes before the kill. A job that ran long, or ran hundreds of times, is the candidate.
Find the transaction →From a transaction, open the trace and read the span's attributes and events — the argument sizes, the row counts, the retry events are all there and are usually enough to name the culprit without a profiler.
3. Turn on heap profiling and redeploy
If the trace is not enough, profile the allocations.
Python components carry a heap profiler. Set memoryProfiling: true on the
component's continuous-profiling trait for that environment and redeploy.
A trait's per-environment values live on the ReleaseBinding, keyed by the
trait's instanceName — not on the Component. ComponentTrait has four fields
(kind, name, instanceName, parameters) and no environmentConfigs, so
the shape this guide used to show applied cleanly and did nothing at all:
# openchoreo/bindings-production.yaml
apiVersion: openchoreo.dev/v1alpha1
kind: ReleaseBinding
metadata: {name: inkwell-enrich-production, namespace: default}
spec:
owner: {projectName: inkwell, componentName: inkwell-enrich}
environment: production
traitEnvironmentConfigs:
profiling: # the trait's instanceName on the Component
memoryProfiling: true
A push applies the per-environment configs out of that file and ignores any
release pin in it, so this is a reviewed change rather than a kubectl patch
the next push would revert.
It is off by default because a heap profiler hooks the allocator, which is a different cost from sampling the CPU. Turn it on for a reason, and turn it off again.
Go components already push the full memory set — alloc_objects, alloc_space,
inuse_objects, inuse_space — with no change needed.
Ruby components do not. The pyroscope gem is CPU-only, and no setting changes that.
For a Rails leak the answers are the CI memory profilers (a fully-profiled build reports
retained bytes and per-test RSS) or a heap dump taken from the running process. This is a
real gap and it is stated here rather than left to be discovered.
4. What to change
-
Leak: fix the retention. On this platform the usual shape is a constant or a class variable that accumulates — a memoisation with no bound.
-
Limit too low: raise the size class, in the same file, and the record is the diff.
sizeClassis per environment, so production can be bigger than staging without a second release;smallis the platform standard for that kind of workload and each step doubles it:spec: componentTypeEnvironmentConfigs: sizeClass: mediumAn explicit
resourcesblock still overrides the class when a workload needs a number rather than a rung. Note that the cell's ResourceQuota caps the sum of container limits, and a rolling update needs the new pod before it can drop the old one — so a promotion that does not fit is refused with the arithmetic, rather than wedging the namespace. -
One expensive operation: bound the operation, not the memory. Paginate the query, stream the file, chunk the batch.
Either way, the alert stays until the pod stops being killed, so the change is its own verification.