My agent system dreams at night, and that is where its memory comes from

· 7 min read

My agents recall durable facts across sessions. None of those facts got there because an agent, mid-task, decided "this is worth remembering forever." Almost everything an agent thinks is memorable in the moment is noise a week later, and an agent given a direct write path to durable memory will fill it with confident junk.

Instead, my system does what brains do: it consolidates offline. Twice a day, on a timer, a process wakes up, reads what happened since it last ran, and decides - slowly, with gates - what deserves to survive. I call it dreaming, and the name has turned out to be more than a joke: the design questions are genuinely sleep-shaped. What gets replayed? What gets promoted to long-term storage? What gets discarded? And what happens when the process misfires?

The shape: a timer, not a daemon

Dreaming is a one-shot process fired by a scheduler - a morning cycle for lighter work (recommendations, "what needs attention"), an evening cycle for the heavy memory-promotion pass. No resident daemon, no queue service. Each run is a fresh process that reads files, thinks, writes files, and exits. Everything in my stack is filesystem-first, and consolidation is no exception: if the machine is off, the cycle is skipped and the next one picks up the unprocessed range.

Two operational rules matter more than the schedule:

Exactly one machine dreams. My state syncs across machines, and two consolidators writing the same durable files would conflict endlessly. One machine owns dreaming; the others read the results. Single-writer is the cheapest concurrency model that exists, and choosing it here removed a whole category of merge problems before they happened.

Every cycle has a hard token budget. The reflection step calls an LLM, and an unbounded loop over a busy day's events is an unbounded bill. A cycle gets a fixed input and output cap; hitting the cap mid-cycle means finish gating what you already produced, log the overrun, and let the next cycle continue from there. A runaway day costs a known maximum. The daily spend lands around one to three dollars, which I consider cheap for a memory that maintains itself.

The pipeline: cluster, reflect, score, gate

The evening pass runs the day's episodic events - tool calls, session summaries, captured thoughts - through a fixed sequence:

  1. Cluster. Events are embedded and grouped by density. A durable fact almost never comes from one event; it comes from the same theme surfacing across sessions.
  2. Reflect. An LLM reads each cluster and proposes candidates: "these events support the fact that X." Each candidate carries the ids of the events supporting it, so provenance survives the whole trip.
  3. Score. Candidates get a confidence score and pass structural checks: enough distinct supporting events, spread over enough time, not a restatement of something already known.
  4. Judge. Two LLM-backed checks run against existing memory: does this contradict a stored fact, and is it a duplicate?
  5. Promote or queue. High-confidence, well-corroborated candidates can be written to durable memory automatically. Everything else lands as a card in a review queue where I accept or reject with a keystroke, and applying accepted items produces a git commit - the memory file's history is an audit log.

The single most important design fact: the interesting engineering is entirely in steps 3 through 5. Generating candidates is easy; any LLM over any event log will happily propose memories. The system's quality is decided by what it refuses to write. Durable memory pollution compounds - a bad fact gets recalled, believed, cited, and built upon by dozens of future sessions - so the write gate is where the paranoia belongs.

Concretely: auto-promotion requires both a confidence threshold and at least three supporting events. A candidate with two supporters can be judged genuinely durable by the reflector, and it still cannot enter memory unattended - it routes to the review queue instead. The rule is not "two events are not evidence"; it is "two events are not enough evidence to skip the human."

Two failure stories worth their tuition

The silent floor mismatch. For a while, the clustering stage was allowed to form two-event clusters, but the scoring stage silently dropped any candidate with fewer than three supporting events. Every two-event cluster the reflector judged durable was structurally discarded - not rejected with a reason, just gone. The pipeline looked healthy: cycles ran green, promotions happened, nothing errored. It was simply quieter than it should have been, and quiet is the hardest defect to notice. The fix lowered the score floor to match the cluster floor and moved the three-event rule to the auto-promote boundary, where it belongs: the candidate now survives to the review queue and the human sees it. The general lesson: when two stages of a pipeline disagree about a threshold, the disagreement does not error - it silently shrinks your output, and you will attribute the quietness to "slow week" for months.

The judges that aborted the dream. The contradiction and duplicate judges originally propagated a hard LLM error - an outage, a quota blip - straight up, aborting the entire cycle and discarding all the clustering and reflection work before them. Meanwhile a garbled LLM response was handled gracefully with a conservative default. That asymmetry made no sense: the transient network error was more destructive than the corrupted answer. The policy now is uniform: any LLM-backed stage that fails hard degrades to its conservative default (no contradiction found, assume novel) and the cycle completes. For a nightly batch process, resilience beats strictness - a conservative default risks one duplicate card in a review queue; an abort discards a day.

Consolidation grows skills, not just facts

The part I did not plan and now value most: the same clustering that finds durable facts also finds recurring work. When the cycle notices the same kind of procedure performed across at least three instances on multiple days with no skill covering it, it proposes one - and above a confidence threshold it auto-writes a draft skill file. The draft is inert: invisible to every agent harness until I explicitly promote it. The human gate did not disappear; it moved from "write the draft" to "activate the draft," which is a cheaper place for me to pay attention.

This is the sleep metaphor completing itself. Consolidation is not just deciding what to remember; it is noticing what you keep doing and turning it into ability. The facts feed recall, the skills feed behavior, and both come out of the same nightly replay of the day's events.

What transfers