A Linux device on a desk gives you a false sense of comfort. You can open a terminal, run journalctl, inspect a boot, add a few messages and try again. The same device inside an electrical cabinet, camera, kiosk or measuring instrument becomes much quieter. Memory is fixed. Storage may be slow or absent. The network can disappear exactly when the bug appears.
Memfault’s starting point is that difference, but the useful question is harder than “where should the logs go?” What should remain of an incident when nobody can touch the machine for weeks?1 Embedded logging is not a stream of text. It is a policy about memory, evidence and acceptable loss.
A log begins with a hole
On a server, you can add a disk or ship events elsewhere. On a device, every byte reserved for logs competes with the system, user data, updates and sometimes flash lifetime. Memfault describes a rotating buffer as a structural constraint: once the buffer fills, older events are overwritten.1
That mechanism is harsh but legible. It forces a window: ten minutes of detail, two hours of summaries, or only the previous boot. The right answer depends on the delay between failure and collection. A router rebooting every thirty seconds needs a different history from an industrial device that fails once a month.
The first calculation is therefore not “how many lines per second?” It is “when will we know something went wrong?” If the team discovers the problem through a customer call, the device must retain enough context before that call. If a heartbeat can trigger collection immediately, the buffer can be shorter. Observability is diagnostic architecture before it is an API.
RAM is often the best place for recent detail: it avoids constant writes and works on devices without non-volatile storage. But a crash or power cut erases it. Flash preserves history at the price of wear, latency and harder power-loss handling. A network backend supplies memory outside the device, but fails in connectivity incidents.7
There is no “complete log.” There is a composition of losses: losing the oldest events, dropping low-priority messages, losing the tail during a power cut, or losing the entire incident because the system tried to transmit everything continuously.
Three layers people often confuse
An embedded Linux system contains several journals. The kernel writes to a ring buffer read through /dev/kmsg or dmesg; Linux documents printk as the standard kernel message mechanism.4 User-space services may write to stderr, syslog or journald. A telemetry agent can then read one of those streams and ship it to a backend.
These layers are not interchangeable. A kernel ring-buffer message is not persistent evidence. A service writing to journald is not automatically visible after reboot if storage remains volatile. A collector reading the journal must remember its position so it does not process the same events repeatedly; Memfault’s Linux documentation describes cursor-based resumption for memfaultd.2
The confusion becomes expensive during failure. A team believes it has “enabled logs” because journalctl shows lines on a development bench. In production, Storage=auto may remain in memory if the persistent directory does not exist. Systemd explicitly distinguishes volatile, persistent, auto and none, as well as the conditions that move data into persistent storage.5
The useful test is not starting a service and seeing one line. Cut power, reboot, nearly fill the disk, remove the network, generate a burst of errors and verify what can still be recovered. The evidence path needs testing like the nominal path.
Journald is not a magic disk
Journald provides useful structure on systemd systems: indexed fields, selection by unit, boot, priority or identifier, export formats and in-line compression for larger objects.5 Its file format is binary, primarily append-based, indexed by fields and designed to remain queryable; systemd also documents in-line compression and optional forward-secure sealing.6
Those properties help when the problem is finding an event in a complex system. They do not remove hardware constraints. You still need to decide where the journal lives, how large it can become, which messages are retained and how it is transferred. Structure adds a choice: a few well-named fields can make a search possible, while a long text message can consume space without explaining more.
Rate limiting is where system clarity meets its violence. Journald applies a default per-service limit and can drop messages beyond a configured burst while reporting that messages were lost.5 On a server, losing part of a burst may be sensible. On a device where the burst is itself the symptom — a link loop or sensor returning an impossible value — dropping its beginning or end can make the incident unreadable.
The question is not “should rate limiting be disabled?” It is “which event should survive it?” A repetition counter, a state transition, the first occurrence and the last occurrence are often more useful than a thousand identical lines. An application can also slow its own emission, aggregate repeats or change detail when it detects a loop.
Write for a machine and a person
Human-facing logs tend to become sentences. Machine-facing logs become structured objects. The opposition is lazy. You often need both: a stable event identifier, priority, reliable timestamp, subsystem, error code and contextual values, plus a short phrase that lets an engineer understand the mechanism.
Fields are a public interface. Renaming wifi_error to network_problem between versions makes comparisons harder even if the software works. Removing a code because it seemed redundant breaks analysis scripts. The vocabulary must be versioned, documented and kept interpretable months later.
The format depends on the layer. JSON is readable, but repeated keys and strings can be expensive on a small system. A compact binary format saves space but needs a decoder that survives versions. Specialized logging systems can keep identifiers in firmware and enrich the text server-side. That is powerful, but it makes the symbol dictionary and build versions part of the long-term evidence chain.
The source of a message is context too. A “timeout” says little without the operation, peripheral, firmware, duration and power state. But logging SSIDs, identifiers, user content or location turns a repair tool into a personal-data export. Observability does not suspend privacy.
The network is an assumption, not an exit
Memfault describes the two classic collection modes: local retrieval with a cable or local access, and remote retrieval that requires transport, storage and privacy rules.1 A real product should plan for both. Local mode rescues isolated devices; remote mode avoids sending a technician to every customer.
Remote collection needs a budget. Shipping every line continuously increases power use and cost. Waiting for a crash may lose the moments immediately before it. A practical compromise is a local buffer, a trigger and a bounded window around the event. The trigger can be a crash, watchdog, missing heartbeat or metric change.
Metrics and logs complement each other. Memfault argues that logs should not be the only way to monitor connectivity: metrics show fleet-level trends, while logs explain an individual case.3 A hundred devices each losing Wi-Fi once produce little text worth reading; a version-by-version rate curve can expose a regression before anyone opens a particular log.
That changes software design. A rare event deserves rich context. A frequent state deserves a metric. A repeating sequence deserves a counter and perhaps a sample. “Log the error” hides three different decisions.
A crash is not the end of the story
A crashing device often produces less information exactly when evidence matters most. The process may die before flushing its buffer. The kernel may reboot. A power cut may leave a file half-written. Logging therefore belongs in the same design conversation as coredumps, boot traces, reset reasons and watchdog counters, even when different tools implement them.
The kernel ring buffer is a useful example of temporary memory: messages remain in a circular structure and are exposed to user space.4 Its size is configurable; log_buf_len sets the kernel log-buffer size.8 Enlarging it does not turn it into an archive. It only delays overwriting and consumes RAM.
A robust design uses several resolutions: the last reset reason, detailed messages from the last few seconds, aggregate counters from recent hours, firmware version and update state. No layer has to remember everything. They need to overlap enough to make the incident reconstructible.
A small design method
Before choosing a library, write five scenarios: network failure, application crash, storage corruption, log loop and total inaccessibility. For each, state what you need to know, the last moment collection is possible, the memory available and the data acceptable to transmit.
Then define events that change a decision. “Temperature is 42 °C” every second may not help; “the sensor crossed the threshold for 18 seconds and the system rebooted” probably does. Capture transitions, reasons and versions, not every heartbeat.
Finally, test loss. Fill the flash. Cut power during a write. Generate ten thousand errors. Remove the network. Boot old firmware with a new decoder. Check that an operator without physical access can distinguish a peripheral failure from a transport failure. A logging system is not reliable because it writes a lot; it is reliable because its losses are planned.
The collector is already a product decision
Between the service producing an event and the backend receiving it there is often an agent: journald, syslogd, rsyslog, Fluent Bit or a custom component. These are described as pipes. They are better understood as regulators. They decide when to read, wait, write, compress, discard and which failures they themselves must survive.
Fluent Bit documents this reality through memory, chunks, files and backpressure.9 In memory mode, a limit can pause an input when the consumer is too slow. That protects the process from a RAM explosion, but a file rotation during the pause can lose lines. Filesystem buffering adds restartability and storage capacity; it does not make disk infinite. You still need a path, a per-destination limit and a policy for when the limit is reached.
The most useful detail is one logging tools usually hide: “reliable” has several meanings. A chunk written to disk but not synchronised has a different guarantee from a chunk durable after power loss. A queue that resumes after an orderly shutdown says nothing about a kernel crash. A policy retaining the newest events may be perfect for debugging and wrong for an audit trail.
Rsyslog makes those distinctions explicit. Its queues can be in memory, on disk or disk-assisted. In the disk-assisted mode, memory remains the fast path and disk is used when the queue crosses a threshold; the in-memory portion may be saved during a clean shutdown but not after a kill, OOM, kernel crash or power loss.10 A pure disk queue is more durable but slower and more write-heavy. The choice describes which loss the product accepts.
On a constrained device, a hierarchy is often better. Keep critical events in RAM and flush them to flash only on a state change. Store a persistent summary rather than every line. Reserve a small recovery area for boot. Upload a window around a crash. Each level has its own promise, and the whole arrangement must remain understandable when an engineer returns two years later.
Flash is not paper
Writing a log feels like adding information without changing anything else. On embedded NAND or flash, every write participates in wear, address translation, block erasure and garbage collection. The details depend on the component, controller, filesystem and workload; there is no honest universal cycle count.
It is reasonable, however, to treat write frequency as a design constraint. Measurement work such as Flashmon exists to observe NAND I/O requests in embedded Linux because the logical volume written by a service does not fully describe the work performed by the storage system.11 A small line repeated every second can lead to much larger writes and erases depending on how the filesystem and controller group data.
This does not mean “never write.” A correctly used flash can retain useful logs for a product’s life. It means measure instead of assuming. How much data is produced per hour? What happens during an incident when the rate increases tenfold? Which health counters exist? Which part of the log deserves persistence, and which part can disappear at reboot?
Compression adds another tradeoff. It reduces bytes written and transmitted but consumes CPU and can delay availability. Journald compresses larger objects inline according to configuration.5 Fluent Bit can keep chunks in memory and files, with sync modes trading performance against durability.9 The choice has to account for energy and temperature as well as disk space.
A battery device may prefer a hundred structured events to heavy compression. A plugged-in device that is difficult to reach may prefer batching to reduce writes. An eMMC-equipped gateway has different headroom from a development board with an SSD. “Embedded” is not the answer; the storage and failure profile is.
Rate limits: losing intelligently
A log loop is a secondary failure. A driver repeating an error can consume CPU, fill the journal, wake a modem, accelerate wear and prevent the important message from being stored. Rate limiting is therefore product protection, not only noise reduction.
Journald applies a per-service burst limit and drops messages beyond the threshold during the interval while generating a message about the loss.5 Rsyslog can slow a queue, rate-limit inputs and choose between throttling a reliable producer or dropping messages when space is unavailable.10 Fluent Bit can pause inputs when memory exceeds its limit, with consequences for sources that continue writing.9
These behaviours create a hierarchy question. If a camera emits the same connection error a thousand times, keep the first error, count, last error and network state. If a watchdog restarts the machine, the reset reason matters more than the last debug messages. If a customer reports an intermittent fault, a detailed window around a state change is worth more than a full day of archive.
The application emitting events can help. It can aggregate repeats, add duration, mark transitions, enter a temporary diagnostic mode or sample. That keeps the collector from guessing the meaning of ten thousand identical lines. It also makes the contract clearer: the producer knows what matters, the transport knows what it may lose.
Crash context: the last moment costs the most
Device diagnosis starts before the support ticket. A useful architecture keeps the last reset reason, software version, boot age, power state, watchdog counters and a few seconds of detailed events.
The kernel ring buffer is good for this kind of volatile memory. printk writes to the circular buffer exported to user space; dmesg reads what the next reboot has not erased.4 log_buf_len can enlarge that area.8, but it consumes RAM and guarantees no persistence. The goal is not to retain the whole boot; it is to retain transitions that explain why the next boot happened.
Logs are not enough by themselves. A coredump gives a process snapshot, a reset counter gives a rough cause, a heartbeat metric shows absence and a version event supplies deployment context. Memfault’s logging article focuses on collection and triggering; its Linux documentation describes how memfaultd resumes reading a journal from a cursor after restart.12
Incident collection should avoid two extremes. If it waits for the network, it may miss context lost during reboot. If it sends everything before the cause is known, it wastes power and bandwidth. A local buffer, trigger, enriched sample and deferred upload are often a more honest compromise.
The field changes the policy
A lab device can expose logs over USB. A customer device needs a local export that does not require revealing a network secret, and a remote channel governed by consent, authentication and retention. A medical device or product installed in a private space adds obligations that cannot be solved by masking one identifier in a string.
Logs may contain network names, addresses, account identifiers, file paths, content fragments or usage times. Even a harmless-looking field can reveal a routine when repeated and correlated. “Log it for debugging” is also a data decision.
Decide which data is necessary at each level. The device can keep a technical identifier locally and send a pseudonymous identifier to the backend. It can keep detailed traces only after a user requests diagnostics. It can omit content and retain state codes. It can offer an “export diagnosis” action that shows what actually leaves the device.
Privacy and maintenance are not opposites. Unnecessary data is poor diagnostic data: it increases risk without improving a decision. An architecture that explains events, limits fields and separates detail levels is easier to protect and easier to analyse.
After failure: turning traces into repair
A log matters only if it reduces a decision. The maintainer needs to decide whether to replace a sensor, reinstall firmware, change network configuration, ask the user to act or recall the product. A sequence of messages without a stable vocabulary only moves noise into a dashboard.
Structured fields help, but they must evolve. Define a schema version, build identifier, a clock whose drift is known and values whose meaning will not change silently. Events should distinguish absent, invalid, estimated and unrequested measurements. Zero is not always zero.
Correlation is also a maintenance choice. A boot identifier links events from one startup; an incident identifier links error, reset and recovery; a device identifier allows version comparison without necessarily exposing a person. These links make support faster and data more sensitive. The right practice is not adding every identifier; it is knowing which link answers which question.
Documentation must survive the team that wrote the code. recover_failed can mean ten things. Describe the condition, attempt count, expected result and possible action. Logs are a repair interface; their audience often includes someone who has never seen the source.
The test plan should provoke loss
Testing logging too often means checking that a line appears. The real test provokes the conditions under which the line would disappear. Fill the disk. Cut power during a write. Shut down the collector. Generate a burst. Run for days on an unstable network. Replay an old firmware with a new parser. Reboot while the agent is flushing its buffer.
Each test needs an answer: what is the oldest event retained? What loss is reported? Does the main service continue when the queue is full? Is a reliable source throttled while an unreliable UDP source is dropped? Is the journal recoverable after power loss? Does the next boot retain the reset reason?
Rsyslog and Fluent Bit documentation show why terms such as “disk-assisted,” “filesystem buffer” and “persistent” must be read alongside their shutdown scenario.910 A configuration safe on a server with stable power may be wrong for a camera that reboots abruptly with a small flash.
Measure cost too: writes per hour, CPU spent formatting and compressing, memory held by chunks, producer wait time, modem energy and storage after a simulated week-long incident. These are product metrics, not system-team trivia.
A design method for makers
Before selecting journald or an agent, write five scenarios: network failure, application crash, storage corruption, log loop and total inaccessibility. For each, state the maintenance question, last possible collection moment, available memory, acceptable privacy level and permitted loss.
Then separate events by function. Critical transitions and reset reasons should be compact and persistent. Debug detail can remain in a RAM buffer. Frequent states should often become metrics or counters. Ticket data should be exportable without granting access to the whole journal.
Define a pressure policy: slow, aggregate, sample, pause, overwrite the oldest or drop the newest. There is no universal answer. There is a decision you can explain. A product that overwrites its oldest events must own that choice; a product that blocks when disk is full must verify that the block cannot create a worse failure.
Finally, write recovery as part of the interface. Where is the diagnostic? Who can read it? How does support distinguish a network problem from a collection problem? How can a fix that adds missing fields be installed? What happens if the team disappears? A log interpretable only by its author is not a maintenance strategy; it is a personal note.
Logs are not factory waste
Memfault’s useful contribution is returning observability to material reality, but the conclusion goes beyond any one vendor. An embedded Linux device does not have an abstract reservoir of logs. It has RAM, flash, a network, a battery, a user, a support budget and an expected life.
Journald demonstrates the value of fields and storage policy. printk demonstrates the usefulness and fragility of a kernel ring buffer. Rsyslog shows that queuing and durability are different decisions. Fluent Bit shows that backpressure can protect memory at the cost of input loss, or use filesystem storage at the cost of writes and limits. The tools do not make the policy for the manufacturer.
The decisive question is the one asked after the device has left the lab: what will you still be able to prove? The answer should not be “everything.” It should specify which events, in what form, for how long, with what privacy and at what material cost.
A good logging system does not collect the world. It preserves enough context for another person to choose a repair. It is a product component, like a connector, recovery mode or OTA firmware. The bytes it writes shape storage life; the bytes it does not write shape diagnostic quality. Designing that boundary before failure is already designing the object after delivery.