Something must have happened along the way, because this was not the original design intent of the database (emphasis mine):
"""
The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that.
journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.
It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).
If you're interested, here are my findings this time so far:
- workrave: 60 second stat sync https://github.com/rcaelers/workrave/pull/717
- kde klipper: saves to disk on every copy, even if permanent storage is disabled https://bugs.kde.org/show_bug.cgi?id=501030
- kde plasmashell: saves qt shader cache each time notification popup disappears https://bugs.kde.org/show_bug.cgi?id=523805
- bitwarden firefox extension: tries to connect to desktop application every 10 seconds, writes about every failure to browser's WebStorage 14+ KB https://github.com/bitwarden/clients/issues/22192
- firefox datareporting/glean: very chatty .mozilla/firefox/xxx/datareporting/glean/db/data.safe
- ipfs: writes every received DHT announce to disk, 20 GB in 3 hours https://discuss.ipfs.tech/t/constant-writes-to-datastore-log/20316
- mailcow: redis saves data every 5 minutes https://github.com/mailcow/mailcow-dockerized/pull/7405
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.
I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.
>This issue report feels like it ought to be accompanied by a fix.
systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such.
These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals to handle it. Sometimes "suggestion" PR is worse than a triaged issue IMO.
Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component
Designing a new on-disk format seems like a pretty far reaching architectural decision... I don't think that's an appropriate target for a drive-by fix from a new contributor
It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on writes.
I'm probably the main person responsible for making journald usable at all.
But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.
Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.
My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.
Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic.
There was mailing list discussion at the time journald was conceived though, you can find it if you look.
> Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance
> Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
Thank you for your work. ISTM the workload is naturally LSM-shaped.
> If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple.
> I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory).
If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.
Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.
Oh how I love totally predictable poetterings reply to previous bug report that got closed because "measuring it wrong" and "this is not a support forum".
Systemd should just use DuckDB. It's perfect for this job.
"But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"
Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.
Parquet is probably an even better option. Columnar, compression, fast, succinct. All good things.
You can read them with DuckDB, but you don't end up with O(log n) writes -- which is, to speak plain English, batshit fucking insane for a system logger.
What those cursed writes buys you is O(log n) reads, but there's just no scenario that is necessary. If you have literally any time or subsystem constraints, parquet's predicate pushdowns means you get plenty fast access even with a full scan.
Something must have happened along the way, because this was not the original design intent of the database (emphasis mine):
"""
The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that.
"""
See https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG...
journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.
I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom
Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.
It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).
If you're interested, here are my findings this time so far:
Unfortunately it's not always easy to move away from systemd. I still run void on one of my machines, but aur make things so much easier.
I noticed plasmashell is write heavy and logs to journald a lot so I have just switched to XFCE partly for that reason.
The two I most often see in Ubuntu's dmesg are:
audit - appears to be some sort of AppArmor logging?
br[] - bridge interface docker uses consistently rebuilds itself? May be related to docker compose networking.
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.
I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
One of the first things I do on win10 is disable most of excess logging.
But why, though? I have never seen a performance hit from it that would warrant that.
https://github.com/systemd/systemd/issues/40262#issuecomment...
Okay, so how do I disable journald and switch to something else, without getting rid of systemd completely?
https://www.freedesktop.org/software/systemd/man/latest/syst...
See StandardOutput= and StandardError=.
Ooh, mmapped writes. I make that mistake once, years ago. :) I posted a comment in that GH issue.
Cool to see @ValdikSS here as well. The guy never sleeps or he is AI in disguise ;-)
This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.
>This issue report feels like it ought to be accompanied by a fix.
systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such.
These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals to handle it. Sometimes "suggestion" PR is worse than a triaged issue IMO.
Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component
Sure, a concrete proposal first would be a good idea.
That said, would you look a gift horse in the mouth?
Designing a new on-disk format seems like a pretty far reaching architectural decision... I don't think that's an appropriate target for a drive-by fix from a new contributor
Well, they are comparing in comments with syslog, and it does better, as you asked. Syslog was there for 46 years.
syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour...
See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG...
It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on writes.
For 99% of installs the basic assumption that local logging (with local reading) is the primary mode is just wrong.
I'm probably the main person responsible for making journald usable at all.
But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.
Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.
My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.
Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.
journald uses hash tables, I think it update it on every new log line, although I didn't debug it in depth yet.
https://github.com/systemd/systemd/blob/199f75205b9c0625bf56...
Why not just have a SQLite file and call it a day?
Also, why mmaped file?
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic.
There was mailing list discussion at the time journald was conceived though, you can find it if you look.
https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/
It doesn't look like Lennart Poettering did a design review; he just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
Well there was an ambition, apparently.
> Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance
> Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
SQLite here okay, but DuckDB or LevelDB better, IMHO. Either way, no need to invent a new storage format.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
Neither DuckDB nor LevelDB existed when journald was created. Not to say it couldn't be done today, but just some historical context.
Thank you for your work. ISTM the workload is naturally LSM-shaped.
> If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple.
> I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory).
If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.
Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.
[dead]
hello ValdikSS! nice to see you alive
journald has never been of great quality. It somehow manages to be visibly slower than grepping gzipped text logs.
it has bad scaling properties esp. if you have many journal files
the last time I contributed to journald upstream was to fix a degenerate behavior with many journal files: https://github.com/systemd/systemd/commit/176f73272e6e3116ca...
that makes a dramatic difference for those hitting this case, but it only gets things from nearly unusable to slow-as-usual.
Oh how I love totally predictable poetterings reply to previous bug report that got closed because "measuring it wrong" and "this is not a support forum".
Systemd should just use DuckDB. It's perfect for this job.
"But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"
Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.
Parquet is probably an even better option. Columnar, compression, fast, succinct. All good things.
You can read them with DuckDB, but you don't end up with O(log n) writes -- which is, to speak plain English, batshit fucking insane for a system logger.
What those cursed writes buys you is O(log n) reads, but there's just no scenario that is necessary. If you have literally any time or subsystem constraints, parquet's predicate pushdowns means you get plenty fast access even with a full scan.
The "gift" of systemd never stops. It's like an STD, just spreading rot across everything it touches