# Journal log messages duplicated

**URL:** https://hub.mender.io/t/journal-log-messages-duplicated/5417
**Category:** General Discussions
**Created:** [November 17, 2022, 12:39pm UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417 "2022-11-17T12:39:45Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![pamolloy](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/pamolloy/32/1789_2.png) [@pamolloy](https://hub.mender.io/u/pamolloy)
#### Post date: [November 17, 2022, 12:39pm UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/1 "2022-11-17T12:39:45Z")

</div>

I’ve installed Mender on my target using Buildroot. When checking the logs for the client I noticed that error and warning log messages are duplicated, but info are not. Once with normal formatting and then again with either bold yellow or red formatting. Perhaps this is just a systemd journal configuration issue on my end, but I was curious if anyone has experienced this same issue.

---

<div class="post-metadata">

### Author: ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)
#### Post date: [November 18, 2022, 7:55am UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/2 "2022-11-18T07:55:24Z")

</div>

Hi @pamolloy,

Thanks for reporting this! As we understand it, this is caused by the client both generating output which is captured by journald, as well as directly writing to syslog. So if journald also writes to the syslog, then you see the duplication. We’re looking into it now.

Greetz,  
Josef

---

<div class="post-metadata">

### Author: ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)
#### Post date: [November 21, 2022, 9:18pm UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/3 "2022-11-21T21:18:08Z")

</div>

Hi @pamolloy,

To confirm our analysis, can you please provide the output of

```auto
file /dev/log /var/run/syslog /var/run/log

```

and the `.service` file in use for the system logger, presumably `syslogd`, `syslog-ng` or `rsyslog`, and while you’re at it, also `systemd-journald`? (EDIT: not `mender-client`)

Thanks!

---

<div class="post-metadata">

### Author: ![pamolloy](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/pamolloy/32/1789_2.png) [@pamolloy](https://hub.mender.io/u/pamolloy)
#### Post date: [November 22, 2022, 10:46am UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/4 "2022-11-22T10:46:04Z")

</div>

```auto
# file /dev/log /var/run/syslog /var/run/log
/dev/log: symbolic link to /run/systemd/journal/dev-log
/var/run/syslog: cannot open `/var/run/syslog' (No such file or directory)
/var/run/log: directory

```

I’m only using systemd-journald and the main service file comes from systemd. I’ve added a diff from the latest upstream template for the service file.

> <https://github.com/systemd/systemd/blob/main/units/systemd-journald.service.in>

```auto
diff systemd-journald.service.in /usr/lib/systemd/system/systemd-journald.service 
--- systemd-journald.service.in
+++ /usr/lib/systemd/system/systemd-journald.service
@@ -15,14 +15,9 @@
 After=systemd-journald.socket systemd-journald-dev-log.socket systemd-journald-audit.socket syslog.socket
 Before=sysinit.target
 
-# Mount and swap units need the journal socket units. If they were removed by
-# an isolate request the mount and swap units would be removed too, hence let's
-# exclude systemd-journald and its sockets from isolate requests.
-IgnoreOnIsolate=yes
-
 [Service]
 DeviceAllow=char-* rw
-ExecStart={{ROOTLIBEXECDIR}}/systemd-journald
+ExecStart=/usr/lib/systemd/systemd-journald
 FileDescriptorStoreMax=4224
 IPAddressDeny=any
 LockPersonality=yes
@@ -44,7 +39,7 @@
 SystemCallErrorNumber=EPERM
 SystemCallFilter=@system-service
 Type=notify
-{{SERVICE_WATCHDOG}}
+WatchdogSec=3min
 
 # In case you're wondering why CAP_SYS_PTRACE is needed, access to
 # /proc/<pid>/exe requires this capability. Thus if this capability is missing
@@ -53,5 +48,4 @@
 
 # If there are many split up journal files we need a lot of fds to access them
 # all in parallel.
-LimitNOFILE={{HIGH_RLIMIT_NOFILE}}
-
+LimitNOFILE=524288

```

Note that when I run `mender daemon` manually I get the following message three times:

```auto
WARN[0063] Returning artifact name from /etc/mender/artifact_info file. This is a fallback, in case the information can not be retrieved from the database, and is only expected when an update has never been installed before. 
WARN[0064] Returning artifact name from /etc/mender/artifact_info file. This is a fallback, in case the information can not be retrieved from the database, and is only expected when an update has never been installed before. 
WARN[0068] Returning artifact name from /etc/mender/artifact_info file. This is a fallback, in case the information can not be retrieved from the database, and is only expected when an update has never been installed before. 

```

---

<div class="post-metadata">

### Author: ![pamolloy](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/pamolloy/32/1789_2.png) [@pamolloy](https://hub.mender.io/u/pamolloy)
#### Post date: [November 22, 2022, 11:06am UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/5 "2022-11-22T11:06:12Z")

</div>

Is there a way to prevent the warning about `artifact_info` from getting printed? Or maybe the default config is causing that polling that too often? I feel like I’m missing some persistent storage for Mender because even after several updates I still see this warning.

---

<div class="post-metadata">

### Author: ![snusifer](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/snusifer/32/1792_2.png) [@snusifer](https://hub.mender.io/u/snusifer)
#### Post date: [November 23, 2022, 11:24am UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/6 "2022-11-23T11:24:09Z")

</div>

Hello @pamolloy ,

Thanks for bringing this up.

Regarding the duplicate log messages in the journal, in your configuration journald symlinks `/dev/log` to `/run/systemd/journal/dev-log`, which is a socket that journald listens to. Additionally, mender-client is configured by default to forward log entries to the standard syslogger ( `syslogd`, `syslog-ng` or `rsyslog`). Most sysloggers listen to `/dev/log`. Presumably, this is why duplicate log messages happen as two identical streams are fed into journald; one from `/run/systemd/journal/dev-log` and one from `stdout`.

To fix the problem, try the following:

- In `/lib/systemd/system/mender-client.service`, change

`ExecStart=/usr/bin/mender daemon`  
to  
`ExecStart=/usr/bin/mender --no-syslog daemon`.

- (optionally) if you want log messages to be forwarded to `/var/log/syslog`, make sure the standard syslogger listens to `/run/systemd/journal/syslog` and set `ForwardToSyslog=yes` in your `/etc/systemd/journald.conf`.

Hope this helps 🙂

---

<div class="post-metadata">

### Author: ![snusifer](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/snusifer/32/1792_2.png) [@snusifer](https://hub.mender.io/u/snusifer)
#### Post date: [November 23, 2022, 12:50pm UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/7 "2022-11-23T12:50:52Z")

</div>

@pamolloy

With regards to the `artifact_info` point that you brought up, I just need to clarify some things. So to understand this correctly, did you run the `mender-client` in the shell alongside a `mender-client` already running and managed by `systemd`, while doing updates?

The one spawned in the shell, does it run with root privileges?

What version of mender are you running?

---

<div class="post-metadata">

### Author: ![pamolloy](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/pamolloy/32/1789_2.png) [@pamolloy](https://hub.mender.io/u/pamolloy)
#### Post date: [November 23, 2022, 1:02pm UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/8 "2022-11-23T13:02:37Z")

</div>

Thanks for the detailed reply! `--no-syslog` is exactly what I needed.

---

<div class="post-metadata">

### Author: ![pamolloy](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/pamolloy/32/1789_2.png) [@pamolloy](https://hub.mender.io/u/pamolloy)
#### Post date: [November 23, 2022, 1:04pm UTC](https://hub.mender.io/t/journal-log-messages-duplicated/5417/9 "2022-11-23T13:04:30Z")

</div>

I believe this was the result of not persisting Mender data between updates. I now pass a directory that is persisted to Mender with `--data`.
