HomeHome ArchiveArchive

wtmp growing fast under systemd

Why wtmp filled the disk and how that was fixed.

One of our virtual machines had a gigantic /var/log/wtmp filling up the disk space:

root@vm:~# ls -l /var/log/wtmp 
-rw-rw-r-- 1 root utmp 22495488 May 12 22:46 /var/log/wtmp

When looking at it with last it would only show a small number of entries and would take an eternity to finish:

root@vm:~# last | wc -l
    26

Looking at the system log with journalctl would show entries every few seconds:

May 02 07:26:06 vm agetty[98992]: /dev/tty6: No such file or directory
May 02 07:26:11 vm systemd[1]: getty@tty5.service has no holdoff time, scheduling restart.
May 02 07:26:11 vm systemd[1]: Stopping Getty on tty5...
May 02 07:26:11 vm systemd[1]: Starting Getty on tty5...
May 02 07:26:11 vm systemd[1]: Started Getty on tty5.
May 02 07:26:11 vm agetty[98993]: /dev/tty5: No such file or directory
May 02 07:26:17 vm systemd[1]: getty@tty6.service has no holdoff time, scheduling restart.
May 02 07:26:17 vm systemd[1]: Stopping Getty on tty6...
May 02 07:26:17 vm systemd[1]: Starting Getty on tty6...
May 02 07:26:17 vm systemd[1]: Started Getty on tty6.
May 02 07:26:17 vm agetty[98994]: /dev/tty6: No such file or directory
May 02 07:26:21 vm systemd[1]: getty@tty5.service has no holdoff time, scheduling restart.
May 02 07:26:21 vm systemd[1]: Stopping Getty on tty5...
May 02 07:26:21 vm systemd[1]: Starting Getty on tty5...
May 02 07:26:21 vm systemd[1]: Started Getty on tty5.
May 02 07:26:21 vm agetty[98995]: /dev/tty5: No such file or directory

Strangely enough neither inittab would have those tty’s:

root@vm:~# cat /etc/inittab | grep tty
1:2345:respawn:/sbin/getty 38400 console
c1:12345:respawn:/sbin/getty 38400 tty1 linux
c2:12345:respawn:/sbin/getty 38400 tty2 linux
c3:12345:respawn:/sbin/getty 38400 tty3 linux
c4:12345:respawn:/sbin/getty 38400 tty4 linux

Nor would they figure in systemd unit definitions:

root@vm:~# ls /etc/systemd/system/getty.target.wants/
getty@tty1.service  getty@tty2.service  getty@tty3.service  getty@tty4.service

And they would not exists in /dev either:

root@vm:~# ls /dev/ | grep tty
tty
tty1
tty2
tty3
tty4

However for some reason they would be known to systemd nevertheless:

root@vm:~# systemctl list-units | grep getty
console-getty.service             loaded active running   Console Getty
getty-static.service              loaded active exited    getty on tty2-tty6 if dbus and logind are not available
getty@tty1.service                loaded active running   Getty on tty1
getty@tty2.service                loaded active running   Getty on tty2
getty@tty3.service                loaded active running   Getty on tty3
getty@tty4.service                loaded active running   Getty on tty4
getty@tty5.service                loaded active running   Getty on tty5
getty@tty6.service                loaded active running   Getty on tty6
system-getty.slice                loaded active active    system-getty.slice
getty.target                      loaded active active    Login Prompts

Disabling and stopping those getty@tty5.service and getty@tty6.service services got rid of the problem:

for i in 5 6; do
  systemctl disable getty@tty$i.service
  systemctl stop    getty@tty$i.service
  systemctl mask    getty@tty$i.service
done

Two other of our systems had the same behavior. Weird. Anybody in the known what’s going on here?

Systems are lxc Debian jessie.

Update: according to this systemd will cleverly regenerate the bad getty services on reboot. So you need to mask out the service too. I’ve added that to the loop above.

Update 2: if you want to fix your wtmp file you can do it like this:

root@vm:~# utmpdump wtmp                                                                                                      \
           | grep -v '\[0\] \[00000\] \[    \] \[        \] \[            \] \[                    \] \[0.0.0.0        \]'    \
           | grep -v tty5                                                                                                     \
           | grep -v tty6                                                                                                     \
           | utmpdump -r > wtmp.fixed

That will remove everything containing a tty5 or a tty6 entry (which were broken - see above) and will remove “empty” lines. Thanks to Chucky.