Why wtmp filled the disk and how that was fixed.
One of our virtual machines had a gigantic /var/log/wtmp filling up the disk space:
root@vm:~# ls -l /var/log/wtmp
-rw-rw-r-- 1 root utmp 22495488 May 12 22:46 /var/log/wtmp
When looking at it with last it would only show a small number of entries and would
take an eternity to finish:
root@vm:~# last | wc -l
26
Looking at the system log with journalctl would show entries every few seconds:
May 02 07:26:06 vm agetty[98992]: /dev/tty6: No such file or directory
May 02 07:26:11 vm systemd[1]: getty@tty5.service has no holdoff time, scheduling restart.
May 02 07:26:11 vm systemd[1]: Stopping Getty on tty5...
May 02 07:26:11 vm systemd[1]: Starting Getty on tty5...
May 02 07:26:11 vm systemd[1]: Started Getty on tty5.
May 02 07:26:11 vm agetty[98993]: /dev/tty5: No such file or directory
May 02 07:26:17 vm systemd[1]: getty@tty6.service has no holdoff time, scheduling restart.
May 02 07:26:17 vm systemd[1]: Stopping Getty on tty6...
May 02 07:26:17 vm systemd[1]: Starting Getty on tty6...
May 02 07:26:17 vm systemd[1]: Started Getty on tty6.
May 02 07:26:17 vm agetty[98994]: /dev/tty6: No such file or directory
May 02 07:26:21 vm systemd[1]: getty@tty5.service has no holdoff time, scheduling restart.
May 02 07:26:21 vm systemd[1]: Stopping Getty on tty5...
May 02 07:26:21 vm systemd[1]: Starting Getty on tty5...
May 02 07:26:21 vm systemd[1]: Started Getty on tty5.
May 02 07:26:21 vm agetty[98995]: /dev/tty5: No such file or directory
Strangely enough neither inittab would have those tty’s:
root@vm:~# cat /etc/inittab | grep tty
1:2345:respawn:/sbin/getty 38400 console
c1:12345:respawn:/sbin/getty 38400 tty1 linux
c2:12345:respawn:/sbin/getty 38400 tty2 linux
c3:12345:respawn:/sbin/getty 38400 tty3 linux
c4:12345:respawn:/sbin/getty 38400 tty4 linux
Nor would they figure in systemd unit definitions:
root@vm:~# ls /etc/systemd/system/getty.target.wants/
getty@tty1.service getty@tty2.service getty@tty3.service getty@tty4.service
And they would not exists in /dev either:
root@vm:~# ls /dev/ | grep tty
tty
tty1
tty2
tty3
tty4
However for some reason they would be known to systemd nevertheless:
root@vm:~# systemctl list-units | grep getty
console-getty.service loaded active running Console Getty
getty-static.service loaded active exited getty on tty2-tty6 if dbus and logind are not available
getty@tty1.service loaded active running Getty on tty1
getty@tty2.service loaded active running Getty on tty2
getty@tty3.service loaded active running Getty on tty3
getty@tty4.service loaded active running Getty on tty4
getty@tty5.service loaded active running Getty on tty5
getty@tty6.service loaded active running Getty on tty6
system-getty.slice loaded active active system-getty.slice
getty.target loaded active active Login Prompts
Disabling and stopping those getty@tty5.service and getty@tty6.service services got
rid of the problem:
for i in 5 6; do
systemctl disable getty@tty$i.service
systemctl stop getty@tty$i.service
systemctl mask getty@tty$i.service
done
Two other of our systems had the same behavior. Weird. Anybody in the known what’s going on here?
Systems are lxc Debian jessie.
Update: according to this systemd will cleverly regenerate the bad getty services on reboot. So you need to mask out the service too. I’ve added that to the loop above.
Update 2: if you want to fix your wtmp file you can do it like this:
root@vm:~# utmpdump wtmp \
| grep -v '\[0\] \[00000\] \[ \] \[ \] \[ \] \[ \] \[0.0.0.0 \]' \
| grep -v tty5 \
| grep -v tty6 \
| utmpdump -r > wtmp.fixed
That will remove everything containing a tty5 or a tty6 entry (which were broken - see above) and will remove “empty” lines. Thanks to Chucky.