logrotate and bash

It took me a while (longer that I should admit) to figure out how to make daemon processes written in bash, work properly with logrotate so that the output from bash gets properly rotated, compressed, closed, and re-opened.

Say, you’re doing this in bash:

#!/bin/bash

logfile=somelog.txt

while :; do

echo -n "Today's date is" >>  ${logfile}

echo date >> ${logfile} 

sleep 60

done

This will run forever, adding a line to the noted logrotate file every minute.  Easy enough, and if logrotate is asked to rotate the somelog.txt file, it will do so happily.

But what if bash has started a process that itself takes a long time to complete:

#!/bin/bash

logfile=somelog.txt

find / -type f -exec cat \{\} \; >>  ${logfile}

which, I think we’d agree, will take a long time.  During this time, it keeps the logfile open for writing.  If logrotate then fires to rotate it, we will lose all data written to the logfile after the rotate occurs.  The find continues to run, but the results are lost.  This isn’t really what we want.

The solution is to change how logs are written.  Instead of using the > ${logfile} syntax, we’re going to let bash itself do the writing.

#!/bin/bash

logfile=somefile.txt

exec 1>>${logfile} 2>&1

find / -type f -exec cat \{\} \;

Now, the output from the find command is written to its stdout, which winds up on bash’s stdout, which because of the exec command there, writes it to the logfile.  If logrotate fires here, we’ll still lose any data written after the rotate.  To solve this, we’d need to have bash close and re-open its logfile.

Logrotate can send a signal, say SIGHUP, to a process, when it rotates its logfile out from underneath it.  On receipt of that signal, the process should close its logfile and reopen it. Here’s how that looks in bash:

#!/bin/bash

logfile=somelog.txt

pidfile=pidfile.txt

function sighup_handler()

{

exec 1>>${logfile} 2>&1

}

trap sighup_handler HUP

trap "rm -f ${pidfile}" QUIT EXIT INT TERM

echo "$$" > ${pidfile}

# fire the sighup handler to redirect stdout/stderr to logfile

sighup_handler

find / -type f -exec cat \{\} \;

and we add to our logrotate snippet:

somelog.txt {

daily

rotate 7

missingok

ifempty

compress

compresscmd /usr/bin/bzip2

uncompresscmd /usr/bin/bunzip2

compressext .bz2

dateext

copytruncate

postrotate

    /bin/kill -HUP `cat pidfile.txt 2>/dev/null` 2>/dev/null || true

endscript

}

Now, when logrotate fires, it sends a SIGHUP signal to our long-running bash process.  Bash catches the SIGHUP, closes and re-opens its logfiles (via the exec command), and continues writing.  There is a brief window between when the logrotate fires, and when bash can re-open the logfile, where those messages may be lost, but that is often pretty minimal.

There you have it.  Effective log rotation of bash-generated log files.

(Update 7/5: missed the ‘copytruncate’ option in the logrotate config before, added it now.)

5 comments on this post.
  1. Joe Klemmer:

    Nice and relatively simple. Thanks for sharing.

  2. syskill:

    I’ve tested this and it doesn’t work. :-( The find process inherits *copies* of the bash process’ stdout and stderr file descriptors as it is spawned. When the bash process reopens its file descriptors, that has no bearing on the find process.

  3. Resuna:

    As I was reading through this, I was thinking that UNIX file descriptors didn’t work that way… the previous commenter isn’t quite correct as to how it works: it’s not that the find gets a complete copy of the file descriptor (it doesn’t, it shares a seek position with bash), but once bash closes that fd it doesn’t matter what it does past that point, but the fact remains that it doesn’t work.

    The correct way to handle logging is to write to a pipe to a program that’s logrotate-aware, or to write to a program that feeds lines to syslog. There are a variety of such programs, I seem to recall there being one included in qmail or tcptools.

  4. adobriyan:

    See “copytruncate”.

  5. Resuna:

    Aha! I didn’t know about that option. Copytruncate should work in most cases, because redirection onto the end of a file opens the file in “append” mode, so it seeks to the end on every write.