command[check_timer]=/usr/lib/nagios/plugins/check_file_age -w 7200 -c 14400 -f /usr/local/playfirst/run/var/TimerService.${HOSTNAME}.alive

and when reading it from nagios_local.cfg, the nrpe daemon reported it as:

/usr/lib/nagios/plugins/check_file_age -w 7200 -c 14400 -f /usr/local/playfirst/run/var/TimerService.${HOSTNAME}.alive$

Oops.

Because then it goes on to substitute the client hostname for ${HOSTNAME} but it turns out the $ at the end of the filename persists and no, there is no file of that name lying around.  So that’s kind of a curious artifact of trying to use an environment variable substitution.  The variable does substitute but you keep a bonus $ at the end.  So then I used a bit of sed trickery so that I push out N different config files, each with the hostname hardcoded.

But it still didn’t work.  Because it turns out that daemon runs as nagios, and that file was in a place where only user and group could see it.  Rather than turn on the bits for everyone else, I decided to let nrpe run  its commands with the sudo prefix option.

Meanwhile, on the server, all of these were flagging file not found.  I sure wish the second error state would have been reported as ‘no permission to read file FOO’ and the first, I wish the substitution didn’t have that weird side effect.

Yeah, I could write patch or patches for these and submit upstream but I’ve had bad luck providing patches to Debian and Ubuntu in the past.  I have to assume it’s something about me at this point.  But if anyone wants to patch something, these are things which bugged me this last week and made my job harder.