Obsolete Your Idols

This was the last day before a 3 day weekend and as is customary around these parts, not many people came in and the ones who did left early. I didn’t really achieve anything worth talking about, just researched some more ideas for my next few proposals.

Specifically, I looked at AWS documentation about Elastic Block Storage, I looked at new Bacula features, I looked at Chef and specifically Knife.

Then I went to drink with co-workers, former and present.

This day was my first day working from home at the new job, something I negotiated to get for myself. One day a week, I work from home. Unfortunately as today was the second attempt at releasing, and things took about as long as they do the first time you do something operational, this was a 14 hour day of work for me. At least I didn’t need pants to do it.

Our proposed deploy process didn’t survive its encounter with the actual server environment but that doesn’t come as a surprise; it’s still very raw and will be refined a lot, soon.

I had hoped to WFH on this day but we were on track to release something important so I came in to the office.

First order of the day was sharing the Windows virtualbox file with my co-workers so they could fire up IE and validate things work in that browser, too.

That led in to a playdate for something on the web, followed by a process meeting about deployment which turned into a pair programming / code review session using a laptop jacked into a projector. I recommend this if you have more than one developer in a meeting, use the projector to display their code as you talk about related topics.

This was my first Monday in the new office and so the focus was on all the orientation activity they only do on Mondays. I got my picture taken for a security badge (taken with a smart phone) and spent the day following up on orientation information as well as researching SSL certificate options before proposing that the company pay for a wildcard SSL certificate for use in production environments and that we self-sign a different wildcard SSL certificate for use in development and testing environments.

Then I dug in to the documentation for how to use an SSL certificate with jboss. That was more opaque than it needed to be and I’ll probably explain what I did in the next post.

I got an early start this day because I woke early for no real reason. So I headed into the office, had some breakfast there (grilled cheese with fried egg and coffee, thanks for asking) and got down to work.

That meant installing Evernote for OS X so I could attach a PDF to a note, then using the information I’d gathered there to write up a proposal concerning alerting. I am trying to be thorough in documenting what I do and why. I tried to capture my thinking, rationalize my decision, and foreshadow future developments. As part of the research for the writing I think I noticed something odd about Pingdom’s pricing.

I’m probably misunderstanding something. But if the costs per check aren’t different at the Business plan level, and I don’t care about SMS notifies, what is my incentive to ever leave the Basic plan? My efficient frontier seems like it’s up and to the left and with a linear progression, it’s a Basic ballgame.

If the cost per check on the business plan is less, then the graph is wrong and there is a break-even point on the expense of checks. But it sure doesn’t look like it.

I ran my proposal for additional monitoring past my boss and got his buy-in and then started deploying it. So that’s my first operational task which is not entirely reactive in nature; there had been an issue earlier with a system going away and no one noticing, but it wasn’t a production system and I was more interested in getting some kind of alerting going for those systems as they come online.

I fully expect to be iterating on the deployed monitoring solution, as it was a trade-off between results and costs (financial and my time/effort/brain) and there are arguably better solutions I didn’t feel like I could invest enough into at this point to get the most out of. It’s a starting point, an incremental improvement over what the company had in place before me.

This wasn’t quite a No Changes Friday (the only religious holiday I observe) but  it seemed worth it to push through that sabbath to get additional awareness of the environment. Arguably it didn’t impact anything production-related, beyond the tiny additional impact of the monitoring checks being done, which are all tickling network listening daemons.

Then I spent the rest of my day researching options for my next proposal, which will be SSL related.

Today I gained a bucket of credentials and started poking around at what exists and what services are being used.

The company has a pair of domain registrars they’ve used, a leased hosting provider, a cloud provider, a connectivity monitoring service, a code repository hosting service. They’ve got the beginnings of a series of development environments, of a production environment, of a staging environment.

So now I feel like I have my hands around what they’ve got going already, in a general sense. At this point, I can start drilling down into specifics and start nudging the pieces to align the way I think they should.

So other than absorb, what did I do today? I discussed with some developers and the release engineer what pieces need to be in place for a go-live coming up, and how to validate their ideas BEFORE it encounters the enemy (live users, those adorable and unforgiving creatures) and we whiteboarded some ideas for them to explore.

I didn’t get to the point I wanted to with the alerting problem I had expected to tackle today, but I’m prone to a surprising degree of optimism, considering my line of work. I did gather some numbers to start estimating costs for different options.

The main options I see:

1. extend the existing availability monitoring service to check more things
   • the cost scales linearly with the things checked
   • it only checks things which can be discerned remotely
   • it checks from multiple different places
   • it has some nice escalation options
2. hand-roll some rudimentary alerting on the most critical things
   • this is a path of brittleness and pain
   • it’s cheap, the only cost is my time and my brain
   • it could do both internal and external checks
   • it won’t scale beyond a dozen checks
3. deploy some paid alerting software package
   • this is paying someone else to have thought and worried about important things already
   • I don’t have a lot of experience with paid alerting software and the systems I did use were awkward, counter-intuitive, and had several false negatives when they should have been screaming bloody murder
   • I’m leery of spending money I’ll regret spending at this stage of things
4. deploy some opensource alerting software package
   • I’ve got some familiarity with a couple different ones
   • the price point is pretty good
   • some of them offer paid support if things get too hairy or I need it to scale faster
   • the ones I’ve used accommodate writing extensions which seems necessary; I’ve never been happy with off the shelf software

Then it becomes a question, if I’m not choosing option 1, of where to run whatever I deploy. It could go on one or more leased hosts, it could go somewhere in the cloud, or I could go full-tilt and buy some servers and co-locate them.

I’m going to finish out my math on costs but I have a hunch I know where the sweet spot will be to start out: a low end leased host running an open source package in the Nagios-space (Nagios, Icinga, Opsview). The cloud would arguably be cheaper but I’m nervous putting something I need to depend on in the cloud at this stage of things.

This day (yesterday as I write this) was less packed than the previous day but somehow just as tiring. I started the day thinking I’d be reading documentation, both for technologies I wasn’t wholly familiar with already in use, and the existing corpus on what processes and systems the company had already put in place before I arrived.

What I actually did was write documentation. Specifically, I started a spreadsheet for information about servers as they became pertinent. At this point, it’s got only four server names in it, with an IP address, where it’s located, and whether I have shell access on it, yet. It was slow going, as I had to extract it from developers who are busily readying to launch a new project.

I got myself set up with access to a Continual Integration server only to learn that this was the deprecated CI server and I actually wanted access to an entirely different CI tool on a different server.

As I’d started accumulating a wild array of bookmarks in my browser already, on top of the motley array inherited from several previous browser migration imports, I took some time to actually go through and organize my bookmarks. It felt silly at the time but as my bookmarks sync between the different places I run browsers, I only had to do it the once.

At lunch I played a 5-person game of Dominion with co-workers. My streak of not winning Dominion continued. I don’t expect it to last much longer, as they evidently play regularly, so now I have no excuse to not get serious about it.

Then my laptop halted for the first time that day, followed by the timely arrival of the replacement I was given.

Then I spent an hour installing software onto the replacement laptop. The same software I installed on the first day onto the original laptop, with one addition, Adium, for old time’s sake. At least it went faster this time as I had the list to work from in seeking software.

I got access to the company’s AWS console and that proved useful when I performed what I’d consider my first operational task of the new job: I rebooted an instance which developers could no longer remotely interact with. Which leads to what I expect today’s tasks to involve, adding additional alerting so I know there’s a problem before the users of the systems have to tell me.

Today was my first day at a new job. I’m the only operations guy minding the deployed and deploying production environment for the place. Not even replacing a previous operations staff; this is green field territory, more or less.

So here’s an abbreviated list of what I accomplished today.

• Set up the MacBookPro issued to me by installing
  • Chrome
  • TrueCrypt
  • Password Gorilla
  • Dropbox
  • KeePassX
  • Git
• Met with my boss to start iterating on a list of things for me to take over from him and from engineers.
• Had lunch with a co-worker, former and present.
• Crashed a meeting on an existing and envisioned release process which wandered far afield as I was just trying to sponge up all the information I could.
• Found out how to file a helpdesk ticket and filed one for the three inexplicable halts the MBP experienced under light use.
• Filed a helpdesk ticket because email sent to me was being delivered to a co-worker.
• Learned that the resolution was going to be swapping a whole new MBP for the one I had today, so I’ll get to re-install that software all over again.
• Got setup with the bugtracking system (Jira) because I’m going to need somewhere to track work and it’s already in place.

I needed to know what the disk utilization on a system was, essentially at all times, with a granularity of one second. Asking for the current iostat every five minutes via munin wasn’t sufficient. So I wrote this munin plugin. It tries to read a file and report the average and maximum disk utilization or util% logged in that file. Then it unlinks the file and starts an iostat running every second for 250 seconds, recorded in that file, setting it up for the next time it’s polled.

It looks like this. I don’t think I have any reason to obfuscate the vertical axis of this picture unlike the #devops people. It’s the graph from an uninteresting system doing some periodic nightly batch work.

It has not escaped me that it reports average averages, maximum averages, average maximums, and maximum maximums.

Here’s the script.

!/usr/bin/perl



# iosmart by Shannon Prickett <shannon.prickett@gmail.com>

# get iostat data and provide max and average values since the last check.



use strict;

use warnings;



use File::stat;



use vars qw{ $argument $iostat_file $iostat_runs };

use subs qw{ print_header };



$iostat_file = '/var/tmp/iosmart';

$iostat_runs = 250; # normally 250, reduce when testing



$argument = $ARGV[0] || 'NONE';



if ($argument =~ /config/) { # hint to munin

  # munin-run calls this to set up graphs, check limits

  print <<EOM;

graph_title Extended iostat coverage

graph_vlabel utilization

graph_category disk

graph_info Collect iostat data every second

sda_max.label sda max util

sda_avg.label sda avg util

sdb_max.label sdb max util

sdb_avg.label sdb avg util

EOM

}

else { # do the work



  if (( -e $iostat_file) &&  # it exists

    ( -r $iostat_file) &&  # we can read it

    ( -s $iostat_file) ) {  # it has something in it



    my $st = stat( $iostat_file ) or

      die "can't stat $iostat_file: $!\n";



    my $mtime = $st->mtime;

    my $now = time( );

    my $delta = $now - $mtime;



    if ( $delta > 300 ) {  # it's been >5 minutes 

      print "${iostat_file} is stale, using it anyway\n";

    }



    open( my $fh, '<', $iostat_file ) or 

      die "failed to open $iostat_file: $!\n";

  

    my %devices;

    my ($max, $skip_until, $stop_skipping);

    READLOOP: while ( my $line = <$fh> ) {

      next READLOOP unless ( $line =~ /^(sd\w).*?(\d+\.\d+)$/ ); # we only care about the disk rows



      # we want to skip the first block of iostat output

      # that's output since boot. we only care about now.

      unless ( defined $skip_until ) {

        $skip_until = $1;

        next READLOOP;

      }



      my $device = $1;  # per device values

      my $util = $2;  # keep the number from the last dolume



      if (( ! defined $stop_skipping ) && # if we are still skipping

        ( $device ne $skip_until ) ) {  # this isn't what we're looking for

        next READLOOP;

      }

      else {

        $stop_skipping = 1;

      }

    

      chomp $util;  # we hates filthy newlines forever

    

      if (( ! exists $devices{$device}{'count'} ) or  # first time

        ( ! defined $devices{$device}{'count'} ) ) {

        $devices{$device}{'count'} = 1;

      }

      else {

        $devices{$device}{'count'} = $devices{$device}{'count'} + 1;

      }



      $devices{$device}{'sum'} += $util;



      if (( ! exists $devices{$device}{'max'} ) or  # we've never set it before

        ( ! defined $devices{$device}{'max'} ) or  # the whole world is crazy

        ( $devices{$device}{'max'} < $util ) ) {  # it's smaller than current

        $devices{$device}{'max'} = $util; 

      }



    }



    for my $device (keys %devices) {

      $devices{$device}{'average'} = $devices{$device}{'sum'} / $devices{$device}{'count'};



      print "${device}_max.value $devices{$device}{'max'}\n";

      print "${device}_avg.value $devices{$device}{'average'}\n";

    }



    close( $fh ) or die "can't close ${iostat_file}? wtf? $!\n";



    unlink( $iostat_file ) or die "can't rm ${iostat_file}: $!\n";



    print_header( );



    # suppress an error about inappropriate ioctl by not testing exit. :(

    system( "iostat -xd 1 $iostat_runs >> $iostat_file &" );

  }

  else {

    print "can't use ${iostat_file}; making new\n";

    open( my $fresh, '>', $iostat_file ) or 

      die "can't make new ${iostat_file}\n";

    print_header( );

  }

}



sub print_header {

  open( my $header, '>>', $iostat_file ) or 

    die "failed to start ${iostat_file}: $!\n";



  print $header "# this file is created by the iosmart munin-plugin\n";

  print $header "# if it's not updating, check the munin-node logs\n";



  close( $header ) or die "can't stop heading ${iostat_file}? HMPH $!\n";

}

ETA: using a new code formatting plugin

During the last two months I have spent a lot of time in close proximity to mongodb. Enough so that I feel like I’ve learned some things I should pass on. These are rooted in mistakes I have made and survived.

• Think really hard about why you need mongodb. This is not because mongodb is bad, but it may be the wrong choice for what you’re trying to do.
  • Do you know what data will be the most valuable to you from your app? Then don’t use mongodb.
  • Can you afford to take downtime to modify the relationships between your data? Then don’t use mongodb.
  • Do you have existing trustable models which project the expected growth for your data? Then don’t use mongodb.
  • Are there one or more glaring unknowns about the data from your app? Then do use mongodb.
• Get yourself in a support contract with 10gen. They are the best source of mongodb information, advice, and help.
• Your  minimum production environment is nine servers.
  • Your servers should as congruent as you can get them for the same reason the drives in a RAID should be.
  • You’re going to want a dedicated RAID-10 device for each mongod process’s datafiles.
  • You want the drives to be maximized for speed. RPMs matter more than capacity when you are getting started.
  • RAM is the other thing which mongod is hungry for. The more the better.
  • The nine servers will be distributed like so
    

    shard 1	shard 2	shard 3
    primary replSet member	primary replset member	primary replSet member
    secondary replSet member	secondary replSet member	secondary replSet member
    delayed secondary replSet member + configdb	delayed secondary replSet member + configdb	delayed secondary replSet member + configdb

  • This is a reasonably robust setup to distribute data in a redundant manner. You can fail over to the non-delayed secondary in any given replSet, you can do stop+copy backups of your data at the delayed secondary member, you can put all three replSets into a shard and avoid some lopsided conditions from, for example, disabling the Balancer process. If you have to cut costs, you can skimp on the bottom row of servers. Try not to have to do that.
• Never delete anything. Have enough disk that you can ‘remove’ files by moving them to another place on the same system. This applies to configuration files, data files, log files, anything mongodb related.
• Find out about numactl. This is a good clearinghouse post about numactl as it applies to mysql. At large core + large memory sizes, it applies to mongodb, as well.
• Graph everything you can. Iostat, memory usage, swap usage, mongodb operations, throughput at every layer in front of mongodb. You’ll need to know when a problem is with mongodb and when it’s higher or lower in the stack.
• Run your mongod instances on alternate ports from the default.
• Favor mv and scp over rsync when moving mongodb data files around.