Obsolete Your Idols

I got an early start this day because I woke early for no real reason. So I headed into the office, had some breakfast there (grilled cheese with fried egg and coffee, thanks for asking) and got down to work.

That meant installing Evernote for OS X so I could attach a PDF to a note, then using the information I’d gathered there to write up a proposal concerning alerting. I am trying to be thorough in documenting what I do and why. I tried to capture my thinking, rationalize my decision, and foreshadow future developments. As part of the research for the writing I think I noticed something odd about Pingdom’s pricing.

I’m probably misunderstanding something. But if the costs per check aren’t different at the Business plan level, and I don’t care about SMS notifies, what is my incentive to ever leave the Basic plan? My efficient frontier seems like it’s up and to the left and with a linear progression, it’s a Basic ballgame.

If the cost per check on the business plan is less, then the graph is wrong and there is a break-even point on the expense of checks. But it sure doesn’t look like it.

I ran my proposal for additional monitoring past my boss and got his buy-in and then started deploying it. So that’s my first operational task which is not entirely reactive in nature; there had been an issue earlier with a system going away and no one noticing, but it wasn’t a production system and I was more interested in getting some kind of alerting going for those systems as they come online.

I fully expect to be iterating on the deployed monitoring solution, as it was a trade-off between results and costs (financial and my time/effort/brain) and there are arguably better solutions I didn’t feel like I could invest enough into at this point to get the most out of. It’s a starting point, an incremental improvement over what the company had in place before me.

This wasn’t quite a No Changes Friday (the only religious holiday I observe) but  it seemed worth it to push through that sabbath to get additional awareness of the environment. Arguably it didn’t impact anything production-related, beyond the tiny additional impact of the monitoring checks being done, which are all tickling network listening daemons.

Then I spent the rest of my day researching options for my next proposal, which will be SSL related.

Today I gained a bucket of credentials and started poking around at what exists and what services are being used.

The company has a pair of domain registrars they’ve used, a leased hosting provider, a cloud provider, a connectivity monitoring service, a code repository hosting service. They’ve got the beginnings of a series of development environments, of a production environment, of a staging environment.

So now I feel like I have my hands around what they’ve got going already, in a general sense. At this point, I can start drilling down into specifics and start nudging the pieces to align the way I think they should.

So other than absorb, what did I do today? I discussed with some developers and the release engineer what pieces need to be in place for a go-live coming up, and how to validate their ideas BEFORE it encounters the enemy (live users, those adorable and unforgiving creatures) and we whiteboarded some ideas for them to explore.

I didn’t get to the point I wanted to with the alerting problem I had expected to tackle today, but I’m prone to a surprising degree of optimism, considering my line of work. I did gather some numbers to start estimating costs for different options.

The main options I see:

1. extend the existing availability monitoring service to check more things
   • the cost scales linearly with the things checked
   • it only checks things which can be discerned remotely
   • it checks from multiple different places
   • it has some nice escalation options
2. hand-roll some rudimentary alerting on the most critical things
   • this is a path of brittleness and pain
   • it’s cheap, the only cost is my time and my brain
   • it could do both internal and external checks
   • it won’t scale beyond a dozen checks
3. deploy some paid alerting software package
   • this is paying someone else to have thought and worried about important things already
   • I don’t have a lot of experience with paid alerting software and the systems I did use were awkward, counter-intuitive, and had several false negatives when they should have been screaming bloody murder
   • I’m leery of spending money I’ll regret spending at this stage of things
4. deploy some opensource alerting software package
   • I’ve got some familiarity with a couple different ones
   • the price point is pretty good
   • some of them offer paid support if things get too hairy or I need it to scale faster
   • the ones I’ve used accommodate writing extensions which seems necessary; I’ve never been happy with off the shelf software

Then it becomes a question, if I’m not choosing option 1, of where to run whatever I deploy. It could go on one or more leased hosts, it could go somewhere in the cloud, or I could go full-tilt and buy some servers and co-locate them.

I’m going to finish out my math on costs but I have a hunch I know where the sweet spot will be to start out: a low end leased host running an open source package in the Nagios-space (Nagios, Icinga, Opsview). The cloud would arguably be cheaper but I’m nervous putting something I need to depend on in the cloud at this stage of things.

This day (yesterday as I write this) was less packed than the previous day but somehow just as tiring. I started the day thinking I’d be reading documentation, both for technologies I wasn’t wholly familiar with already in use, and the existing corpus on what processes and systems the company had already put in place before I arrived.

What I actually did was write documentation. Specifically, I started a spreadsheet for information about servers as they became pertinent. At this point, it’s got only four server names in it, with an IP address, where it’s located, and whether I have shell access on it, yet. It was slow going, as I had to extract it from developers who are busily readying to launch a new project.

I got myself set up with access to a Continual Integration server only to learn that this was the deprecated CI server and I actually wanted access to an entirely different CI tool on a different server.

As I’d started accumulating a wild array of bookmarks in my browser already, on top of the motley array inherited from several previous browser migration imports, I took some time to actually go through and organize my bookmarks. It felt silly at the time but as my bookmarks sync between the different places I run browsers, I only had to do it the once.

At lunch I played a 5-person game of Dominion with co-workers. My streak of not winning Dominion continued. I don’t expect it to last much longer, as they evidently play regularly, so now I have no excuse to not get serious about it.

Then my laptop halted for the first time that day, followed by the timely arrival of the replacement I was given.

Then I spent an hour installing software onto the replacement laptop. The same software I installed on the first day onto the original laptop, with one addition, Adium, for old time’s sake. At least it went faster this time as I had the list to work from in seeking software.

I got access to the company’s AWS console and that proved useful when I performed what I’d consider my first operational task of the new job: I rebooted an instance which developers could no longer remotely interact with. Which leads to what I expect today’s tasks to involve, adding additional alerting so I know there’s a problem before the users of the systems have to tell me.

Today was my first day at a new job. I’m the only operations guy minding the deployed and deploying production environment for the place. Not even replacing a previous operations staff; this is green field territory, more or less.

So here’s an abbreviated list of what I accomplished today.

• Set up the MacBookPro issued to me by installing
  • Chrome
  • TrueCrypt
  • Password Gorilla
  • Dropbox
  • KeePassX
  • Git
• Met with my boss to start iterating on a list of things for me to take over from him and from engineers.
• Had lunch with a co-worker, former and present.
• Crashed a meeting on an existing and envisioned release process which wandered far afield as I was just trying to sponge up all the information I could.
• Found out how to file a helpdesk ticket and filed one for the three inexplicable halts the MBP experienced under light use.
• Filed a helpdesk ticket because email sent to me was being delivered to a co-worker.
• Learned that the resolution was going to be swapping a whole new MBP for the one I had today, so I’ll get to re-install that software all over again.
• Got setup with the bugtracking system (Jira) because I’m going to need somewhere to track work and it’s already in place.