Cursory searches turned up 5 public git repositories containing Puppet modules to manage Postfix. Success! But, too much success! How could I pick the right one? I had some rough idea of what I wanted the one I used to do but maybe I’d overlooked some important feature. I’d better compare them. To start us off, here’s the list in the order the search found them.

• camptocamp on github
• example42 on github
• ghoneycutt on github
• deck on github
• riseup-shared

So I did what any self-respecting lazy person would, I fired off 5 clones in parallel and went to force  a machine to make me coffee.

Once everything was in place, I started doing tree and diff in various places to get a feel for how the repos were laid out, which features were visible in module names, which systems were supported. The very first thing I noticed is that the deck and riseup repos are virtually identical, and so I surmise lazily that the latter is built upon the former. This also meant that I can eliminate deck from consideration in favor of riseup. So now there were four.

But now it’s time to drink that coffee.

node default {

  package { 'rubygems':

    ensure => installed,

  }



  exec { 'gem install rspec-puppet > /tmp/rspec.out':

    path => '/usr/bin',

    creates => '/var/lib/gems/1.8/gems/rspec-puppet-0.1.3/',

    require => Package['rubygems'],

  }

}

OK, so what’s that all about?

What it does

It installs the package named rubygems which, on at least some Debian-ish systems, provides the ‘gem’ command, beloved of Rubyists and with which I’m attempting to become better acquainted THEN it runs a command to install the rspec-puppet gem, IF the system doesn’t already have version 0.1.3 of the gem installed in a ruby version 1.8 environment. It does these things if you put it onto a system which has puppet installed and execute the command ‘sudo puppet apply <file>’ on newer puppet installs or just ‘sudo puppet <file>’ on older puppet installs.

Interestingly, this is the second version of this manifest. The first lacked the require attribute and so when I ran it on my test system, it failed to execute the gem install, first, and then installed the rubygems package. Puppet is tricky like that and if you require things happen in a certain order, you have to make that happen. You could consider that a strength or an annoyance; I didn’t think much about it until I saw a presentation by Jason Wright at Puppetcamp 2010 where he talked about how his group had designed their Puppet manifests to run once and if a second run introduced a change, it was considered an error.

Why do this?

So continuing with my simplest thing iteration of using AWS, I thought I’d try to bootstrap up a masterless Puppet environment. Why? Well, because I have no idea what my environment will look like. At this point I should probably emphasize that all of this is for my own personal AWS stuff and while I do use Puppet at my dayjob and I do use AWS at my dayjob, my usage of both is quite a bit different, there.

More specifically, we have carved out essentially a standard-if-evolving application stack and put infrastructure in place to support that but here I’m just sort of dabbling with ideas. One of those ideas is the idea of running Puppet without a Puppetmaster node or nodes.

It used to be a somewhat heretical idea to do without a Puppetmaster, though I gather PuppetLabs now fully embraces the idea, which is good. While having a central Puppetmaster has some advantages (centralized source of truth, centralized reporting), it has (for me, in AWS) some disadvantages (requires a dedicated persistent node, requires managing The Cert Situation, potentially requires special scaling considerations). There are many many use cases where running a Puppetmaster is brilliant idea.
But if you don’t run one, you trade that for needing to solve some things on your own.

The problems I’m solving right now are ‘where does a node get its catalog’ (my answer of the moment: github) and ‘how does the node know what manifests to apply’ (maotm: a manual application via puppet agent invocations). Neither of these are yet well-solved by me, I’d say, but I’m bootstrapping to something which I suspect will look more like ‘catalogs live in S3, the nodes use s3fs to get them, instance tags are used to decide which manifests to apply’. But I think I’m on the right track by writing manifests which refer to node default, because now I don’t have to worry about updating if my instance identifier changes.

Oh, and the whole point of this stupid manifest is that now I can spin up an instance to develop my Puppet catalog on and get ready to test my Puppet manifests. I haven’t quite convinced myself that I should write a test to test this manifest which sets up the ability to test a manifest (y0, d4wg) but doing so would have a sickly sweet kind of self-referentialism to it.

What I did:

• got an Amazon Web Services account; that was easy, because I’m a white dude in America
• went to the Ubuntu UEC page
• clicked on one of the (EBS!) images which took me to the AWS launch instances page
• went through all the clicky bits to make sure that the instance would allow ssh traffic in the security group
• logged into the instance, and sudo apt-get update && sudo apt-get dist-upgrade’d it
• installed puppet because puppet is fucking awesome
• clicked on Create Image from the AWS instances webpage
• waited for that to finish, then terminated the running instance
• launched an instance from the AMI I’d made
• logged into the instance and did sudo apt-get install git because, yes, git is fucking awesome
• clicked on Create Image again
• waited for that to finish, then terminated the running instance

So now I’ve got an AMI for doing development for my personal projects. Which is not, at this time, a euphemism for pornography.

Then I got too busy with outside of work to write about non-work.

Then it became A Thing, that I’d gone so long without blogging.

So that’s why I’m not blogging.

Specifically, I looked at AWS documentation about Elastic Block Storage, I looked at new Bacula features, I looked at Chef and specifically Knife.

Then I went to drink with co-workers, former and present.

Our proposed deploy process didn’t survive its encounter with the actual server environment but that doesn’t come as a surprise; it’s still very raw and will be refined a lot, soon.

First order of the day was sharing the Windows virtualbox file with my co-workers so they could fire up IE and validate things work in that browser, too.

That led in to a playdate for something on the web, followed by a process meeting about deployment which turned into a pair programming / code review session using a laptop jacked into a projector. I recommend this if you have more than one developer in a meeting, use the projector to display their code as you talk about related topics.

I started the morning using a company card to pay for SSL certificates and then spent the rest of the day making them work with JBoss and Apache. The Apache part was the easy part, as the certificates were already in the correct form. Using Subject Alternate Names with the wildcard certificate meant that I could secure a variety of systems using it, with varying depths of subdomains.

But JBoss was a problem and here’s why. All the examples I could find on the web gloss over using a real certificate. They assume that it’s good enough to show you the syntax for a self-signed certificate you generate using keytool. In my case, it wasn’t. The process for using a CA-signed certificate turned out to be very different.

So here’s the key thing to know if you try this craziness. Keytool is a vicious betrayer of hope and will quietly do the wrong thing if you don’t fully grok what it does/wants. You need to turn to something else, in this case, openssl. This command will take your private key and put it in a form keytool can use:

openssl pkcs8 -topk8 -nocrypt -in /your/private/ssl.key -inform PEM -out /someplace/safe/key.der -outform DER

Then this command will turn your SSL certificate into something keytool can use:

openssl x509 -in /your/private/ssl.crt -inform PEM -out /someplace/safe/cert.der -outform DER

Then you run those files through a program someone else wrote. Not entirely excited about it but I read the source and it didn’t seem too dire.

import java.security.*;
import java.io.IOException;
import java.io.InputStream;
import java.io.FileInputStream;
import java.io.DataInputStream;
import java.io.ByteArrayInputStream;
import java.io.FileOutputStream;
import java.security.spec.*;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;
import java.util.Collection;
import java.util.Iterator;

/**
* ImportKey.java
*
*

This class imports a key and a certificate into a keystore
* ($home/keystore.ImportKey). If the keystore is
* already present, it is simply deleted. Both the key and the
* certificate file must be in DER-format. The key must be
* encoded with PKCS#8-format. The certificate must be
* encoded in X.509-format.

*
*

Key format:

*

openssl pkcs8 -topk8 -nocrypt -in YOUR.KEY -out YOUR.KEY.der

* -outform der

*

Format of the certificate:

*

openssl x509 -in YOUR.CERT -out YOUR.CERT.der -outform

* der

*

Import key and certificate:

*

java comu.ImportKey YOUR.KEY.der YOUR.CERT.der
*
*

Caution: the old keystore.ImportKey-file is
* deleted and replaced with a keystore only containing YOUR.KEY
* and YOUR.CERT. The keystore and the key has no password;
* they can be set by the keytool -keypasswd-command for setting
* the key password, and the keytool -storepasswd-command to set
* the keystore password.
*

The key and the certificate is stored under the alias
* importkey; to change this, use keytool -keyclone.
*
* Created: Fri Apr 13 18:15:07 2001
* Updated: Fri Apr 19 11:03:00 2002
*
* @author Joachim Karrer, Jens Carlberg
* @version 1.1
**/
public class ImportKey {

/**
*

Creates an InputStream from a file, and fills it with the complete
* file. Thus, available() on the returned InputStream will return the
* full number of bytes the file contains

* @param fname The filename
* @return The filled InputStream
* @exception IOException, if the Streams couldn’t be created.
**/
private static InputStream fullStream ( String fname ) throws IOException {
FileInputStream fis = new FileInputStream(fname);
DataInputStream dis = new DataInputStream(fis);
byte[] bytes = new byte[dis.available()];
dis.readFully(bytes);
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
return bais;
}

/**
*

Takes two file names for a key and the certificate for the key,
* and imports those into a keystore. Optionally it takes an alias
* for the key.
*

The first argument is the filename for the key. The key should be
* in PKCS8-format.
*

The second argument is the filename for the certificate for the key.
*

If a third argument is given it is used as the alias. If missing,
* the key is imported with the alias importkey
*

The name of the keystore file can be controlled by setting
* the keystore property (java -Dkeystore=mykeystore). If no name
* is given, the file is named keystore.ImportKey
* and placed in your home directory.
* @param args [0] Name of the key file, [1] Name of the certificate file
* [2] Alias for the key.
**/
public static void main ( String args[]) {

// change this if you want another password by default
String keypass = “importkey”;

// change this if you want another alias by default
String defaultalias = “importkey”;

// change this if you want another keystorefile by default
String keystorename = System.getProperty(“keystore”);

if (keystorename == null)
keystorename = System.getProperty(“user.home”)+
System.getProperty(“file.separator”)+
“keystore.ImportKey”; // especially this

// parsing command line input
String keyfile = “”;
String certfile = “”;
if (args.length < 2 || args.length>3) {
System.out.println(“Usage: java comu.ImportKey keyfile certfile [alias]“);
System.exit(0);
} else {
keyfile = args[0];
certfile = args[1];
if (args.length>2)
defaultalias = args[2];
}

try {
// initializing and clearing keystore
KeyStore ks = KeyStore.getInstance(“JKS”, “SUN”);
ks.load( null , keypass.toCharArray());
System.out.println(“Using keystore-file : “+keystorename);
ks.store(new FileOutputStream ( keystorename ),
keypass.toCharArray());
ks.load(new FileInputStream ( keystorename ),
keypass.toCharArray());

// loading Key
InputStream fl = fullStream (keyfile);
byte[] key = new byte[fl.available()];
KeyFactory kf = KeyFactory.getInstance(“RSA”);
fl.read ( key, 0, fl.available() );
fl.close();
PKCS8EncodedKeySpec keysp = new PKCS8EncodedKeySpec ( key );
PrivateKey ff = kf.generatePrivate (keysp);

// loading CertificateChain
CertificateFactory cf = CertificateFactory.getInstance(“X.509″);
InputStream certstream = fullStream (certfile);

Collection c = cf.generateCertificates(certstream) ;
Certificate[] certs = new Certificate[c.toArray().length];

if (c.size() == 1) {
certstream = fullStream (certfile);
System.out.println(“One certificate, no chain.”);
Certificate cert = cf.generateCertificate(certstream) ;
certs[0] = cert;
} else {
System.out.println(“Certificate chain length: “+c.size());
certs = (Certificate[])c.toArray();
}

// storing keystore
ks.setKeyEntry(defaultalias, ff,
keypass.toCharArray(),
certs );
System.out.println (“Key and certificate stored.”);
System.out.println (“Alias:”+defaultalias+” Password:”+keypass);
ks.store(new FileOutputStream ( keystorename ),
keypass.toCharArray());
} catch (Exception ex) {
ex.printStackTrace();
}
}

}// KeyStore

As found at Agent Bob.

Then the last remaining tricky bit is getting it out of the keystore this program creates and in to the one you intend to use. That could be as easy as just pointing jboss at this keystore but I made it harder and did this to copy from one keystore into another.

keytool -importkeystore -srckeystore /the/keystore/made/by/importkey.java/keystore.ImportKey -srcstorepass importkey -destkeystore /the/one/you/use/jboss.keystore -deststorepass somethingclever -alias importkey -destalias production -srckeypass importkey -destkeypass somethingclever

Then I edited /usr/local/java/jboss-VERSION/server/default/deploy/jboss-web.deployer/server.xml to modify the 8443 SSL connector they ship to do this:

    <Connector port="443" protocol="HTTP/1.1" SSLEnabled="true"

               maxThreads="150" scheme="https" secure="true"

        address="${jboss.bind.address}"

                keystoreFile="/usr/local/java/jboss-VERSION/server/default/conf/ark.keystore"

                keyAlias="production"

                keystorePass="somethingclever"

               clientAuth="false" sslProtocol="TLS" />

ETA: the key has its own password separate from the keystore. You can change it when you import that key into the new keystore, so I updated the command to reflect that.

Then I dug in to the documentation for how to use an SSL certificate with jboss. That was more opaque than it needed to be and I’ll probably explain what I did in the next post.

That meant installing Evernote for OS X so I could attach a PDF to a note, then using the information I’d gathered there to write up a proposal concerning alerting. I am trying to be thorough in documenting what I do and why. I tried to capture my thinking, rationalize my decision, and foreshadow future developments. As part of the research for the writing I think I noticed something odd about Pingdom’s pricing.

I’m probably misunderstanding something. But if the costs per check aren’t different at the Business plan level, and I don’t care about SMS notifies, what is my incentive to ever leave the Basic plan? My efficient frontier seems like it’s up and to the left and with a linear progression, it’s a Basic ballgame.

If the cost per check on the business plan is less, then the graph is wrong and there is a break-even point on the expense of checks. But it sure doesn’t look like it.

I ran my proposal for additional monitoring past my boss and got his buy-in and then started deploying it. So that’s my first operational task which is not entirely reactive in nature; there had been an issue earlier with a system going away and no one noticing, but it wasn’t a production system and I was more interested in getting some kind of alerting going for those systems as they come online.

I fully expect to be iterating on the deployed monitoring solution, as it was a trade-off between results and costs (financial and my time/effort/brain) and there are arguably better solutions I didn’t feel like I could invest enough into at this point to get the most out of. It’s a starting point, an incremental improvement over what the company had in place before me.

This wasn’t quite a No Changes Friday (the only religious holiday I observe) but  it seemed worth it to push through that sabbath to get additional awareness of the environment. Arguably it didn’t impact anything production-related, beyond the tiny additional impact of the monitoring checks being done, which are all tickling network listening daemons.

Then I spent the rest of my day researching options for my next proposal, which will be SSL related.