Snipet: Using Puppet to manage preexisting servers

Using Puppet to manage preexisting servers

Source: Puppet user list

From 

Luke Bigum Luke.Bigum@lmax.com

On 16/05/12 13:07, Mister IT Guru wrote:

Good Afternoon Puppet List,

I'm investigating the possibility of using puppet to take over an already existing cluster of servers. Ordinarily this doesn't sound like anything out the ordinary, but I've had my hands tied by the cluster owner saying he doesn't want anything to break/fail while I'm wrestling the cluster to be under my control.

Ordinarily, I'd say lets upgrade your working practices while we're at it, and properly manage your whole workflow, manage access and manage who can make changes. Lets start by rebuilding your boxes to specification, and migrating your existing data over. Sounds great, "Client says no" huh? - Now I need to rethink my strategy.

I was planning on using my current manifests, but I think I'll end up over writing some config file that had a critical setting in it, or change a password somewhere. Usually, I'm not too worried about that, but it did occur to me that I don't have any real sanity checks in my modules.

       • Are there any recommended ways of using puppet to take over a production environment?

I've done this a lot. Unfortunately the only way you'll be absolutely sure that Puppet isn't breaking something is to look at the log/report of each host. You can do breadth first: get Puppet everywhere just managing itself, then slowly add managed resources to Puppet. Or you can do depth first: get Puppet managing everything on one machine, prove to your cluster manager that this one machine still works, then teach other people how to convert a cluster node to Puppet and check the logs (might be faster in your case).

       • How can I ensure that when I'm spreading my manifests across the globe, I mean cluster, that any changes can be reverted?

Not easily... Puppet has no inbuilt 'revert' functionality and how do you know what state everything is in to start with? You could use Ralsh to query the state of all the resources you want to manage before you apply any changes. The log of a run will show a resource's previous state and the state Puppet changed it to as well. Also be aware that the log is dependent on how the provider reports what it's done. Most of the native Puppet resource types are ok (Service changed 'stopped' to 'running') but other native types (I can't think of any off hand) could simply say "Changed 'absent' to 'present'" and that could be a bit ambiguous depending on the provider's definition of "absent".

               • I understand that every file puppet changes, gets backed up? Where? (embarrassed he doesn't know!)

Look at the filebucket feature, generally stored on the Puppet Master in /var/lib/puppet/bucket: http://docs.puppetlabs.com/references/2.7.0/type.html#filebucket

       • Finally, can I force puppet to replace changed files, or is this a manual process?

I'm not sure what you mean by changed files. If you have Puppet manage a file's contents with the File type, it will continually put in that file what it thinks should be in there. Other providers can make inline edits of files, like Augeas or PuppetLab's Parsedfile.

------------

Peter Bukowinski pmbuko@gmail.com My first experience with puppet was puppetizing an already deployed 500-node HPC cluster. The cluster admin was kind enough to pull a node out of the cluster so I could test my manifest/modules on it without affecting production nodes. Once I was confident I had it dialed in, I converted while racks at a time until all nodes were under puppet's watchful eye. If this is not possible for you, you could use VMware or Virtualbox and clone a server to a VM during a maintenance window and use that for your development. Snapshots are very nice to have when testing your manifests as they allow rapid and complete undos. As for reverting changes on non-virtual servers, you can use puppet's filebucket -- it is well documented. I, however, am a great fan of having regular, non-puppet backups from which to restore original files. ------------ Aaron Grewell aaron.grewell@gmail.com On Wed, May 16, 2012 at 5:07 AM, Mister IT Guru <misteritguru@gmx.com> wrote: > Ordinarily, I'd say lets upgrade your working practices while we're at it, and properly manage your whole workflow, manage access and manage who can make changes. Lets start by rebuilding your boxes to specification, and migrating your existing data over. Sounds great, "Client says no" huh? - Now I need to rethink my strategy. One of the tools that's helpful in this situation is Puppet's no-op mode.  This gives you the ability to report in advance on what would change during a Puppet run.  It's helpful for reassuring angsty customers as well as making a list of what might need to be reverted in an emergency. ------- Brian Gupta brian.gupta@brandorr.com On Wed, May 16, 2012 at 12:08 PM, Aaron Grewell <aaron.grewell@gmail.com> wrote: On Wed, May 16, 2012 at 5:07 AM, Mister IT Guru <misteritguru@gmx.com> wrote: > Ordinarily, I'd say lets upgrade your working practices while we're at it, and properly manage your whole workflow, manage access and manage who can make changes. Lets start by rebuilding your boxes to specification, and migrating your existing data over. Sounds great, "Client says no" huh? - Now I need to rethink my strategy. One of the tools that's helpful in this situation is Puppet's no-op mode.  This gives you the ability to report in advance on what would change during a Puppet run.  It's helpful for reassuring angsty customers as well as making a list of what might need to be reverted in an emergency.  I concur to the advice all posters have given. I'll add that the three tools we've used to do stuff like this are: 1) puppet resource 2) blueprint http://pratikamin.wordpress.com/2011/03/22/generate-puppet-recipe-from-running-system/ (These recipes that get generated really need hand tuning, as they are massive.) 3) puppet in --noop mode to validate. Generally we start with the manage nothing and go through and add resources and services one at a time. It's slow and tedious, and we don't like to do it, but you sometimes have to. Cheers, ------------------------