Puppi is a Puppet module that lets users manage and automate the deployment of applications or generally every kind of batch activity that involves the execution of a sequence of scripts and commands.
Its structure provides complete flexibility on the actions required for virtually any kind of application deployment but, in order to make usable out of the box, some example defines and scripts are provided to manage common scenarios and needs.
The module provides:
- The puppi command and its whole working environment
- A set of common usage scripts that can be used in chain
- Sample defines that can be used for many common application deployment scenarios.

The whole picture in an example

Before diving into details let's see a brief example.
On your node or role you can use a similar define:

puppi::project::war { "myapp":
    source       => "http://repository.example42.com/myapp/myapp.war",
    user         => "myappuser",
    deploy_root  => "/srv/tomcat/myapp/webapps",
    report_email => "[email protected]",
    enable       => "true",
}

This creates a set of scripts in /etc/puppi/projects/myapp that make it possibile to write the simple command:

puppi deploy myapp

to run the whole deployment procedure, that, in this case, involves retrieving the war file from the http://repository.example42.com/myapp/myapp.war, backing up /srv/tomcat/myapp/webapps, deploying the new war in the same directory as myappuser and notifying via email [email protected].

This case is not particularly complex but consider that the same puppi::project::war define user before has many more options, that let you run custom pre or post commands (in customizable order and with defineable user), stop and start, during the process any custom service, block access from a specific IP (such as the one of your load balancer) and so on.

If something wrong happens you can simply type:

puppi rollback myapp

and choose to what version you want to rollback (latest or any other previous (dated) backup).

Let's see a real life example of a deploy:

root@metaportali-mpc:~# puppi deploy configurator
Puppi setup: 00-configurator-RuntimeConfig-Initialization  [  OK  ]

Deploy: 10-configurator-Run_PRE-Checks                     [  OK  ]
PROCS OK: 4 processes with command name 'apache2'
TCP OK - 0.001 second response time on port 80|time=0.000651s;;;0.000000;10.000000
[ ... ]
Deploy: 20-configurator-Retrieve_WAR                       [  OK  ]

Deploy: 30-configurator-Backup_existing_WAR                [  OK  ]

Deploy: 36-configurator-Disable_extra_services             [  OK  ]

Deploy: 37-configurator-Check_undeploy                     [  OK  ]

Deploy: 38-configurator-Service_stop                       [  OK  ]
 * Stopping tomcat-mpc instance
Using CATALINA_BASE:   /store/tomcat/mpc
[ ... ]
Deploy: 39-configurator-Run_Custom_PreDeploy_Script        [  OK  ]

Deploy: 40-configurator-Deploy_WAR                         [  OK  ]

Deploy: 42-configurator-Service_start                      [  OK  ]
 * Starting tomcat-mpc instance
Using CATALINA_BASE:   /store/tomcat/mpc
[ ... ]
Deploy: 43-configurator-Check_deploy                       [  OK  ]

Deploy: 44-configurator-Enable_extra_services              [  OK  ]

Deploy: 80-configurator-Run_POST-Checks                    [  OK  ]
PROCS OK: 4 processes with command name 'apache2'
TCP OK - 0.004 second response time on port 80|time=0.003824s;;;0.000000;10.000000
[ ... ]

Reporting: 20-configurator-Mail_Notification               [  OK  ]

REPORT FOR PUPPI - STATUS OK
Summary of operations is: /var/log/puppi/configurator/20110224-114729/summary 
Details are in: /var/log/puppi/configurator/20110224-114729/
Temporary workdir has been: /tmp/puppi/configurator/ (Will be rewritten at the next puppi run)
Runtime config file is: /tmp/puppi/configurator/config
Files have been archived in: /var/lib/puppi/archive/configurator/20110224-114729

If you use Example42 modules, enable their automatic monitoring features ($monitor=yes) and use Puppi, among others, as monitoring tool (ie: $monitor_tool=["nagios","munin","puppi"] ) you have out of the box the possibility of running local checks of the applications managed by Puppet with the command puppi check. Let's see an example:

root@metaportali-mpc:/# puppi check
Host check: 50-apache_process                              [  OK  ]
PROCS OK: 4 processes with command name 'apache2'

Host check: 50-apache_tcp_80                               [  OK  ]
TCP OK - 0.001 second response time on port 80|time=0.000668s;;;0.000000;10.000000

Host check: 50-mcollective_process                         [  OK  ]
PROCS OK: 1 process with command name 'ruby', args 'mcollectived'

Host check: 50-munin_process                               [  OK  ]
PROCS OK: 1 process with command name 'munin-node'

Host check: 50-munin_tcp_4949                              [  OK  ]
TCP OK - 0.003 second response time on port 4949|time=0.003184s;;;0.000000;10.000000

Host check: 50-nrpe_process                                [  OK  ]
PROCS OK: 1 process with command name 'nrpe'

Host check: 50-nrpe_tcp_5666                               [  OK  ]
TCP OK - 0.005 second response time on port 5666|time=0.005355s;;;0.000000;10.000000

Host check: 50-openssh_process                             [  OK  ]
PROCS OK: 13 processes with command name 'sshd'

Host check: 50-openssh_tcp_22                              [  OK  ]
TCP OK - 0.001 second response time on port 22|time=0.000857s;;;0.000000;10.000000

Host check: 50-puppet_process                              [  OK  ]
PROCS OK: 1 process with command name 'puppetd'

Host check: 50-rsyslog_process                             [  OK  ]
PROCS OK: 1 process with command name 'rsyslogd'

Host check: 50-snmpd_process                               [  OK  ]
PROCS OK: 1 process with command name 'snmpd'

Host check: 50-tomcat-mpc                                  [  OK  ]
PROCS OK: 2 processes with command name 'java', args 'mpc'

Host check: 50-tomcat-verticali                            [  OK  ]
PROCS OK: 1 process with command name 'java', args 'verticali'

Host check: 50-tomcat_tcp_8200                             [  OK  ]
TCP OK - 0.001 second response time on port 8200|time=0.000697s;;;0.000000;10.000000

Host check: 50-tomcat_tcp_8201                             [  OK  ]
TCP OK - 0.001 second response time on port 8201|time=0.001045s;;;0.000000;10.000000

From these simple examples it should be clear that with Puppi you have:
- A common command to manage deploys and rollbacks on any host.
- A simple to use, coherent and still flexible and customizable set of defines to describe in Puppet manifests all the elements you need for your application deployments.
- A quick way to check if everything is ok on your local node (if you use Example42 module set).

Think about it: with just a Puppet define you build the whole deploy logic
- Reporting for each deploy/rollback is built-in and extensible
- Automatic checks can be built in the deploy procedure
- You have a common, growing, set of general-use scripts for typical actions
- You can define custom deployments workflows (puppi::project::myprocedure) for specific cases not covered by the provided sample defines.

Using the puppi command

Puppi provides various actions and options, let's see the main ones:

puppi <action> <project_name> [ -options ]
These are, currently, the available actions:
- puppi init <project_name> : First time initialization of the defined project
- puppi deploy <project_name> : Deploys the defined project
- puppi rollback <project_name> : Rollback to a previous deploy state
- puppi check [project_name] : Runs project specific and host wide checks
For each of these actions puppi runs the commands in /etc/puppi/projects/$project_name/$action, logs their status and then runs the commands in /etc/puppi/projects/$project_name/report to provide reporting, in whatever, pluggable, way.
You can also provide some options:
-f : Force puppi commands execution also on CRITICAL errors
-i : Interactively ask confirmation for every command
-t : Test mode. Just show the commands that should be executed without doing anything
-d <yes|full>: Debug mode. Show debugging info during execution
-o "parameter=value parameter2=value2" : Set manual options to override defaults. The options must be in parameter=value syntax, separated by spaces and inside double quotes.

Here are some command line samples:

puppi check : Run host-wide checks.
puppi init myapp : Make the first deploy of "myapp". Can be optional.
puppi deploy myapp : Deploys myapp with the standard logic/parameters defined in Puppet
puppi deploy myapp -f : Deploys myapp and doesn't stop in case of Critical errors
puppi deploy myapp -i : Deploys myapp in interactive mode. Confirmation is asked for each step
puppi deploy myapp -t : Test mode. Just show the commands that would be executed
puppi deploy myapp -d full : Deploys myapp with full debugging output
puppi deploy myapp -i -o "version=1.1 source_url=http://dev.example42.com/code/my_app/": Deploys myapp in interactive mode and sets some custom options that override the standard Puppet params.
Note that these parameters change according to the script you use (and the scripts must honour this override).
puppi rollback myapp : Rollbacks myapp to a previous archived state. User is prompted to choose which deploy to override.

File Paths

All the paths used by the puppi scripts are provided, and can be configured, in the puppi Puppet module:
/usr/sbin/puppi - Where the puppi command is placed. Currently is a bash script.
/etc/puppi/puppi.conf - Puppi main config file. Here various puppi wide paths are defined
/etc/puppi/checks/ ($checksdir) - Here can be placed all the host wide checks. If you use the Example42 monitor module and have "puppi" as $monitor_tool, this directory is automatically filled with checks based on Nagios plugins.
/etc/puppi/projects/ ($projectsdir) - In this directory you can have one or more projects subdirs, with the commands to be run for deploy, rollback and check actions. They are completely built (and purged) by the Puppet module.
/etc/puppi/scripts/ ($scriptsdir) - The general-use scripts directory, these are used by the above commands and may require one or more arguments.
/var/lib/puppi/archive/ ($archivedir) - Where all data to rollback is placed.
/var/log/puppi/ ($logdir) - Where logs and reports of the different commands are placed.
/tmp/puppi/ ($workdir) - Temporary, scratchable, directory where Puppi places temporary files. It's wiped at every puppi run.
/tmp/puppi/$project/config - A runtime configuration file, where are dinamically placed variables usable by all the scripts invoked by puppi. This is necessary to mantain "state" information that changes on every puppi run (such as the deploy datetime, used for backups).

Understanding the Puppi module

The puppi module provides few basic defines to manage puppi's setup elements and some example templates that use these defines to build deployment workflows.
The basic defines are:
puppi::project - Creates the main project structure. One or more different deployment projects can exist on a node.
puppi::initialize - Creates a single command to be placed in the init sequence. It's not required for every project.
puppi::deploy - Creates a single command to be placed in the deploy sequence. More than one is generally needed for each project.
puppi::rollback - Creates a single command to be placed in the rollback sequence. More than one is generally needed for each project.
puppi::check - Creates a single check (based on Nagios plugins) for a project or for the whole host (host wide checks are auto generated by Example42 monitor module)
puppi::report - Creates a reporting command to be placed in the report sequence.

The above init, deploy, rollback, check and report defines have generally a standard structure and similar arguments. Every one is reversable (enable => false) but you can wipe out the whole /etc/puppi directory to have it rebuilt from scratch.
Here is an example for a deploy command:

puppi::deploy { "Retrieve files":       # The $name of the define
    command  => "get_curl.sh",          # General-use script to use 
    argument => "file:///storage/file", # Argument(s) for the script
    priority => "10",                   # Execution order
    user     => "root",                 # Execution user
    project  => "myapp",                # The name of the project
}

This define creates a file named /etc/puppi/projects/${project}/deploy/${priority}-${name}
Its content is, simply:
su - ${user} -c "export project=${project} && /etc/puppi/scripts/${command} ${arguments}"

You can glue together, with the desired order according to the priority argument, different basic defines to create a complex project template and use this in your manifests. Various sample defines that can be used to many common deployment scenarios are present in puppi/manifests/project/ You can use puppi::project::war to manage the deployment of a simple war file, as seen before, or other defines to manage deployments of a tarball, an arbitrary list of files, or the maven artifacts published on a Nexus repository.

The puppi/files/scripts directory in the module contains some general usage scripts that can be used in custom deployments.
They are generally made to work together according to a specific logic, which is at the base of the sample defines in puppi/manifests/project/ but you're free to write your own scripts, in whatever language, according to your needs, and integrate them with custom Puppet defines.

The default scripts are engineered to follow this procedure for a deployment:
- Remote files are downloaded in /tmp/puppi/$project/store or directly in the predeploy directory: /tmp/puppi/$project/deploy
- If  necessary the downloaded files are expanded in one or more predeploy directories (default:/tmp/puppi/$project/deploy)- Runtime configuration entries might be saved in /tmp/puppi/$project/config- Files are eventually backed from the deploy directory (Apache' docroot, Tomcat webapps or whatever) to the archive directory (/var/lib/puppi/archive/$project)
- Files are copied from the predeploy directory to the deploy dir.
- Relevant services are eventually sopped and started

The most used common scripts are (they might have different arguments, some of them are quite simple):
get_file.sh - Retrieves a remove file via ssh/http/rsync/svn and places it in a temporary directory (store or predeploy)
deploy.sh - Copies the files in the predeploy dir to deploy dir
archive.sh - Backups and restores files in deploy dir
service.sh - Stops or starts one or more services
wait.sh - Waits for the presence or absence of a file, for the presence of a string in a file or a defined number or seconds.
get_metadata.sh - Extracts metadata from various sources in order to provide info to other scripts
report_mail.sh - Sends a mail with the report of the operations done

How to customize

It should be clear that with puppi you have full flexibility in the definition of a deployment procedure, since the puppi command is basically a wrapper that executes arbitrary scripts with a given sequence, in pure KISS logic.

There are different parts where you can customize the behaviour of puppi:

- The set of general-use scripts in /etc/puppi/scripts/ ( this dir is filled with the content of puppi/files/scripts/ ) can/should be enhanced. As been, these can be arbitrary scripts in whatever language. If you want to follow puppi's logic, though, consider that they should import the common and runtime configuration files and have an exit code logic similar to the one of Nagios plugins: 0 is OK, 1 is WARNING, 2 is CRITICAL.
Note that by default a script that exits with WARNING doesn't block the deploy procedure, on the other hand, if a script exits with CRITICAL (exit 2) by default it blocks the procedure.
Take a second, also, to explore the runtime config file created by the puppi command that contains variables that can be set and used by the scripts invoked by puppi.

- The custom project defines that describe deployments procedures. These are placed in puppi/manifests/project/ and can request all the arguments you want to feed your scripts with.
Generally is a good idea to design a standard enough template that can be used for all the cases where the deployment procedure involves similar steps. Consider also that you can handle exceptions with variables (see the $loadbalancer_ip usage in puppi/manifests/project/maven.pp)

What's next

The Puppi module is a work in progress that is already used in production. It has been designed to automate deployments in a specific environment but a lot of effort has been done to standardize as much as possible the procedures and the options for differents needs and logic and to make it modular by design.
The same puppi command can be re-written from scratch. It's has been made in bash and not in fancier languanges for simplicity and portability.
What it does at the moment, after all, is something quite simple (basically execute commands in a sequence).
The whole point of the Puppi module is to define quickly and standardize in Puppet manifests any kind of deployment procedure (at least, at the moment, any kind of procedure that can be executed from the same node, orchestration is planned but not yet functional).

Future plans involve:
- dedicated agents for orchestration tools like Mcollective or ControlTier
- more reporting scripts (for example reporting on Google Calendar and Docs)
- a web interface to manage the deployment procedure and see reports