Deployment pitfalls
This is part 1 of a 2 part series, Part two is here.
Except for very simple software, most code deployments of any substance relies on a number of dependencies, both at the Operating System level and in the form of application libraries and support packages. The software deployment process has often been done by a single person in small companies and two or more people in more advanced business environments with a developer and the system administrator working hand in hand to deploy applications in critical production environments.
Deployments by hand or with scripts manually run often result in failure when all the dependencies are not documented on the part of the developer or the instal process is done by hand and leaves out components pre-installed in the development environment and assumed to be live in a production environment. Common issues include packages and tools with mismatched versions and settings not documented or overwritten by the development environment settings resulting in a down or unstable production environment. This of course is just a small sample and horror stories of failed deployments can be many and varied from business to business.
Mismatches between Development and Production environments due to hand tweeking of settings and parameters is VERY common, the main issue for support staff is little to no documentation or scripted process in place to ensure repeatability. Repeatability is vital, especially in the event of failure and a new environment needs to be built quickly. In the past, many businesses allowed poor deployment processes to continue and put at risk their chance of both recovering from disaster and the reliable operation of their production environments due to a lack of a suitable deployment and config management processes and tool sets.
One of the least effective tools to use for deployments into production environments are version control systems (VCS). Perfect for development environments to control code, VCS tools do not generally instal dependency packages, check OS dependencies and build environments outside of the code added into the VCS. So in short a version control tool should remain in a development environment and only supply a deployable package for installation by another tool. Enter SaltStack aka Salt!
Why use SaltStack?
Without going into a long sales pitch on SaltStack, which goes something like "SaltStack – Fast, scalable and flexible systems management software for data center automation, cloud orchestration, server provisioning, configuration management and more!" we will briefly touch on some key aspects.
The basic process of crafting an environment, copying files, installing packages and adding/changing or updating a remote hosts configuration is easy to script in SaltStack. Its also easy to apply and repeat as needed on any number of host servers. This includes package management across different Operating System environments like Ubuntu, Red Hat, Fedora, CentOS or a host of others (even Windows Servers) as SaltStack has built in modules that work automatically in the target environment.
For the developer, crafting a SaltStack script to create the environment needed to deploy their application package is quick and easy and substantially smaller in complexity to other tools like Chef, Puppet or Vagrant. Its often been quoted that a 10,000 line Chef script can be done in less than 1,000 lines using SaltStack. Having said that, you would be lucky to write scripts past 100 lines in most cases, and basic tasks are done in single lines with ease!
As a result of this, SaltStack has been gaining popularity in leaps and bounds, with many supporters who were previously using other tools now exclusively using SaltStack for all tasks. One of our most common tasks here at Conetix is fixing third party application vulnerabilities that occur almost weekly in a host of common applications .
The key thing with using SaltStack is the ability for an administrator to deploy scripts repeatedly to an environment as needed knowing the environment will be in the right "state". the word "state" is important here as Configuration Management and Code Deployment puts the target into a set state. That includes file ownership and permissions as well as directories and file contents. If the state of a file changes due to tweeking by an individual, re-running the SaltScript will put the affected files back into the required state and highlight what has changed. Therefore SaltScripts are in fact "state files" not "scripts" by definition. Here is an example of a simple file copy from the SaltStack Server to a target host (known as a "Minion").
install-cleanup-script: file: - managed - source: salt://cleanup-directories.sh - name: /usr/local/bin/cleanup-directories.sh - user: root - group: root - mode: 754
This copies the file "cleanup-directories.sh" to /usr/local/bin if its not there already or has been manually edited and sets the ownership and permissions accordingly. The end result is a working script in a known location and in a known "state".
Another simple example includes creating a directory:
/var/www/vhost/production.website.com: file.directory: - user: apache - group: webserver - mode: 755
The obvious benefit of this is when you re-run it on 10 minions (target servers), all ten will have the directory and ownership permissions set correctly – ALWAYS.
A more detailed example showing how to create a series of databases and users to access them using variables and "for" loops:
{% set DATABASE_PASS = 'our-db-password' %}
{% for DB in ['keystone','nova','cinder','glance','heat','neutron'] %}
{{ DB }}:
mysql_database:
- present
mysql_grants.present:
- grant: all privileges
- database: {{ DB }}.*
- user: {{ DB }}
- host: localhost
mysql_user.present:
- host: localhost
- password: {{ DATABASE_PASS }}
- connection_charset: utf8
{% endfor %}
SaltStack supports a wide range of common environments and server applications including common databases, Amazon Web Services, nginx, Apache and a whole lot more.
A more complex Deployment Example
Showing simple snippets of Salt code (its actually YAML code used by Salt) is fine but a typical deployment example might show the power a bit better. The following script deploys a package called grafana and since the environment we use could be Ubuntu or CentOS it has variables it sets accordingly. In the process of deploying it, we include a number of other scripts with "standard" functions like production directories we have on our prod server, prod packages we expect to have, cron scripts to keep the system cleaned up at night (remove old session files, cleanup /tmp etc). Then we download our application packages, create the required directories for it instal the code and restart the web server. The script determines the OS environment (CentOS or Ubuntu) and sets permissions as needed using Salt "grains", a simple concept once you understand it.
#============================================================
# determine OS type
{% if grains['os'] == 'CentOS' %}
{% set apache = 'apache' %}
{% set webserver = 'httpd' %}
{% elif grains['os'] == 'Ubuntu' %}
{% set apache = 'www-data' %}
{% set webserver = 'apache2' %}
{% endif %}
include:
- .webserverpackages
- .webserverdirs
- .webservercrontasks
git:
pkg.installed
# Node Package Manager
npm:
pkg.installed
# Grunt - JS build system
grunt-cli:
npm:
- installed
- require:
- pkg: npm
#
# Make Directory
#
/opt/graphite/webapp/grafana:
file.directory:
- user: {{ apache }}
- group: {{ apache }}
- mode: 775
- makedirs: true
#
# Download to /tmp
#
https://github.com/grafana/grafana.git:
git.latest:
- rev: master
- target: /tmp/grafana
#
# cleanup old code if present
#
rm -rf /opt/graphite/webapp/grafana/*:
cmd.run
#
# Use grunt to compile the CSS etc
#
cmd.run:
- name: grunt
- cwd: /tmp/grafana
- watch:
- git: https://github.com/grafana/grafana.git
#
# place it where we want it to run from
#
cp -R /tmp/grafana/src/* /opt/graphite/webapp/grafana/:
cmd.run
#
# Custom JS file for production use
#
/opt/graphite/webapp/grafana/config.js:
file.managed:
- source: salt://cnx_servers/grafana/config.js
#
# Setup perms for the right web server and restart services for the OS type
#
chown -R {{ apache }}:{{ apache }} /opt/graphite/webapp/grafana/:
cmd.run
{% if grains['os'] == 'CentOS' %}
/etc/httpd/conf.d/grafana.conf:
file.managed:
- source: salt://cnx_servers/grafana/grafana.conf
service httpd restart:
cmd.run
{% elif grains['os'] == 'Ubuntu' %}
/etc/init.d/apache2 restart:
cmd.run
{% endif %}
The key items to draw from this is your Version Control software is not geared to do server management and deployment. Its designed to hold and version control source code specifically, apart from continuous build features which are related to source code development any extra features outside of that are best avoided. Also, System Administrators dont use version control tools to manage servers.
Regardless of your environment, Linux or Windows, deploying code, support packages and crafting an environment for them to run in is best done with a dedicated Configuration and Management tool like SaltStack. SaltStack modules are also available to support AWS instances and Docker container instances. A complete list of modules is here.
Managing Docker Containers
SaltStack is constantly evolving and so is its native Docker support. With SaltStack you can pull Docker images from Docker Hub, create containers, start/stop containers, watch for changes on an image and remove the container. There are a host of other commands but these will feature in a later article.
SaltStack Use Cases
At Conetix we use SaltStack for all software deployment, package management and general tasks or all servers. Some usage scenarios include:
- When an exploit hits we develop a package to fix it and watch for it. We then write a Salt script to target the exploit and use SaltStack to push the script, CRON job and monitoring solution to all applicable servers and container environments sequentially. Its stress free and rock solid to use.
- When we deploy a new container environment we use SaltStack to complete the setup and ensure all the basic configuration is in place, we can re-run this at any stage without fear.
- As important software updates need to be applied we use the SaltStack master to initiate package upgrades using the built in package manager, on the target this will use the "yum", "apt-get" or native package manager to download and deploy the package.
- When new software repositories become available we can craft a Salt script to instal and configure the repo for the host. This is handy when new versions of PHP become available.
- Using "Grains" we can define a host to be a certain type and selectively deploy using that type as an identifier. Very handy if you want to target an OS or if a specific application is installed (like PHP5.5)
Lessons Learnt
- Without SaltStack time is wasted logging into hosts to then copy scripts and manually run them.
- The chance of manually making changes to one environment that then creates an unknown dependency not present in another environment is high without a tool like SaltStack, so without it we introduce risks down the track.
- Allowing users to log into production systems risks changes being made that are not documented, its even worse on their own development environments where changes are rampant and done with little to no thought as to the maintainability in the long term.
- Salt scripts put the relevant target in a set known state and can be repeated as needed anytime. If a change occurs due to normal application running then this needs to be factored in the deployment process and NOT scripted.
- Configuration data MUST NOT be kept in Version Control Systems or databases, it should be defined by SaltStack and pushed to the relevant environment using grains in the script to ensure the right environment is present and crafted for. This prevents development settings being set in a production environment, think bulk emailing clients wrong data and invoicing clients wrong or repeated billing data when production settings are configured in a development environment.
Where to Now?
- Go to the Saltstack documentation site and review the tutorials on the modules.
- Review some online Tutorials like SaltStack – Getting started with SaltStack – Part 1
Part 2 – https://www.conetix.com.au/blog/software-deployment-saltstack-part-2
