Introduction¶
Try it out / quick start¶
Install Consfigurator (Installation) and ensure that some implementation of ssh-askpass_ is available.
Create a new directory
consfig
somewhere where ASDF will pick it up, such as~/common-lisp/consfig
.Define a Lisp system which represents your configuration.
~/common-lisp/consfig/com.example.consfig.asd:
(asdf:defsystem :com.example.consfig :serial t :depends-on (#:consfigurator #:cl-interpol) :components ((:file "package") (:file "consfig")))
~/common-lisp/consfig/package.lisp:
(in-package :cl-user) ;; this macro is a simple wrapper of DEFPACKAGE which sets up local ;; nicknames for packages providing properties and data sources (consfigurator:defpackage-consfig :com.example.consfig (:use #:cl #:alexandria #:consfigurator))
Define some hosts and deployments.
~/common-lisp/consfig/consfig.lisp:
(in-package :com.example.consfig) (in-consfig "com.example.consfig") (named-readtables:in-readtable :consfigurator) (try-register-data-source :ssh-askpass :iden1-re "^--user-passwd--" :iden2-re "") (defparameter my-substitution "substititions") (defhost athena.example.com (:deploy (:ssh (:sudo :as "spwhitton@athena.example.com") :sbcl)) "Web and file server." (os:debian-stable "buster" :amd64) (apt:mirror "http://my.local.mirror.example.com/") (apt:uses-local-cacher) ; sets up apt-cacher-ng (apt:standard-sources.list) ;; Set key--value pairs in INI-style files. Consfigurator will ;; uncomment existing lines in preference to adding new ones, for ;; readability. (file:contains-ini-settings "/etc/systemd/logind.conf" '("Login" "KillUserProcesses" "no")) (apt:service-installed-running "apache2") ;; Apply some properties as a non-root user. ;; (as "spwhitton" ;; (gnupg:public-key-imported "8DC2 487E 51AB DD90 B5C4 753F 0F56 D055 3B6D 411B")) (file:has-content "/etc/foo" #?{Here is my file content. You can use ${my-substitution} thanks to CL-INTERPOL. And it's multiline. CL-HEREDOC is also available; useful for shell scripts.}) (file:has-content "/etc/bar" '("or" "specify" "a" "list" "of" "lines"")) (file:contains-lines "/etc/some.conf" '("FOO=bar")) ; preserve rest of file contents ;; This will call debootstrap(1) in a way which respects the apt ;; cacher and mirror configured above, so setting up multiple ;; chroots with the same OS will be fast. (chroot:os-bootstrapped. nil "/srv/chroot/test" (os:debian-unstable :amd64) (apt:standard-sources.list) ;; These two properties are not for debootstrap(1) for but apt ;; inside the chroot. (apt:uses-parent-proxy) ; use the apt-cacher-ng set up outside chroot (apt:uses-parent-mirror))) ; use the apt mirror set up above
Here, “spwhitton” is my username on athena; we have to tell Consfigurator what user it will be when it tries to sudo, so it knows whose password it needs. If you have passwordless sudo access configured, you can skip the
:AS
keyword parameter and its argument.Get a Lisp REPL started up –
M-x slime
in Emacs orsbcl
at a shell prompt. Evaluate(asdf:load-system "com.example.consfig")
, then(in-package :com.example.consfig)
(orC-c ~
in Emacs).You should now be able to evaluate
(athena.example.com)
to deploy properties to athena, using the connection chain of SSH, sudo and then handing over to a remote Lisp image.
Other things to try¶
Note that some of these violate some of the ideas of declarative configuration management, because they apply individual properties without updating the definitions of hosts. Sometimes that’s the right thing to do, though, and Consfigurator makes it easy to reuse your property definitions in these non-declarative ways.
Try deploying properties to athena using a different connection type¶
Evaluate something like:
(deploy :ssh athena.example.com)
Apply a security update to all your systems¶
It’s useful to be able to quickly apply a security update across multiple
machines without otherwise interacting with their configuration. Supposing
you have defined a variable *ALL-MY-SERVERS*
which is a list hosts defined
with DEFHOST
, you can evaluate:
(dolist (server *all-my-servers*)
(deploy-these :ssh server
(cmd:single "apt-get update && apt-get upgrade openssl")))
Regex replace a file across hosts¶
With *ALL-MY-SERVERS*
as in the previous example,:
(dolist (server *all-my-servers*)
(deploy-these :ssh server
(file:regex-replace-lines "/etc/baz" #?/foo/ "bar")))
(relies on CL-INTERPOL syntax being enabled, as it is in the example consfig above)
Concepts and terminology¶
We make some simplifications. More precise definitions appear later in this manual.
Host¶
A machine, container, chroot, or similar. Has a plist of static informational host attributes (“hostattrs”), usually including at least a hostname, and a property application specification defining the properties it has.
Property¶
Some configuration which a host can have or lack, and which can be added to a host by running some code, possibly just by applying a series of other properties.
For example: the presence of some lines in a config file; a package being installed or absent; the availability of a website.
Connection¶
A means by which properties can be applied to hosts, and multihop connections to other hosts can be established. There are two types of connections: those which interact with the remote host by means of a POSIX shell, and those which apply properties by executing them in a Lisp image running on the host.
POSIX connections can pass input to and return output from processes, but cannot start asynchronous processes for interaction with your Lisp functions. This is so that POSIX connections can be defined to control hosts for which any kind of shell multiplexing is hard or impossible, such as with serial connections providing only a single interactive POSIX sh. For asynchronous interaction, use a Lisp connection.
Deployment¶
The combination of a connection and a host. Executing a connection deploys
all of a host’s usual properties to that host by means of the given
connection. To deploy just a few particular properties, you can use
DEPLOY-THESE
.
A deployment is itself a property. This is one way in which connections can be nested: one remote host can be used to deploy others, as a controller.
Root Lisp¶
The Lisp image you control directly when you execute deployments. Typically
running on your development laptop/workstation (and not as the root
user).
Property application specification (“propspec”)¶
A specification, in Consfigurator’s DSL, for the properties that a host has and/or lacks. For example,:
(eseqprops (apt:installed postfix)
(etc-default:set "locale" "LANG" "en_GB.UTF-8")
(unapply (com.example.consfig.services:mail-satellite)))
Property application specifications are applied in order, so properties later in the list usually implicitly depend on properties earlier in the list, though some property combinators can change this.
Unevaluated property application specification (“unevaluated propspec”)¶
A property application specification, except in atomic property applications
of the form (PROPERTY . ARGS)
, ARGS
are expressions to be evaluated to
produce the arguments to pass to PROPERTY
, rather than those arguments
themselves. An unevaluated property application specification can be
converted into a property application specification by evaluating each of
ARGS
.
The main places you will find an unevaluated property application
specification is in calls to DEFHOST
, DEPLOY
, DEPLOY-THESE
and
DEFPROPLIST
. Theses macros converts an unevaluated property application
specification into code which will produce the corresponding property
application specification.
Prerequisite data¶
File contents required to apply a property which should be generated or
extracted, by the root Lisp, at the time of deployment: a tarball containing
the latest version of the web service to be deployed; a secret extracted from
an encrypted store; a git bundle from localhost which the target host cannot
just git clone
to itself.
Prerequisite data is versioned. To replace a secret key, for example, you change the data and bump the version. If there is no version bump, Consfigurator will assume connections can re-use old copies of prerequisite data; this avoids uploading the same data over and over again.
In addition to secrets management, prerequisite data is Consfigurator’s
mechanism for the common need to upload files to controlled hosts. The same
mechanism is used internally to upload the Lisp code needed to start up remote
Lisp images for :lisp
connections.
Consfig¶
An ASDF system in which you define your hosts and initialise sources of
prerequisite data. This system might also define some site-specific
properties, default deployments, and helper functions. Typically the system
is named COM.EXAMPLE.CONSFIG
where example.com
is your primary domain
name.
The system can contain multiple packages, perhaps to divide up your
definitions of hosts and default deployments from your site-specific
properties (e.g. you might have a package called
COM.EXAMPLE.CONSFIG.SITES
).
You can have multiple independent Consfigs loaded into the root Lisp at once,
but if you do, then you should avoid using the *CONSFIG*
global variable.
Documentation conventions¶
All unqualified names of Lisp symbols refer to those exported from the
CONSFIGURATOR
package, because it is assumed that this package is imported
unqualified into both user consfigs and Lisp packages providing properties,
connection types and sources of prerequisite data.
FOO.BAR:BAZ
means a symbol BAZ
defined in
CONSFIGURATOR.PROPERTY.FOO.BAR
, except that DATA.FOO:BAR
means a
symbol BAR
defined in CONSFIGURATOR.PROPERTY.DATA.FOO
. These are the
recommended package nicknaming schemes for use in consfigs, e.g.:
(defpackage :com.example.consfig
(:use #:cl #:consfigurator)
(:local-nicknames (#:file #:consfigurator.property.file)
(#:cmd #:consfigurator.property.cmd)
(#:data.pgp #:consfigurator.data.pgp)))
You can use the DEFPACKAGE-CONSFIG
macro to set up all these local
nicknames.
Portability and stability¶
Consfigurator is still stabilising and so there may be lots of breaking changes.
All of the code in the core library should be portable ANSI Common Lisp, though optional packages providing properties and connection types might use implementation-specific functionality. There is one exception: we require an implementation of
MACROEXPAND-ALL
, but most Lisps in use today provide this. Little to no testing is done by the author on implementations other than SBCL, so testing and portability patches are welcome.Little attempt is made by the author to support systems other than Debian GNU/Linux, but again, portability patches are welcome, and the design of Consfigurator should enable supporting other systems.
Credits¶
Many of the good ideas here come straight from Joey Hess’s Propellor_. I’m working on Consfigurator because I think Propellor is great, but wanted to add Consfigurator’s POSIX-type connections and arbitrary connection nesting, and I wanted to implement that in Lisp (Propellor only supports something equivalent to a single, unnested Lisp-type connection). Additionally, after five years of using and extending Propellor, I’ve come to disagree with Joey about whether Haskell’s type system helps or hinders using and extending Propellor.