README-Installation.txt

This is a system (Robin Powell) have been using to containerize
services/projects/systems.  The overall thing that you're managing
all goes in one directory and is called a "bundle" in this document.
Bundles have one or more containers, and containers have zero or
more addons.  Containers and addons both run as systemd user
services.

An example of a bundle is your instance of mediawiki, which might
have a web container for nginx and a db container for mariadb.

If you're building a new bundle using these scripts, this is *not*
plug-and-play; you really want to read this entire document.  If
you're just running a bundle that uses them, you should be fine
with README-Basic-Usage.txt

Requirements
============

- ruby 2.5+ (specifically for ERB#result_with_hash)
- bash
- podman (or docker, if you must; there's a section on that below)

Intro
=====

The general idea is that each bundle gets its own user. That user
runs containers for the various parts of the bundle (db, web,
whatever). The source trees and data directories for each container
are all under the user's homedir, and are mounted into the
containers.

I run all of this with rootless podman, so these users have no
special permissions, and unlike with Docker the fact that they can
run containers with directory mounts is not fundamentally equivalent
to giving them root.  Having said that, you can do all this with
Docker, too.  There's not really any secret magic here, it's more a
standard for laying out the bundles, plus a few scripts.

It is expected that the entire bundle directory will be in source
control (i.e. git).  I personally simply move directories around in
the actual bundle's code source tree to match the layout below.

General Layout
--------------

In this documentation, "main directory" is the place that has the
files you are symlinking to items in this repo.  You should have all
your bundles' source and data under the main directory.  Typically
the main directory is in the homedir of a dedicated serivice user;
typically it is the only non-trivial directory.

This is a subdirectory under the homedir because you're typically
going to want this to be a git repo, and having your *homedir* be a
git repo means having to ignore a *bunch* of stuff.

Example layout:

~user/
    .config/systemd/
            web.service         <-- autogenerated by setup.sh on every run
            database.service    <-- autogenerated by setup.sh on every run
    mybundle/                                     <-- this is what we're calling the "main directory"
        setup.sh -> /opt/lbcs/setup.sh                                  <-- start with this
        README-Basic-Usage.txt -> /opt/lbcs/README-Basic-Usage.txt      <-- link made by setup.sh on first run
        build_image.sh -> /opt/lbcs/build_image.sh                      <-- link made by setup.sh on first run
        destroy_container.sh -> /opt/lbcs/destroy_container.sh          <-- link made by setup.sh on first run
        initial_setup.sh -> /opt/lbcs/initial_setup.sh                  <-- link made by setup.sh on first run
        run_container.sh -> /opt/lbcs/run_container.sh                  <-- link made by setup.sh on first run
        .gitignore
        LICENSE
        README.txt
        ...
        secrets   <-- avaliable in template expansion
        config    <-- avaliable in template expansion
        cron/
            run_backups.sh
            crontab.erb
            cron-run-inside.sh -> /opt/lbcs/cron/cron-run-inside.sh     <-- link made by setup.sh on first run
        misc/
            web-httpd.conf
            secrets.conf.erb
        containers/
            web/
                Dockerfile.erb
                config    <-- avaliable in template expansion
                secrets   <-- avaliable in template expansion
                src/
                    index.py
                    images/...
                addons/
                    bots/
                        config <-- used to generate systemd service files that run inside the containing container
            database/
                Dockerfile.erb
                config    <-- avaliable in template expansion
                secrets   <-- avaliable in template expansion
                data/
                    postgresql.conf
                    pg_wal/...
                    backups/
                        db_backup.20201008.psql.tgz

The ./build_image.sh script runs "podman build" in the context of
the main directory, so you can use any file under the main directory
in COPY clauses or whatever.

If you're using docker instead of podman, you will almost certainly
want to use a .dockerignore file to exclude your bundles' main data
directories; anything large that you're not using for COPY commands
in the Dockerfiles should be excluded, because Docker has to package
up *the whole dir* and send it to the daemon.  Podman doesn't have
that issue at all.

Getting Started
===============

https://github.com/lojban/haproxy makes a really good example of a
thoroughly minimal setup, whereas
https://github.com/lojban/jbovlaste-containers is much less minimal.

$ ln -s /opt/lbcs/setup.sh
$ ./setup.sh

If you use SELinux, you'll want to run initial_setup.sh as
root.

Then you need to edit at least the following files:

    config
    containers/[name]/Dockerfile.erb
    containers/[name]/config

You can make as many directories under containers/ as you want; each
must have a config file and Dockerfile.erb file.

Run ./setup.sh every time you make changes outside the container
src/ directories.

If cron/crontab exists, ./setup.sh will instally it for you.  This
will destroy whatever's already there.

Put any secrets files in .gitignore

Ordering
--------

If your stuff needs the network to be up before running (as most
things do), add "needs_network=true" to the container config file.

If a container needs to start after other container(s), put
something like this in the container config file:

    after_containers='jvs-web jvs-db'

or

    after_containers=jvs-db

You can also force something to start when its dependencies do.
Let's say that you take the DB down to fix something, and later you
bring the DB back up; assuming you've used after_containers in the
obvious way, that will leave the web container down until you remember to bring
it up.  If you do this:

    before_containers=web

in the db config and this:

    after_containers=db

in the web config, stopping the db will stop the web, and starting the db will
start the web.

All addons are treated internally as though they had an after_containers line
for the container they belong to, and all containers with addons are treated
internally as if they had before_containers for all their addons.  This means
that stopping the container stops all the addons, and starting the container
starts all the addons.

Configuration
=============

The Actual System Source Code
-----------------------------

Presumably your underlying system(s) also have source code; this
leaves you with a git-in-git situation, which always sucks.

I can't be arsed to managed submodules, so what I usually do is either:

- Move all the LBCS stuff into the system's source repo ; i.e.
  the run_container.sh links and so on are in the top level of the
  system's source repo, and the actual source is in
  container/web/src or whatever in that same repo.

  Advantages: only one repo.  Disadvantages: when people come to
  look at that repo they're going to be like "... what is all this
  crap?".

- Have the source repo in a subdirectory, and simply use .gitignore,
  i.e. container/web/src has a whole other git repo in it, and
  .gitignore in the system management repo has "container/web/src"
  in it.

  Advantages: source is clearly separated.  Disadvantages: you have
  to remember to update each repo separately.

The misc Directory
------------------

The misc directory typically contains things that you are going to
COPY into the image that don't belong in the main data directories
for the systems.  For example, when running Apache you'll typically
need to copy a file into /etc/httpd/conf.d/ in the running image;
putting it in the web container is also OK, but it's more of a
system level item than a source level item.

The addons Directory
----------------------

This is for things where we want to use the LBCS mechanism to
generate systemd service files, but we don't want to actually run a
new container.  Typically this means we want to run a second thing
inside an already extant container.  The canonical example is
https://github.com/lojban/vlasisku-containers/blob/main/container/web/addons/bots/config

secrets vs. config
------------------

The only difference between the two files is that "secrets" is
typically in .gitignore and isn't checked in.  There's no, like,
special encryption or anything, nor does the code go to any *great*
lengths to make sure that secrets aren't leaked in process table
output or whatever.

Templating
----------

We use ERB for templating, for the simple reason that I'm used to
it.  You will, therefore, need Ruby installed.

All you really need to know is that "<%= foo %>" means "replace this
blob with the value of foo as a string".

You'll want to put the non-.erb file names for your .erb files in
.gitignore, so you don't accidentally check them in.

Places templating is used:

  cron/*.erb
  misc/*.erb
  container/*/Dockerfile.erb
  Any file listed in files_to_erb_on_run in containers/*/config; example:
      files_to_erb_on_run='containers/web/src/bin/db.sh containers/web/src/lib/db.pm'

pod_args Gotcha
---------------

You have to manually restart the pod if you change anything about
it, like the port list; this generally means "manually rm the pod on
any changes to pod_args in config".

This goes like:

$ systemctl stop [everything]
$ podman pod list
$ podman pod kill [name]
$ podman pod rm [name]
$ ./setup.sh
$ systemctl start [everything]

Docker vs. Podman And Related Issues
====================================

This document, and indeed the code itself, assumes that you're
running podman, and specifically rootless podman with user
namespacing turned on (that is, pods are run with --userns=keep-id
).

The big benefit of podman over docker is that podman containers
actually run as children of the calling process, rather than via
indirection through a server process, so there's a clear process
tree that makes sense, you can kill sub-processes and it works, you
can get return values that are useful, and so on.

Rootless podman means that everything is run by the actual user
itself, without any sudo or anything.

--userns=keep-id means that if the user running podman has uid 1001,
files on shared mounts owned by 1001 will show up inside the pod as
owned by uid 1001.  So the idea is that you run your stuff inside
the pod as the same UID as the user running podman, and everything
Just Works.

This leaves us with a really, really good security posture: even if
an attacker breaks out of the container, they have no special
privileges beyond those of the calling user.

The disadvantage is that you have to convince the container to run
its main stuff with a UID that it probably isn't configured to use.
Example:

    # Make the web user match the userid of the running user; makes things way easier
    RUN usermod -u <%= userid %> apache
    RUN groupmod -g <%= groupid %> apache
    RUN find / -xdev -user 48 -print0 | xargs -0 chown apache
    RUN find / -xdev -group 48 -print0 | xargs -0 chgrp apache

This makes the "apache" user have the same userid and groupid as the
calling user, and changes the ownership of files that are owned by
the previous apache user id or group id to the new values.

"userid" and "groupid" are valid variables in all erb template
situations in lbcs for exactly this reason.

I Really Want To Run Docker
===========================

Everything we're doing here *should* work with Docker, but as I
haven't tested at all there's probably things that are hard-coded in
(like use of podman pods) that won't work.

The CONTAINER_BIN entry in /opt/lbcs/config is intended to provide a
starting point to such work.