Containers generally run from an image and have no access to the host file system. This is fine for stand-alone ephemeral containers, but others require persistent data. While we can pass in environment variables to configure the app inside the container, sometimes a place to read and write more complex files is needed. When you update or replace such a container (perhaps with a new release) you want the new container to have access to the same data as the previous one.
The tricky thing with rootless containers is that you’re not root on the host and, as per my previous post, containers can run as any user id. If the container runs as root (uid 0) then that is fine as it actually maps to your non-root user on the host (e.g. 1000) and management of the data is therefore easy. However, containers running as other users (e.g. 123) will map to a uid on the host based on the subuid offset range (e.g. 100122) which will not match your non-root user and therefore data management is harder.
Providing persistent storage to a container is done by setting up a bind mounts using the --volume (or -v) option with podman. Note that volumes are not restricted to just one per container, simply repeat the -v option for as many volumes as required!
The volume option uses a colon separated format specifying the source on the host and then the destination directory where the volume is mounted to inside the container (e.g. src:/dest).
The source is generally one of two types; either a container volume (like a container image) or a regular host directory.
Container volumes
The simplest and easiest form of persistent data is to use a container volume, created at the time of container launch, as it will configure it the right permissions. These volumes are actually just a directory on the host, but managed nicely with podman volume set of commands. You can create them manually if you prefer.
If the source of the volume is just a name and not a path then podman expects a volume. If the volume does not already exist when podman run is executed, it will be created automatically and have the appropriate SELinux context set for you.
$ podman run -dit --volume src:/dest busybox
You will then be able to see the src volume on the host (along with any others).
$ podman volume ls
Host dir volumes
If the source is either a relative or absolute path, then podman expects this to be a directory on the host (which must already exist on the host).
Host dir volumes effectively work the same way as container volumes, in that they are a bind mount of a directory, it’s just that you have to specify the path and it has to already exist. Furthermore, the permissions need to be correct, else the container will not be able to access the directory. It must also have the correct SELinux context, although podman can set that for us at runtime if we use the :z option (more on that in the next section).
This mounts the directory /src on the host to /dest inside the container.
$ podman run -dit --volume /src:/dest:z busybox
Volume options
Volumes support a number of comma separated options after their source and destination declaration (separated by another colon), such as z (or Z) for setting SELinux context, ro makes the volume read only, while others such as noexec and nosuid enforce those features.
It’s easy to test, let’s try to write something to a volume that’s read only (:ro option).
$ podman run -dit --rm --name busybox -v src:/dest:ro docker.io/busybox
48a99f8e0454cb035adfd95097e5f92f6704a96153aeebcdfa63747637212d1f
$ podman exec busybox touch /dest/file
touch: /dest/file: Read-only file system
A note about SELinux
I have a follow-up post about SELinux, but in short, if your host has SELinux enabled then make sure you at least include the :z option after declaring your volumes (or :Z if you want it to be restricted to that container). This will make sure that the appropriate context is applied for the volume, otherwise the container will not be able to access it.
Volumes and uids
As per my previous post, we know that images run their process inside the container as a specific user. For example, by default busybox runs as root (uid 0) while grafana runs as grafana (uid 472).
If you run a rootless container with an image that is configured to run its process as root (uid 0), then it actually runs on the host as your non-root user (e.g. 1000). For volumes in this case, passing in any host directory owned by your non-root host user (for example, a directory in your home dir) will work just fine. The root user in the container ultimately maps to the same uid as your host user and so files and dirs have the same permissions.
Thanks to subuid ranges (see previous post), we can actually run multiple rootless containers all with different user IDs as a form of separation and security. This means the process from the container runs on the host with a unique uid, but unlike root, it will be different to our non-root user id. This can make persistent data more tricky as your host user uid won’t match subuids used by the container, and thus directories and files will have different permissions. This is not a problem if the data is only managed inside the container. We’ll investigate this a bit further in a future post, for now let’s undertand how volumes work.
Volumes and rootless containers, running as root
Let’s take a look at a rootless busybox container (which runs internally as root) and see what permissions the source (src volume) gets mounted with inside the container (at /dest).
$ podman run -dit --volume src:/dest --name busybox busybox
4ff42120b65c79ed42a48f6532c5cd259b79ec6e80351fbe30404496542a99ec
$ podman exec busybox ls -ld /dest
drwxr-xr-x 1 root root 0 Jan 29 23:07 /dest
Great! We can see that the permissions are correctly set to root inside the container.
We now also have a brand new volume called src which podman has created for us, which we can inspect.
$ podman volume inspect src
[
{
"Name": "src",
"Driver": "local",
"Mountpoint": "/home/csmart/.local/share/containers/storage/volumes/src/_data",
"CreatedAt": "2021-01-30T10:07:18.340278469+11:00",
"Labels": {},
"Scope": "local",
"Options": {},
"UID": 0,
"GID": 0,
"Anonymous": false
}
]
Note that the container is really a directory at the location /home/csmart/.local/share/containers/storage/volumes/src/_data. If we look at that directory on the host, we can see the permissions match our non-root user (remember root in a container maps to our non-root user on the host).
$ id
uid=1000(csmart) gid=1000(csmart) groups=1000(csmart)
$ ls -ldZ /home/csmart/.local/share/containers/storage/volumes/src/_data
drwxr-xr-x. 1 csmart csmart system_u:object_r:container_file_t:s0 0 Jan 30 10:07 _data
Also note that the SELinux context of the _data directory is set to container_file_t (the parent directory will instead be data_home_t, as per regular home dirs). This is set for you automatically, but it is important. Without that context, the container will not be able to operate on that directory, so it becomes important when using host-dir instead of volumes, which we’ll see later.
OK, now let’s create a file on the host and see that it appears inside the container.
$ touch /home/csmart/.local/share/containers/storage/volumes/src/_data/file
$ podman exec -it busybox ls -l /dest/file
-rw-rw-r-- 1 root root 0 Jan 29 23:10 file
Similarly, we can touch a file in the container and have it appear on the host.
$ podman exec busybox touch /dest/file2
$ ls -laZ /home/csmart/.local/share/containers/storage/volumes/src/_data/file*
-rw-rw-r--. 1 csmart csmart unconfined_u:object_r:container_file_t:s0 0 Jan 30 10:10 file
-rw-r--r--. 1 csmart csmart system_u:object_r:container_file_t:s0 0 Jan 30 10:11 file2
OK, so that hopefully shows how volumes and containers interact. It’s easy when the container runs as root, as it matches your host user.
Let’s clean up that container and volume.
$ podman rm -f busybox
4ff42120b65c79ed42a48f6532c5cd259b79ec6e80351fbe30404496542a99ec
$ podman volume remove src
src
Host-dir volumes and rootless containers, running as root
Using a host-dir volume is easy when running a rootless container as root because the uids match. Just make the directory and use the full path when passing in the volume. Note that if you’re running SELinux you must specify either :z or :Z directly after the destination argument for the volume, in order to have podman set the right SELinux context on the directory.
$ mkdir ~/src
$ podman run -dit --volume ~/src:/dest:z --name busybox busybox
57181b4ab70173f8796fda62509082e56ce334a51805c9689fee09e868cc9753
$ podman exec busybox ls -ld /dest
drwxr-xr-x 1 root root 0 Jan 29 23:07 /dest
This is effectively the same as running rootless containers as root we did above with a volume, except that the path is different (e.g. ~/src instead of /home/csmart/.local/share/containers/storage/volumes/src/_data). Everything there applies here.
Volumes and rootless containers, running as non-root
Let’s create a new container running as a different user (123) and we can see that inside the container it uses 123 but on the host it uses 100122 (remembering that according to our subuid map, uid 1 in a container maps to user 100000 on the host).
$ podman run -dit --volume src:/dest --user 123:123 --name busybox busybox
d4babfb54b2d162c4edc6061d0a2f28f666bec38433243923f418ac2d80dfa5f
$ podman exec busybox ls -ld /dest
drwxr-xr-x 1 123 123 0 Jan 29 23:14 /dest
Again, we can see that the container volume was re-created and mounted with the appropriate permissions for user 123 inside the container.
Looking on the directory on the host, we can see that this directory has been set up with the uid of 100122. This is correct because the container is not running as root, and so the subuid offset is being used.
$ ls -ldZ /home/csmart/.local/share/containers/storage/volumes/src/_data
drwxr-xr-x. 1 100122 100122 system_u:object_r:container_file_t:s0 0 Jan 30 10:14 _data
Of course, if we try to write to that directly now as our non-root user on the host, it will fail.
$ touch /home/csmart/.local/share/containers/storage/volumes/src/_data/file
touch: cannot touch 'file': Permission denied
Great! So, when it comes to running lots of rootless containers with different uids, the easiest way to do this with persistent data is with container volumes.
Host-dir volumes and rootless containers, running as root
OK so this is where things start to get a little more interesting as the uid in the container is NOT the same as our user (and we’re not root on the host) and podman will not help us out here (as it does with container volumes above).
We cannot just create a host directory as our non-root host user and pass it through, as the permissions inside the container will be root.
$ mkdir src
$ ls -lZd src
drwxrwxr-x. 2 csmart csmart system_u:object_r:container_file_t:s0 6 Jan 31 19:15 src
$ podman run -dit --volume ./src:/dest:z --user 123:123 --name busybox busybox
bd9b8e7685acaa3b03380a10724bd2c06ddaa48dbfb67b49339068e16f51d57c
$ podman exec busybox id
uid=123(123) gid=123(123)
$ podman exec busybox ls -ld /dest
drwxrwxr-x 2 root root 6 Jan 31 08:15 /dest
$ podman exec busybox touch /dest/file
touch: /dest/file: Permission denied
Obviously the container user is not able to write to the volume. So what do we do? Well we need to change the permissions so that they match the user (similar to what podman does for us automatically when using a container volume).
If you have root on the box, that’s pretty easy.
$ chown 100122:100122 src
$ ls -lZd src
drwxrwxr-x. 2 100122 100122 system_u:object_r:container_file_t:s0 6 Jan 31 19:15 src
$ podman exec busybox touch /dest/file
$ ls -lZ src/file
-rw-r--r--. 1 100122 100122 system_u:object_r:container_file_t:s0 0 Jan 31 19:20 src/file
OK, but remember we’re running rootless containers, so how do you do that as your regular old non-root user if you don’t have root on the host?
$ chown 100122:100122 src/
chown: changing ownership of 'src/': Operation not permitted
Remember that podman unshare command (see previous post) which under a new user namespace? We can use this to set the permissions, but remember we don’t actually set it to 100122 as it will be on the host, we use the container uid of 123:123.
$ podman unshare chown 123:123 ./src
$ ls -lZd src
drwxrwxr-x. 2 100122 100122 system_u:object_r:container_file_t:s0 18 Jan 31 19:20 src
Now the host directory has the right permissions and the container user will be able to write just fine!
$ podman exec busybox touch /dest/file
$ ls -lZ src/file
-rw-r--r--. 1 100122 100122 system_u:object_r:container_file_t:s0 0 Jan 31 19:23 src/file
Accessing data as non-root user and non-root container user
Although they are very similar, if you want to be able to access the data as both your non-root user on the host and the container, you’re probably better off using host dir volumes over container volumes as it’s easier to see and manage.
In a future post I’ll talk about how we can modify these host directories so that we can access them both as our non-root host user and also as the user in the container. I think this post has gotten long enough for now…
There is much that irks me in academia. The way that disciplines are almost randomly assigned to artium, scientiae, or legum, without any reference to their means of verification or falsification. Or, for that matter, the Dewey (or Universal) Decimal Classification for libraries, which, in its insanity, places computer applications in the same category as "Fundamentals of knowledge and culture" and "Propaedeutics". One could also describe ask why the value "Dead languages of unknown affiliation" also belongs with "Caucasian languages". I suppose most of them are "near dead", right?
Scraping a site is fragile, impossible to transform into the full content, and provides an incomplete export of the site: anything that’s only available in the site dashboard can’t be obtained through scraping.
(@exaltiora) 
A 
