Add passwd/group file entries when running images as assigned user ID not in passwd file.

[GrahamDumpleton]

Details of the reason for this change in #552

The fix for the problem works by providing a script for the container entry point which makes changes to the passwd and group files and then executes the original tini command that was previously used for the container entry point.

[GrahamDumpleton]

Forgot to modify Dockerfile.ppc64le. Further update to come.

[GrahamDumpleton]

Should be noted that Dockerfile.ppc64le doesn't seem to have been tracking Dockerfile for other changes. So looks like it is out of date and missing things.

[minrk]

This makes it trivial for any user to become root in their container:

cp /etc/passwd /tmp/passwd
rootpass=`openssl passwd -1 root`
cat /tmp/passwd | sed "s/root:x/root:${rootpass}/" > /etc/passwd

Now the user can su root with the password 'root'

I think we need another solution that doesn't grant universal su root access.

This does only affect putting users into the root group, so not the default case, but it does mean that putting users into the root group is equivalent to letting the users run as root. Is that the intended effect?

[GrahamDumpleton]

I will check, but I don't believe that will work in secure environments like Kubernetes/OpenShift where you are blocked from running setuid executables and various related system calls behind the ability to change real user/group.

[GrahamDumpleton]

Yep, this is restricted in Kubernetes/OpenShift.

jovyan@image:~$ su root
Password:
initgroups: Operation not permitted

This is because secure container environments will drop capabilities.

                "securityContext": {
                    "capabilities": {
                        "drop": [
                            "KILL",
                            "MKNOD",
                            "SETGID",
                            "SETUID"
                        ]
                    },
                    "privileged": false,
                    "runAsUser": 1000030000,
                    "seLinuxOptions": {
                        "level": "s0:c6,c0"
                    }
                },

[parente]

Re: Dockerfile.ppc64le, I think it's safe to leave that be for now. It's a good example of items we're not able to effectively maintain in this repo that should probably reside in another repository.

[parente]

Is it possible for this logic to live in the start* scripts after tini runs instead of introducing additional scripts? I understand that Python and other programs may balk when the UID is not found in /etc/users and /etc/group, but does tini have that problem?

Simply looking to keep the number of separate workflow scripts we need to maintain low.

[GrahamDumpleton]

The reason for adding it in an ENTRYPOINT script and doing it in there is that it gets applied even if someone overrides the CMD to do something else. For example, in JupyterHub, the CMD for the notebook images is overridden to execute jupyterhub-singleuser directly, bypassing start.sh.

[minrk]

This will depend on the Spawner. It is true of KubeSpawner, but isn't true of DockerSpawner in general (deployments may override CMD, but it is not overridden by default).

Doing this as an entrypoint makes sense to me. I think the same might be said for much of what's in start.sh, though. Effectively, putting this and only this in entrypoint is saying that the passwd entry is more important than and should be harder to opt-out of than anything in start.sh. That may be right, but I think we should think carefully about this in the future.

[GrahamDumpleton]

Note, I will make one more change to this. I will add a check that doesn't do anything if id -g is not 0. This is because need to have group gid of 0 to update files. This only occurs when image run with UID not in passwd file to begin with. So further constrain this to narrow use case where required.

[GrahamDumpleton]

Have made change to check running as GID of 0 since relying on group being that to write passwd and group files. So should all be good to go.

Summarising what we have.

• Add container-entrypoint script from which can make passwd/group changes because need to be done even if someone overrides CMD such as in JupyterHub.
• Add separate fix-passwd-entry script instead of having it on container-entrypoint so can be executed by someone in custom container entrypoint script of derived image if necessary.
• Make passwd/group files writable by root group. In normal use cases they still aren't able to be updated. Only time would be able to be updated is specific case where UID not in passwd file is used and so falls back to using GID of 0 as a result, such as is case in multitenant Kubernetes/OpenShift.
• Although could technically do whenever GID is 0, only do it if UID greater than 100000. Kubernetes/OpenShift actually use user IDs 1000000000 and above, which is well above that value. Not touching things when less than 100000 avoids potential that people with derived images already would be affected.
• Add the new entry in passwd file for arbitrary user ID being used which isn't in passwd file.
• Map original 1000 user ID of jovyan to different name (nayvoj - jovyan backwards), so those user IDs still show a name.
• In specific case where supplemental group ID same as user ID is added, add an entry in group for that as well.

As far as is known there is no risk with making passwd/group files writable to root and doesn't provide any ability for use to get elevated privileges.

FWIW, the only other way this whole thing could be solved is by by fiddling around with shared libraries and forced preloading of them into processes. This is talked about in:

• http://blog.dscpl.com.au/2015/12/unknown-user-when-running-docker.html

That other methods has lots of problems, including that nss_wrapper isn't packaged in stable for Debian/Ubuntu and that LD_PRELOAD and other environment variables will not be set if someone uses docker exec to enter container, or if application strips environment variables when running sub processes etc. So right now the writable passwd/group files and doing it from container entrypoint has been found to be best option.

[minrk]

We have similar logic to this in start.sh, and it currently enables:

• setting user id
• setting user name
• granting sudo access

The start.sh logic requires launching the image as the root user rather than the root group. Effectively, we have a fork of two different behaviors with the same goal - one dedicated for openshift/some kubernetes deployments (this one) and one for most others (in start.sh). They accomplish the same thing in different ways and in different code paths. Perhaps we should reconcile them. It would probably be nice if we could accomplish the existing root-requiring logic in start.sh via root group rather than root user. Would you like to take a stab at that?

[GrahamDumpleton]

The problem still gets back to fact that JupyterHub only runs jupyterhub-singleuser and not start-singleuser.sh. Because of my use case for JupyterHub that isn't a complete stumbling block though as it is already necessary to configure KubeSpawner to set supplemental group of users on the container since docker-stacks images aren't setup to use group root for everything, but rely on groups users when user ID is not 1000. Since already doing that, no drama to also override the startup command to be start-singleuser.sh. Since it internally uses start.sh to run jupyterhub-singleuser then integrating it with start.sh is probably okay.

I can't see that this will change the need to make passwd and group files writable to root group. The only way around that is messy and requires compiled C programs that are setuid so can become effective user of root, but not real user as kernel setuid() call blocked, such that can update files in that C program. Am not confident that isn't blocked by some configurations for running containers.

Even if make passwd and group writable, my recollection is that commands like usermod and groupmod don't work as that change is not enough. When I tested that it may have been on CentOS though so possible that Debian is different. So chance that have to still make changes to passwd and group manually. I will do some more testing and see what can be done using start.sh.

[GrahamDumpleton]

See instead #559 for new attempt to address this, with changes being done from start.sh.

[minrk]

Thanks for #559!