How to ssh into a Kubernetes container.
Intro
ssh is a fundamental, extremely useful and orthogonal system administration tool, that can be used in and integrated into toolchains. It’s also useful in combination with Kubernetes.
In the following I’ll assume that:
- you do not intend to throw away your tools just to stay kubernetically pure
- you only want to start the ssh server on demand
- one set of ssh host keys get generated per pod. Every container in the pod has the same keys.
Howto
First let’s add the ssh server to your docker image.
RUN dnf -y --setopt=install_weak_deps=False install \
openssh-server \
&& sed --in-place 's/#ListenAddress ::/#ListenAddress ::\nListenAddress localhost/' /etc/ssh/sshd_config \
&& dnf clean all
Note that above I’m setting up sshd
to listen on
localhost only.
Create the ssh secrets. I create two different secrets:
- one for ssh ingress (secrets for incoming connections)
- one for ssh egress (secrets for outgoing connections)
Since I keep those secrets crypted with ansible-vault
, I’d do:
$ ansible-vault edit ssh-egress.yaml.vault
$ ansible-vault edit ssh-ingress.yaml.vault
… and enter the secrets:
apiVersion: v1
kind: Secret
metadata:
name: ssh-egress
type: Opaque
stringData:
# Generated with ssh-keygen -b 2048 -t rsa -f id_rsa -q -N ''
id_rsa: |-
-----BEGIN RSA PRIVATE KEY-----
Your secret key here
-----END RSA PRIVATE KEY-----
id_rsa.pub: |-
ssh-rsa your public key here
known_hosts: |-
|1|in case you want to have prepared known_host entries...
|1|then log into them from your own machine and copy those...
|1|entries over here.
apiVersion: v1
kind: Secret
metadata:
name: ssh-ingress
type: Opaque
stringData:
authorized_keys: |-
# That's Joe's key
ssh-rsa AAAAB... your public key(s) here
#
# generated with: ssh-keygen -N '' -t rsa
ssh_host_rsa_key: |
-----BEGIN OPENSSH PRIVATE KEY-----
key contents here
-----END OPENSSH PRIVATE KEY-----
ssh_host_rsa_key.pub: |
ssh-rsa AAAA... key contents here
#
# generated with: ssh-keygen -N '' -t ecdsa
ssh_host_ecdsa_key: |
-----BEGIN OPENSSH PRIVATE KEY-----
key contents here
-----END OPENSSH PRIVATE KEY-----
ssh_host_ecdsa_key.pub: |
ecdsa-sha2-nistp256 AAAA... key contents here
#
# generated with: ssh-keygen -N '' -t ed25519
ssh_host_ed25519_key: |
-----BEGIN OPENSSH PRIVATE KEY-----
key contents here
-----END OPENSSH PRIVATE KEY-----
ssh_host_ed25519_key.pub: |
ssh-ed25519 AAAA... key contents here
Note that for some reason my sshd will not recognize keys without a trailing newline.
Therefore I’m using the somethingkey: |
notation instead of the somethingkey: |-
notation here.
Now apply the secrets:
$ ansible-vault view ssh-ingress.yaml.vault | kubectl apply -f -
$ ansible-vault view ssh-egress.yaml.vault | kubectl apply -f -
And use those secrets in a pod:
---
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- image: your-image-URL-here
name: my-pod
# https://github.com/kubernetes/kubernetes/issues/34982
lifecycle:
postStart:
exec:
command:
- bash
- -c
- mkdir /root/.ssh && chmod go-rwx /root/.ssh && cp -aL /root/.ssh-egress/* /root/.ssh && cp -aL /root/.ssh-ingress/authorized_keys /root/.ssh && cp -aL /root/.ssh-ingress/ssh_host_*_key* /etc/ssh/
volumeMounts:
- name: ssh-ingress
# https://github.com/kubernetes/kubernetes/issues/34982
mountPath: "/root/.ssh-ingress"
readOnly: true
- name: ssh-egress
mountPath: "/root/.ssh-egress"
readOnly: true
imagePullSecrets:
- name: regcred-your-handle-here
volumes:
- name: ssh-egress
secret:
secretName: ssh-egress
# 0400 -> 256
defaultMode: 256
items:
- key: id_rsa
path: id_rsa
- key: id_rsa.pub
path: id_rsa.pub
- key: known_hosts
path: known_hosts
- name: ssh-ingress
secret:
secretName: ssh-ingress
# 0400 -> 256
defaultMode: 256
items:
- key: authorized_keys
path: authorized_keys
- key: ssh_host_rsa_key
path: ssh_host_rsa_key
- key: ssh_host_rsa_key.pub
path: ssh_host_rsa_key.pub
- key: ssh_host_ecdsa_key
path: ssh_host_ecdsa_key
- key: ssh_host_ecdsa_key.pub
path: ssh_host_ecdsa_key.pub
- key: ssh_host_ed25519_key
path: ssh_host_ed25519_key
- key: ssh_host_ed25519_key.pub
path: ssh_host_ed25519_key.pub
Apply that pod config and start sshd in the pod:
$ kubectl exec my-pod -- /usr/sbin/sshd
Portforward a random local port of your choice to that pod:
kubectl port-forward my-pod 2222:22
Create a useful alias for that pod (so that ssh won’t complain about changing keys when you’d connect to localhost instead):
$ cat /etc/hosts | grep my-pod
127.0.0.1 localhost my-pod
Connect to the pod via ssh:
$ ssh root@my-pod -p 2222
You may want to kill sshd
once you do not need it any more.
Note a lot of the trouble here is because Kubernetes’ way of setting permissions seems to be unworkable - see https://github.com/kubernetes/kubernetes/issues/34982 .
Please let me know if there’s progress on that issue and I’ll try to update this article.
Update: Article was updated so that ssh host keys are not present in the docker container image.