Allow interactive command sessions in `deis run`

[Cryptophobia]

From @jeroenvisser101 on June 24, 2016 15:34

(from deis/deis#117 (comment))

This seems to be a longstanding issue but something that would be super nice for Workflow.

Is this planned yet for v2? I couldn't find any issues referencing it.

Copied from original issue: deis/workflow-cli#98

[Cryptophobia]

From @bacongobbler on June 24, 2016 15:40

Hey @jeroenvisser101, this is not yet slated as something for v2, however now that we are using k8s this should be relatively easier to implement compared to v1.

[Cryptophobia]

From @jeroenvisser101 on June 24, 2016 15:57

@bacongobbler would proxying this locally to kubectl be an option? That would make this very simple to implement, I think. (it would create a new runner pod, get the pod name, and then exec -ti in that pod, delete when done?)

[Cryptophobia]

From @bacongobbler on June 24, 2016 15:59

I don't think we can assume that the kubernetes api server is publicly accessible, which strikes out the idea of proxying to kubectl.

[Cryptophobia]

From @bacongobbler on June 24, 2016 16:3

The run implementation currently works like that (create a runner pod, get the pod name, wait for it to finish, retrieve the logs, delete when done), however the limitations are still the same as deis/deis#117. The limitation being that WSGI cannot handle long-running request/responses such as a websocket. We need to effectively create a new component to handle the proxying between the pod and the client outside of the controller w/auth... same for deis logs -f for deis/deis#465.

[Cryptophobia]

From @rimusz on October 11, 2016 13:32

any update on this?

[felixbuenemann]

FYI I've spent some time thinking about a solution to this problem.

One problem is that the deis controller currently does not support long running connections and also the kubernetes api protocol for interactive connections uses either websockets or spdy.

The idea I had was to handle the authentication/authorization and protocol negotiation in the controller and then hand of the incoming TCP client socket over a UNIX socket to an external process (using cmsg(3)/sendmsg(2) with SCM_RIGHTS) that handles proxying to the kubernetes API endpoint.

The deis client could then borrow the same code used by kubectl to implement the bidirectional connection.

Btw. the same approach could also be used for non-interactive deis run to work around the current issue of output being shown only after the run instead of as it happens and missing output for short-lived runs.

[Cryptophobia]

@felixbuenemann , this is an interesting proposal. Do you have any examples of how this TCP client socket can be handed over to the external UNIX socket which will handle the proxying to the k8s API endpoint? I'm not very versed in that area myself. How much dev work would this be in general compared to the other solutions discussed?

[bacongobbler]

we did attempt this approach a long time ago but it failed due to the limitations around serving and proxying a TCP connection from the controller to the kubernetes API.

One such example can be found here: https://github.com/mboersma/controller/tree/deis-run-bash

[felixbuenemann]

@Cryptophobia Do you have any examples of how this TCP client socket can be handed over to the external UNIX socket which will handle the proxying to the k8s API endpoint?

The proxying would be handled by an external process that listens on the unix socket and runs in the same container as the controller. An example of how to transfer file descriptors using SCM_RIGHT in python can be found in the docs for socket.sendmsg().

This process would do the proper api call for the interactive session to the k8s api emulating the same protocol handshake as the client did with the controller and then bridge the connections by copying packets bidirectionally between the client socket it inherited over the unix socket and it's own tcp connection to the api.

How much dev work would this be in general compared to the other solutions discussed?

Hard to say, but it would make it possible to leverage the existing console support from the k8s api and I think building that from scratch would be much harder.

@bacongobbler we did attempt this approach a long time ago but it failed due to the limitations around serving and proxying a TCP connection from the controller to the kubernetes API.

That's why I had the idea to hijack the client socket from django/wsgi and hand it off to another process that does not have the same restrictions and could be written in something like golang.

Transferring file handles (which includes sockets) over a unix socket is implemented in the sendmsg syscall on linux and it is accessible using the Python built-in socket module, quoting the docs:

On some systems, sendmsg() and recvmsg() can be used to pass file descriptors between processes over an AF_UNIX socket. When this facility is used (it is often restricted to SOCK_STREAM sockets), recvmsg() will return, in its ancillary data, items of the form (socket.SOL_SOCKET, socket.SCM_RIGHTS, fds), where fds is a bytes object representing the new file descriptors as a binary array of the native C int type. If recvmsg() raises an exception after the system call returns, it will first attempt to close any file descriptors received via this mechanism.

I think the latest haproxy uses the same trick to do seamless reloads (read Truly Seamless Reloads with HAProxy – No More Hacks! for details).

[Cryptophobia]

I think the latest haproxy uses the same trick to do seamless reloads (read Truly Seamless Reloads with HAProxy – No More Hacks! for details).

This is a great article about the internals of sharing sockets in Linux. Lots of knowledge I did not have before reading this. Thanks a lot. 👍 I'm also a fan of HAproxy too. 🥇

@felixbuenemann : Seems like a great solution. How would we share the authentication between the controller process and the secondary Go process in order to reuse the auth to the k8s api already established?

@felixbuenemann : You are well versed in the internals of linux socket connections. Would you be interested in joining the fork and working on this? The most difficult part I forsee is designing the Go process that is spawned by the controller and sends and receives messages via the shared socket. Otherwise, the python socket module seems straightforward.

[felixbuenemann]

@felixbuenemann : Seems like a great solution. How would we share the authentication between the controller process and the secondary Go process in order to reuse the auth to the k8s api already established?

Since the sidecar process is running in the same controller it can access the same service account used by the controller.

As for authenticating the client connecting to the deis api: This would be handled before handing off the socket handle to the external process, which is the reason why the protocol negotiation (eg. HTTP Upgrade to websockets) needs to be handled inside the controller.

@felixbuenemann : You are well versed in the internals of linux socket connections. Would you be interested in joining the fork and working on this? The most difficult part I forsee is designing the Go process that is spawned by the controller and sends and receives messages via the shared socket. Otherwise, the python socket module seems straightforward.

Unfortunately I don't have time to work on infrastructure stuff in the near future as I'm working full time on other projects, but I'm happy to share my input on implementation.

By the way, the sidecar process doesn't have to be written in golang, something like python with gevent would work just as well, since the work done by the proxy is pretty light.

I have some proof of concept scripts somewhere on disk that do a websocket handshake in django on wsgi and then bridge tie websocket connection to an echo server on the internet.

Since I wrote that code sometime in June last year I can't really say what state it is in, but it looks to handle at least the highjacking and bridging part, the hand-off to another process is not yet implemented.

I also have another script that implements a stand alone websocket proxy in python that uses polling with select and queues to handle multiple connections and rewrites the Host and Origin headers before bridging the connection.

I could share those scripts if someone is interested in implementing this approach.

The code isn't really complex but it drops down to lower layers than you're usually working with, since you need to directly work with things like http headers on the fly.

Of course it would also be possible to have the proxy run as a standalone app in a different pod and effectively redirect the deis client with a one time token that can be verified by the proxy, that way you could get around the need to hijack the client socket from wsgi, but would have to implement an extra authentication layer (and require another vhost on the deis router).