[jaypipes]

Updates the controller/namespace package to use the temporalio/kube-temporal/pkg/reconciler/base.Reconciler struct and adds a resource-operator pkg/reconciler/base.Reconciler that builds on top of that and adds shared Temporal Cloud connection handling.

Left the implementation of the types.Resource.ResourceSetLatest, types.Resource.ResourceDelete and types.Resource.ResourceSync as TODOs for Zain to tackle but left example unit tests and testutil infrastructure to make the common unit testing easier.

Issue #7

[jaypipes]

Note to reviewers: CI jobs will fail until the temporalio/kube-temporal#41 is merged and we cut a release of the kube-temporal package. /cc @tomba7 @zainawaisn @niyomukiza-mechack

[tomba7]

This is a great setup @jaypipes. Thank you very much. The seams are smooth like butter. On another note, it would be good to bump the go.temporal.io/cloud-sdk to the latest to pick up project ID and description on the namespcae resource

[tomba7]

Its missing WithKubeNamespace

[tomba7]

Should this return cloudservice.CloudServiceClient instead of tcclient.Client? Then reconcilers could take a fake Cloud client through a With option, and unit tests wouldn't need to build a real gRPC client.

[tomba7]

You've probably already thought this through, but I'm wondering if we have a generic implementation of types.Resource in kube-temporal? Then we wouldn't have to write this wrapper for each resource. I haven't looked closely enough yet, but at a glance it looks like it would have the same shape for every resource.

[tomba7]

Is this the only path the Reconcile is intended to go through at the moment? i.e. no manager with a default client set via WithDefaultCloudClient? But on the other hand I also don't see the default client being set anywhere starting from main.

[jaypipes]

Is this the only path the Reconcile is intended to go through at the moment? i.e. no manager with a default client set via WithDefaultCloudClient? But on the other hand I also don't see the default client being set anywhere starting from main.

@tomba7 I still need to plumb some configuration fields from a resource-operator/pkg/controller/config package into cmd/resource-operator/main.go that will allow the default cloud connection information to be passed to the controller/namespace.New() function.

[tomba7]

I couldn't find where this annotation gets set. I'm assuming users need to add it to each CR by hand for now? Plus some additional plumbing for the API key to work? i.e. a Secret mounted as a vol in the operator Deployment manifest?

[jaypipes]

I couldn't find where this annotation gets set. I'm assuming users need to add it to each CR by hand for now? Plus some additional plumbing for the API key to work? i.e. a Secret mounted as a vol in the operator Deployment manifest?

It's an optional override so that the creator of the cloud.temporal.io/Namespace CR can indicate which Temporal Cloud API endpoint the resource-operator should contact. The creator of the cloud.temporal.io/Namespace CR should not need to know anything about credentials or API keys.

Let's walk through an example where TRO is set up with some default connection credentials and another connection credentials for some Temporal Cloud API endpoint called "other-conn".

Basic flow I'm envisioning is this:

1. Admin persona installs TRO and configures its default connection credentials to the default Temporal Cloud API endpoint:

TRO_NAMESPACE=temporal-resource-operator
TRO_DEPLOYMENT_NAME=temporal-resource-operator

# install Temporal Resource Operator using Kustomize (built in to kubectl as its -k option)
kubectl apply -n $TRO_NAMESPACE \
  -k "https://github.com/temporalio/resource-operator/deploy/kustomize?ref=main"

# or ... helm upgrade --install <TRO_CHART> ...

# create a Kubernetes Secret that will contain the API Key that TRO will use when connecting
# to the default Temporal Cloud API endpoint

DEFAULT_APIKEY="supersecret!"
DEFAULT_APIKEY_B64=$(echo -n $DEFAULT_APIKEY | base64)
DEFAULT_APIKEY_SECRET_NAME=default-tmprl-cloud-api-key

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: $DEFAULT_APIKEY_SECRET_NAME
  namespace: $TRO_NAMESPACE
type: Opaque
data:
  apikey: $DEFAULT_APIKEY_B64
EOF

# create a cloud.temporal.io/ClusterConnection that TRO will use to connect to the
# default Temporal Cloud API endpoint that will load its API Key from the Secret

DEFAULT_CLOUD_API_ENDPOINT="saas-api.tmprl.cloud:443"
DEFAULT_CONN_NAME=default-tmprl-cloud

cat <<EOF | kubectl apply -f -
apiVersion: cloud.temporal.io/v1alpha1
kind: ClusterConnection
metadata:
  name: $DEFAULT_CONN_NAME
spec:
  hostPort: $DEFAULT_CLOUD_API_ENDPOINT
  credentials:
    apiKey:
      secretRef:
         name: $DEFAULT_APIKEY_SECRET_NAME
         key: apikey
EOF

# tell TRO to use the cloud.temporal.io/ClusterConnection as its default connection config
kubectl patch deployment $TRO_DEPLOYMENT_NAME -n $TRO_NAMESPACE \
  --patch-file=/dev/stdin <<'EOF'
{
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "name": "resource-operator",
            "env": [
              {
                "name": "DEFAULT_CLOUD_CONNECTION_NAME",
                "value": "$DEFAULT_CONN_NAME"
              }
            ]
          }
        ]
      }
    }
  }
}
EOF

2. Admin persona creates another Connection with credentials for connecting to another Temporal Cloud API endpoint

# create a Kubernetes Secret that will contain the API Key that TRO will use when connecting
# to the other Temporal Cloud API endpoint

OTHER_APIKEY="supersecret!"
OTHER_APIKEY_B64=$(echo -n $OTHER_APIKEY | base64)
OTHER_APIKEY_SECRET_NAME=other-tmprl-cloud-api-key

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: $OTHER_APIKEY_SECRET_NAME
  namespace: $TRO_NAMESPACE
type: Opaque
data:
  apikey: $OTHER_APIKEY_B64
EOF

# create a cloud.temporal.io/Connection that TRO will use to connect to the
# other Temporal Cloud API endpoint that will load its API Key from the Secret

TRO_CONNECTION_LABEL_KEY="resource-operator.temporal.io/connection"
TRO_CONNECTION_LABEL_VALUE="resource-operator"
OTHER_CLOUD_API_ENDPOINT="other.saas-api.tmprl.cloud:443"
OTHER_CONN_NAME=other-tmprl-cloud

cat <<EOF | kubectl apply -f -
apiVersion: cloud.temporal.io/v1alpha1
kind: Connection
metadata:
  name: $OTHER_CONN_NAME
  namespace: $TRO_NAMESPACE
  labels:
    $TRO_CONNECTION_LABEL_KEY: $TRO_CONNECTION_LABEL_VALUE
spec:
  hostPort: $OTHER_CLOUD_API_ENDPOINT
  credentials:
    apiKey:
      secretRef:
         name: $OTHER_APIKEY_SECRET_NAME
         key: apikey
EOF

# tell TRO to watch and use any cloud.temporal.io/ClusterConnection or
# cloud.temporal.io/Connection resources that match a particular label.
kubectl patch deployment $TRO_DEPLOYMENT_NAME -n $TRO_NAMESPACE \
  --patch-file=/dev/stdin <<'EOF'
{
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "name": "resource-operator",
            "env": [
              {
                "name": "CLOUD_CONNECTION_WATCH_LABEL",
                "value": "$TRO_CONNECTION_LABEL_KEY=$TRO_CONNECTION_LABEL_VALUE"
              }
            ]
          }
        ]
      }
    }
  }
}
EOF

The above will cause the TRO deployment to restart, and on startup TRO will add a watch on all ClusterConnection and Connection resources that have the associated label.

3. Operator persona (could be same as Admin persona but doesn't need to be) applies their cloud.temporal.io/Namespace resources

If the user intends to apply the resource to the default Temporal Cloud API endpoint, they don't need to do anything special. Simply apply the manifest:

cat <<EOF | kubectl apply -f -
apiVersion: cloud.temporal.io/v1alpha1
kind: Namespace
metadata:
  name: my-temporal-namespace
  namespace: my-temporal-resources
spec:
  project: my-temporal-project-id
  retentionDays: 7
EOF

If the user intends to apply the resource to the other Temporal Cloud API endpoint, they would set the cloud.temporal.io/hostport annotation:

cat <<EOF | kubectl apply -f -
apiVersion: cloud.temporal.io/v1alpha1
kind: Namespace
metadata:
  name: my-other-temporal-namespace
  namespace: my-temporal-resources
  annotations:
    cloud.temporal.io/hostport: $OTHER_CLOUD_API_ENDPOINT
spec:
  project: my-temporal-project-id
  retentionDays: 28
EOF

This way, the persona creating the Temporal Cloud Namespaces doesn't know anything about credentials and only needs to specify (optionally) which Temporal Cloud API endpoint to operate against.