Secrets management in tencent Kubernetes Engine

The challenge lies in retrieving secrets in a "cloud-agnostic" manner and storing them for the cluster.

Typically, we utilize Crossplane to generate and store secrets in the cloud provider. For this purpose, we use the Secret resource from Crossplane. The object in the Tencent provider exists and functions perfectly, creating the secret. However, difficulties arise when we attempt to retrieve the secret from the cluster and store it in the SSM. The Secret Version object lacks the capability to use a secret from the cluster, and we can only fill it manually, which poses a security risk.

A viable alternative is to employ the external-secrets operator. We can push secrets using the operator’s PushSecret feature. Unfortunately, the operator’s feature implementation for Tencent Cloud is not yet available.

Our final implementation for storing the secret involves using Crossplane with the Upbound Terraform Provider. This provider allows us to retrieve a secret from Kubernetes and add it as a variable, as shown below:

apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  name: hxops-thanos-dev-shanghai-ten-secret-version
spec:
  forProvider:
    entrypoint: ''
    env:
      - name: TF_VAR_secret_id
        secretKeyRef:
          key: attribute.secret_id
          name: hxops-thanos-dev-shanghai-ten-user-connection
          namespace: monitoring
      - name: TF_VAR_secret_key
        secretKeyRef:
          key: attribute.secret_key
          name: hxops-thanos-dev-shanghai-ten-user-connection
          namespace: monitoring
    module: |
      resource "tencentcloud_ssm_secret_version" "secret_version" {
        secret_name   = "hxops-thanos-dev-shanghai-ten"
        version_id    = "latest"
        secret_string = jsonencode({
          secret_id=var.secret_id
          secret_key=var.secret_key
        })
      }
      variable "secret_id" {
        type = string
      }
      variable "secret_key" {
        type = string
      }        

After pushing the secret to the SSM, we utilize the External Secrets Operator. The official Tencent documentation provides instructions on how to enable it. Currently, we enable it through the console, but it would be more efficient to have the ability to enable it using the official Helm Chart.

This is example utilization of the external-secrets operator:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: thanos-objstore-secret
spec:
  data:
    - remoteRef:
        key: hxops-thanos-dev-shanghai-ten
        property: secret_id
        version: latest
      secretKey: secret_id
    - remoteRef:
        key: hxops-thanos-dev-shanghai-ten
        property: secret_key
        version: latest
      secretKey: secret_key
  refreshInterval: 5m
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: thanos-objstore-secret
    template:
      data:
        objstore.yml: |
          type: COS
          config:
            bucket: "hxops-thanos-dev-shanghai-ten"
            region: "ap-shanghai"
            app_id: "1325766665"
            secret_key: '{{ .secret_key }}'
            secret_id: '{{ .secret_id }}'