Customizing ArgoCD for SOPS usage

Nasazeni SOPS pro GITOPS pouziti a auto decrypting prez Azure KV. Využití ManagedIdentity a AAD pod identity pro přístup k KV

2021-02-24

OPENSHIFT

ARGOCD

Content

*file: 01-argoCD-SOPS-MI.md *

ArgoCD - pouziti SOPS pro auto decrypting prez managed Identitu

AAD pod identity

Nakonfigurujeme aad pod identity tak aby propisoval UserManagedIdentitu nedefinovanou v Azure.

# AKS v azure pouziva kubenet
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install aad-pod-identity aad-pod-identity/aad-pod-identity --set nmi.allowNetworkPluginKubenet=true
# jedna replika nam staci
kb scale --replicas=1  deploy/aad-pod-identity-mic

# identity in azure v RG AKS
export IDENTITY_RESOURCE_GROUP='MC_opsdemo_opsdemoAKS_westeurope'
export IDENTITY_NAME='pod-identity'
az identity create -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME}
export SUBSCRIPTION_ID=$(az account show --query id -o tsv) 
export IDENTITY_RESOURCE_ID="$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} --query id -otsv)" && echo ${IDENTITY_RESOURCE_ID}
export IDENTITY_CLIENT_ID="$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} --query clientId -otsv)" && echo ${IDENTITY_CLIENT_ID}
export IDENTITY_ASSIGNMENT_ID="$(az role assignment create --role Reader --assignee ${IDENTITY_CLIENT_ID} --scope /subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${IDENTITY_RESOURCE_GROUP} --query id -otsv)"

# vytvoreni CR
cat <<EOF | kubectl apply -f -
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: $(echo ${IDENTITY_NAME}|tr '[:upper:]' '[:lower:]|tr '_' '-')
spec:
  type: 0
  resourceID: ${IDENTITY_RESOURCE_ID}
  clientID: ${IDENTITY_CLIENT_ID}
EOF

cat <<EOF | kubectl apply -f -
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: $(echo ${IDENTITY_NAME}-binding|tr '[:upper:]' '[:lower:]')
spec:
  azureIdentity: ${IDENTITY_NAME}
  selector: ${IDENTITY_NAME}
EOF

Pro labeling v ramci deploye pouzijeme aadpodidbinding z hodnotou z fieldu selector

      labels:
        aadpodidbinding: pod-identity

ARGOCD install CUSTOMTOOLS (REPO-SERVER)

Pro ArgoCD bude potreba upravit deploy repo serveru tak aby obsahoval binárku SOPS a používal Managed Identity pro
autentifikaci oproti AzureKV (sops key).

[ openshift/GITOPS/argocd_sops_plugin/ag-customtools.yaml ]

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: argo
    meta.helm.sh/release-namespace: default
  labels:
    app.kubernetes.io/component: repo-server
    app.kubernetes.io/instance: argo
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-repo-server
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v1.7.6
    helm.sh/chart: argo-cd-2.11.0
  name: argo-argocd-repo-server
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app.kubernetes.io/instance: argo
      app.kubernetes.io/name: argocd-repo-server
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        aadpodidbinding: pod-identity
        app.kubernetes.io/component: repo-server
        app.kubernetes.io/instance: argo
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: argocd-repo-server
        app.kubernetes.io/part-of: argocd
        app.kubernetes.io/version: v1.7.6
        helm.sh/chart: argo-cd-2.11.0
    spec:
      initContainers:
      - name: download-tools
        image: alpine:3.8
        command: [sh, -c]
        args:
        - wget -qO sops https://github.com/mozilla/sops/releases/download/v3.6.1/sops-v3.6.1.linux &&
          mv sops /custom-tools/ && chmod 777 /custom-tools/sops
        volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
      containers:
      - command:
        - uid_entrypoint.sh
        - argocd-repo-server
        - --redis
        - argo-argocd-redis:6379
        - --logformat
        - text
        - --loglevel
        - info
        image: argoproj/argocd:v1.7.6
        imagePullPolicy: IfNotPresent
        env:
          - name: AZURE_AUTH_METHOD
            value: msi
          - name: AZURE_CLIENT_ID
            value: 1c179ad3-69b0-4d18-a5a4-98346b10acce
        livenessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8081
          timeoutSeconds: 1
        name: repo-server
        ports:
        - containerPort: 8081
          name: repo-server
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8081
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /app/config/ssh
          name: ssh-known-hosts
        - mountPath: /usr/local/bin/sops
          name: custom-tools
          subPath: sops
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: argocd-ssh-known-hosts-cm
        name: ssh-known-hosts
      - name: custom-tools
        emptyDir: {}

AZURE KV

Vytvorime Azure KV a klic, priradime AccessPolicy pro vytvorenou MSI (UserManagedIdentitu)

KOMFIGURACE MOZILLA SOPS

Zakladni konfigurace SOPS muze vypadat treba takto.

cat <<EOF >.sops.yaml
creation_rules:
# cesta ke klici
  - azure_keyvault: 'https://td-sops-keyvault.vault.azure.net/keys/sops-key1/e3499117c89b4d40ac6e651bc5e9f80b'
  # elementy pro encrypt
    encrypted_regex: "^(data)$"
EOF

cat <<EOF >secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: secret-name
type: Opaque
data:
  ahoj: cmFkIHRlIHZpZGlt
EOF

sops --encrypt secret.yaml

ConfigManagementPlugins-ArgoCD

Konfigurace je ulozena v ConfigMap argocd-cm.
Pro pridani pluginu

[ openshift/GITOPS/argocd_sops_plugin/argocd-cm.yaml ]

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: argo
    meta.helm.sh/release-namespace: default
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argo
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
    helm.sh/chart: argo-cd-2.11.0
  name: argocd-cm
  namespace: default
data:
  application.instanceLabelKey: argocd.argoproj.io/instance
  url: https://argocd.example.com
  configManagementPlugins: |
    - name: sops
      generate:                          # Optional command to initialize application source directory
        command: ["/bin/sh"]
        args:
          - -c
          - |
            /bin/bash << 'EOF'
            for filename in $(find . -name \*yaml.enc)
            do
            if  (sops -d --output-type yaml --input-type yaml ${filename})
            then
              sops -d --output-type yaml --input-type yaml $filename
            fi
            done

            EOF

Myslenka je takova ze zakryptovane soubory budou mit koncovku .enc. V ramci loopy se vyctou jejich vysledek posle na STDOUT a je zpracovan jako
kubectl apply -

# definice aplikace
kb port-forward argo-argocd-server-5d44f7f9f-mqlj9 8888:8080
argocd login 127.0.0.1:8888 --username admin --password 'argo-argocd-server-5d44f7f9f-g7q6b'
argocd app create sops --repo https://github.com/tomasdedic/sops.git --path . --dest-namespace default --dest-server https://kubernetes.default.svc --config-management-plugin sops

# definice aplikace s pouzitim pluginu SOPS
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sops
  namespace: default
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: .
    plugin:
      name: sops
    repoURL: https://github.com/tomasdedic/sops.git

TODO

pokud se pouzije plugin tak pak uz se dal neudela apply na zbyvajici manifesty ktere nejsou encrypted, chtelo by to nejake retezeni.
Pripadne v init fazi pluginu vyrendrovat manifesty z .enc a v generate fazi je pak apply. Stav je takovy ze se provede pouze ten plugin.
predani parametru z application.argoproj.io (potrebujeme vedet recursivitu)

# tak takhle to nepujde
    plugin:
      name: sops
    env:
      - name: RECURSE
        valueFrom:
          fieldRef:
            fieldPath: spec.source.directory.recurse

prepsat mozilla sops tak aby pokuv v manifestu neni nic co by mel encryptovat tak pouze manifest print na stdout

ARGOCD