When managing multiple databases—whether SQL or NoSQL—with your application running on Kubernetes, securing credentials becomes critical. A common approach is to use Kubernetes Secrets, which store credentials as Base64-encoded values. However, without an encrypted etcd storage solution, this method poses significant risks. These secrets are not inherently secure, and if the etcd backend is compromised, your credentials could be exposed.

As a leading cloud consulting company, we help businesses implement more secure alternatives for managing sensitive data in Kubernetes environments. From leveraging external secret management systems like AWS Secrets Manager or HashiCorp Vault to enforcing strict access controls and encryption practices, we ensure your infrastructure remains secure, compliant, and resilient against potential breaches.

By leveraging cloud DevOps services, organizations can implement more secure secret management solutions, such as HashiCorp Vault or AWS Secrets Manager, to ensure encrypted storage, controlled access, and automated secret rotation. This enhances security, reduces the risk of credential leaks, and strengthens overall cloud infrastructure protection

Vault from HashiCorp is a secure and reliable solution for storing and managing secrets, especially for organizations that do not want to rely on Kubernetes’ built-in secrets management. It automates secret management, eliminating the need for an administrator to manually handle credentials. By bridging the gap between development and operations teams, Vault enhances security, compliance, and efficiency.

Cloud companies in India, such as CloudifyOps, can leverage Vault to offer robust security solutions for enterprises managing sensitive data across hybrid and multi-cloud environments. Vault from HashiCorp provides a secure and reliable way to store and manage secrets, eliminating the need for Kubernetes’ built-in secret management. It automates credential handling, reducing administrative overhead and enhancing security. By bridging the gap between development and operations teams, Vault ensures seamless access control, compliance, and data protection in cloud-native environments.

The above was a sample case for using static secrets.

This blog, the first of two parts, explores managing static secrets in Kubernetes through Vault. In the next part, we will look at dynamic secrets with MongoDB.

Prerequisites are:

• Kubernetes cluster up and running
• Kubectl installed
• Helm installed

Tech Stack:

• HashiCorp Vault
• DynamoDB

What is Vault?

Vault is an identity-based secrets and encryption management system. A secret can include API encryption keys, passwords or certificates, anything you want to strictly control access to. Vault provides encryption services guarded by authentication and authorization methods. Using Vault’s UI, CLI, or HTTP API, secrets and other sensitive data can be securely stored and their access can be managed, tightly controlled (restricted), and auditable.

Architecture:

Installation

Install the Vault using Helm chart

• helm repo add hashicorp https://helm.releases.hashicorp.com

Edit the helm chart:

1. Open values.yaml

– We are not going with the gp2 type of pv, we will use dynamodb for HA.

–  Go to the helm default folder of your user.cache/helm/repository

2. Disable backend storage –  dataStorage

3. Enable HA – For high availability

4. Under “ha”: provide dynamodb config details as shown below.  Enable the UI so we can access the Vault server from the browser.

*the user credential which we need to pass must have dynamodb full access.

• Install Vault using helm charts:

– helm install vault hashicorp/vault

• Now check the status:

– kubectl get pods -n monitoring |grep vault

Initialize and unseal one Vault pod

Vault starts uninitialized and in the sealed state. Before initialization, the Integrated Storage backend is not prepared to receive data.

• Initialize Vault with one key share and one key threshold.

kubectl exec vault-0 — vault operator init

-key-shares=1

-key-threshold=1

-format=json > cluster-keys.json

The operator initcommand generates a root key that it disassembles into key shares -key-shares=n  (means it will split it n no of share key) and

-key-threshold=x (means to unseal the vault server the minimum no of share keys you will need to have is x).

-format=json  (it will define the property of your credential file and you can put it in any path)

Here the output is redirected to a file named cluster-keys.json.

• Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key

VAULT_UNSEAL_KEY=$(cat cluster-keys.json | jq -r “.unseal_keys_b64[]”)

kubectl exec vault-0 — vault operator unseal $VAULT_UNSEAL_KEY

kubectl exec vault-0 — vault status

Setting a static secret in Vault

• Connect to the Vault server – pod

kubectl exec -it vault-0 — /bin/sh

• Enable kv-v2 secrets at the path internal
• Vault login

vault secrets enable -path=internal kv-v2

• Create a secret at path internal/database/clops-app

vault kv put internal/database/clops-app DATABASE_USERNAME=”clops” DATABASE_PASSWORD=”random_passord” DATABASE_HOST=”database_endpoint”

• Verify the secret

vault kv get internal/database/clops-app

Enabling Kubernetes authentication

Vault provides a Kubernetes authentication method that enables authentication with a K8s service account token to the client who wants to access the Vault server. Every pod when created is provided a token.

• kubectl exec -it vault-0 — /bin/sh
• vault auth enable kubernetes
• vault write auth/kubernetes/config

kubernetes_host=”https://$KUBERNETES_PORT_443_TCP_ADDR:443″

For a client that wants to read the secret from the Vault, the read capability should be enabled. We will be creating a policy in the Vault to read the path of the secret i.e internal/data/database/clops-app.

We then need a role on the policy with the bounded service account and namespace.

Write out the policy named internal-app that enables the read capability for secrets at the path internal/data/database/creol-api.

Create a Kubernetes authentication role named internal-app

vault policy write internal-app – <<EOF

path “internal/data/database/clops-app” {

capabilities = [“read”]

}

EOF

vault write auth/kubernetes/role/internal-app

bound_service_account_names=internal-app

bound_service_account_namespaces=apis

policies=internal-app

ttl=24h

The role connects the Kubernetes service account, internal-app, and namespace, with the Vault policy, internal-app. The tokens returned after authentication are valid for 24 hours (we can decide the TTL based on our use case)

Define a Kubernetes service account

The Vault Kubernetes authentication role defined a Kubernetes service account named internal-app.

• kubectl -n apis create sa internal-app

Launch an application

Here is the deployment file

apiVersion: apps/v1

kind: Deployment

metadata:

name: clops-app

namespace: apis

labels:

app: clops-app

spec:

replicas: 1

selector:

matchLabels:

app: clops-app

template:

metadata:

annotations:

vault.hashicorp.com/agent-inject: ‘true’

vault.hashicorp.com/agent-inject-status: ‘update’

vault.hashicorp.com/role: ‘internal-app’

vault.hashicorp.com/agent-inject-secret-database-creol-api.env: ‘internal/data/database/clops-appi’

vault.hashicorp.com/agent-inject-template-database-clopos-app.env: |

{{- with secret “internal/data/database/clops-app” -}}

export SPRING_DATA_MONGODB_URI=”mongodb+srv://{{ .Data.data.DATABASE_USERNAME }}:{{ .Data.data.DATABASE_PASSWORD }}@{{ .Data.data.DATABASE_HOST }}”

{{- end -}}

labels:

app: clops-app

spec:

serviceAccountName: internal-app

containers:

– name: clops-app

image: registry.gitlab.com/cloudifyops/engineering/clops-app:3.1.21

imagePullPolicy: Always

command: [“/bin/sh”]

args:

[‘-c’, ‘source /vault/secrets/database-clops-app.env && env > /vault/secrets/testenv && java “$MAX_HEAP_SIZE” -jar /opt/clops-app/clops-app.jar’]

ports:

– name: clopos-app

containerPort: 7400

env:

– name: AWS_ACCESS_KEY_ID

valueFrom:

secretKeyRef:

name: clops-app

key: AWS_ACCESS_KEY_ID

– name: AWS_SECRET_ACCESS_KEY

valueFrom:

secretKeyRef:

name: clops-app

key: AWS_SECRET_ACCESS_KEY

– name: GIT_ACCESSTOKEN

valueFrom:

secretKeyRef:

name: clops-app

key: GIT_ACCESSTOKEN

volumeMounts:

– name: logs

mountPath: /var/log

– name: promtail-container

image: grafana/promtail

args:

– -config.file=/etc/promtail/promtail.yaml

volumeMounts:

– name: logs

mountPath: /var/log

– name: promtail-config

mountPath: /etc/promtail

volumes:

– name: logs

emptyDir: {}

– name: promtail-config

configMap:

name: promtail-config-creol-api

imagePullSecrets:

– name: gitlab-auth

Vault.hashicorp annotation will inject the secret into the pod in /vaults/secrets path.

We should source the secret file from there and start the jar file.

The logs of the apis

We can see that connection with mongodb has been established and jvm is started.

Bonus tip

If you want to store files that have some sensitive information in the Vault, that can also be stored in KV Engen.

Let’s say in promtail config file we were having the uid and password of the loki database so we need to store that file in the same Vault database path.

vault kv put internal/database/clops-app DATABASE_USERNAME=”clops” DATABASE_PASSWORD=”random_passord” DATABASE_HOST=”database_endpoint” PROMTAIL=@promtail.yaml

Follow us on our LinkedIn Page. To know more about our services, visit our website.