How to achieve secure Kubernetes deployments in the enterprise environment

Kubernetes has become the de facto compute platform for running and managing microservices at scale. However, as with any powerful system, secure deployment to Kubernetes clusters—especially in enterprise environments—presents a number of non-trivial challenges.

In this article, we’ll walk through the architecture and implementation of a secure deployment solution that avoids the complexity of traditional agent-based approaches and ensures that secrets and cluster access are properly protected.

The challenge of secure Kubernetes deployments

At its core, Kubernetes deployments involve interacting with the Kubernetes API server. In cloud environments, that API server typically resides inside a private network—exactly where it should be from a security perspective. Public access to the Kubernetes API is a security risk and must be avoided in enterprise setups.

This introduces a primary challenge: How do we deploy services to Kubernetes clusters when we cannot access the Kubernetes API from outside the private network?

Furthermore, deploying applications to Kubernetes often involves Helm charts, which require several configuration parameters. Many of these parameters are secrets—API keys, credentials, tokens—that should never be committed to source control or exposed in plain text.

That’s our second challenge: How do we securely populate secrets into Helm chart values?

Existing solutions: Too much overhead

There are several tools available today that attempt to enable secure Kubernetes deployments:

• HCP Terraform agents: These agents run inside the private network and allow HCP Terraform (hosted on the public internet) to deploy resources securely. While effective, these agents require complex setup and ongoing maintenance. They also need outbound internet access and introduce additional moving parts.
  
• GitOps tools like Argo CD: Argo CD can be deployed inside the cluster to perform Helm-based deployments. However, it requires its own management lifecycle, plug-ins for secret management, and integration with source control. Helm secrets are usually stored in external Kubernetes secret objects, requiring chart customization or complex overlays.

These approaches work but often introduce operational burdens, brittle configurations, and unnecessary complexity, particularly for smaller teams or simpler use cases.

Novel solution: A lightweight deployment agent for secure Kubernetes deployments

To overcome these challenges, my team developed a lightweight, secure deployment mechanism built around a containerized script we call the deployment agent. 

Here’s how it works: 

1. It runs inside the private network. 
2. Secrets are managed via cloud provider secret stores.
3. Deployment recipes as code.
4. A secure trigger from the CI/CD pipeline.

Below is an architecture diagram of the secure Kubernetes deployment solution: 

Let’s go through a secure Kubernetes deployment step by step. 

1. The deployment agent runs inside the private network

The deployment agent runs as a containerized job inside the same private network as the Kubernetes cluster. This ensures that access to the Kubernetes API server is secure and local—no need to expose it to the internet.

2. Secrets managed via cloud provider secret stores

Managing secrets securely is critical for production-grade Kubernetes deployments. In our architecture, secrets are never hardcoded or stored in source control. Instead, we leverage native secret management services provided by the cloud provider:

• AWS Secrets Manager
• Azure Key Vault
• Other cloud-native alternatives

These secrets are created and maintained using Terraform, which ensures that access policies and secret lifecycles are fully defined as code. The deployment agent uses its associated IAM role or Azure service principal to authenticate and retrieve the secrets securely at runtime.

To simplify secret integration with Helm, we use a placeholder system in our values.yaml files. Rather than embedding raw secret values, we define them as templated references. For example:

database:

  password: {{ az:kv:db-password }}

  username: my-app-user

Here’s how this system works:

• az indicates the cloud provider (Azure in this case)
• kv refers to the backing secret service (Key Vault)
• db-password is the key within that secret store

The deployment agent parses the values.yaml file before deployment. When it encounters a placeholder like {{ az:kv:db-password }}, it queries the designated secret store, fetches the secret value using the configured credentials, and replaces the placeholder in-memory. The final rendered values.yaml—with real values substituted—is passed to Helm for deployment.

This process ensures that:

• Secrets never appear in source control
• Helm charts remain reusable and cloud-agnostic
• All secret access is audit-logged and controlled via IAM policies

This flexible and secure templating mechanism lets us use standard Helm workflows without customizing upstream charts to explicitly reference Kubernetes Secret objects. It keeps secrets external, dynamic, and decoupled from chart logic.

3. Deployment recipes as code

Deployment logic is abstracted into simple YAML-based deployment scenarios. Each scenario defines:

• The target Helm chart (stored in a private OCI registry)
• Parameters to apply (secrets and config)
• Target namespace and release name

This makes deployments repeatable, declarative, and version-controlled.

4. Secure trigger from the CI/CD pipeline

The agent is triggered by an external CI/CD system, which is authenticated via a limited-permission identity. Depending on the environment, the setup looks like this:

AWS Deployment:

• A CI/CD process running in a separate AWS account
• An IAM role with permissions only to launch the deployment agent in the target account

Azure Deployment:

• A GitHub Actions workflow authenticated via OIDC-based Azure service principal
• The service principal can only launch the container job in the target Azure subscription

This separation of concerns ensures that the CI/CD pipeline doesn’t have direct access to the Kubernetes API, secrets are never exposed outside the private network, and deployment actions are scoped and auditable.

Benefits of the deployment agent architecture

There are multiple benefits to this secure Kubernetes deployment architecture: 

• Enhanced security: By restricting API access, securely managing secrets with cloud providers, and employing granular permissions, we significantly reduce the attack surface.
• Operational simplicity: No long-lived agents or complex GitOps tooling. The lightweight nature of the deployment agent and the use of "deployment recipes" reduce the complexity often associated with agents and external tools.
• Cloud-native secret integration: Uses existing cloud infrastructure for secret management.
• Flexible: Supports AWS, Azure, and other cloud providers.
• Faster, More Reliable Deployments: Automation through the CI/CD pipeline and predefined scenarios ensures consistent and repeatable deployments.

A solution for enterprise Kubernetes deployment challenges

Kubernetes provides powerful orchestration capabilities, but deploying to it securely requires thoughtful design. By placing a minimal deployment agent inside the private network, integrating with native secret stores, and tightly controlling CI/CD roles, we’ve built a solution that balances security, simplicity, and scalability.

This architecture has proven effective in real-world deployments and can be adapted to fit a variety of organizational setups. If you're looking for a secure and manageable way to deploy to Kubernetes without exposing your cluster or secrets, this approach may be the right fit.

We'd love to answer any questions you have. If you'd like to learn more, be sure to reach out.