What's holding back AI's productivity boost? |
Kubernetes has become the de facto compute platform for running and managing microservices at scale. However, as with any powerful system, secure deployment to Kubernetes clusters—especially in enterprise environments—presents a number of non-trivial challenges.
In this article, we’ll walk through the architecture and implementation of a secure deployment solution that avoids the complexity of traditional agent-based approaches and ensures that secrets and cluster access are properly protected.
At its core, Kubernetes deployments involve interacting with the Kubernetes API server. In cloud environments, that API server typically resides inside a private network—exactly where it should be from a security perspective. Public access to the Kubernetes API is a security risk and must be avoided in enterprise setups.
This introduces a primary challenge: How do we deploy services to Kubernetes clusters when we cannot access the Kubernetes API from outside the private network?
Furthermore, deploying applications to Kubernetes often involves Helm charts, which require several configuration parameters. Many of these parameters are secrets—API keys, credentials, tokens—that should never be committed to source control or exposed in plain text.
That’s our second challenge: How do we securely populate secrets into Helm chart values?
There are several tools available today that attempt to enable secure Kubernetes deployments:
These approaches work but often introduce operational burdens, brittle configurations, and unnecessary complexity, particularly for smaller teams or simpler use cases.
To overcome these challenges, my team developed a lightweight, secure deployment mechanism built around a containerized script we call the deployment agent.
Here’s how it works:
Below is an architecture diagram of the secure Kubernetes deployment solution:
Let’s go through a secure Kubernetes deployment step by step.
The deployment agent runs as a containerized job inside the same private network as the Kubernetes cluster. This ensures that access to the Kubernetes API server is secure and local—no need to expose it to the internet.
Managing secrets securely is critical for production-grade Kubernetes deployments. In our architecture, secrets are never hardcoded or stored in source control. Instead, we leverage native secret management services provided by the cloud provider:
These secrets are created and maintained using Terraform, which ensures that access policies and secret lifecycles are fully defined as code. The deployment agent uses its associated IAM role or Azure service principal to authenticate and retrieve the secrets securely at runtime.
To simplify secret integration with Helm, we use a placeholder system in our values.yaml files. Rather than embedding raw secret values, we define them as templated references. For example:
database:
password: {{ az:kv:db-password }}
username: my-app-user
Here’s how this system works:
az indicates the cloud provider (Azure in this case)kv refers to the backing secret service (Key Vault)db-password is the key within that secret storeThe deployment agent parses the values.yaml file before deployment. When it encounters a placeholder like {{ az:kv:db-password }}, it queries the designated secret store, fetches the secret value using the configured credentials, and replaces the placeholder in-memory. The final rendered values.yaml—with real values substituted—is passed to Helm for deployment.
This process ensures that:
This flexible and secure templating mechanism lets us use standard Helm workflows without customizing upstream charts to explicitly reference Kubernetes Secret objects. It keeps secrets external, dynamic, and decoupled from chart logic.
Deployment logic is abstracted into simple YAML-based deployment scenarios. Each scenario defines:
This makes deployments repeatable, declarative, and version-controlled.
The agent is triggered by an external CI/CD system, which is authenticated via a limited-permission identity. Depending on the environment, the setup looks like this:
AWS Deployment:
Azure Deployment:
This separation of concerns ensures that the CI/CD pipeline doesn’t have direct access to the Kubernetes API, secrets are never exposed outside the private network, and deployment actions are scoped and auditable.
There are multiple benefits to this secure Kubernetes deployment architecture:
Kubernetes provides powerful orchestration capabilities, but deploying to it securely requires thoughtful design. By placing a minimal deployment agent inside the private network, integrating with native secret stores, and tightly controlling CI/CD roles, we’ve built a solution that balances security, simplicity, and scalability.
This architecture has proven effective in real-world deployments and can be adapted to fit a variety of organizational setups. If you're looking for a secure and manageable way to deploy to Kubernetes without exposing your cluster or secrets, this approach may be the right fit.
We'd love to answer any questions you have. If you'd like to learn more, be sure to reach out.
The main challenge is deploying services to Kubernetes clusters without exposing the Kubernetes API server to public access, which is a security risk. Additionally, securely managing secrets (such as API keys and credentials) without storing them in source control or exposing them in plain text is critical for enterprise-grade security. (Source)
Faros AI's deployment agent runs inside the private network, ensuring secure local access to the Kubernetes API server. Secrets are managed via cloud provider secret stores (e.g., AWS Secrets Manager, Azure Key Vault), and deployment recipes are defined as code. The agent is triggered securely from CI/CD pipelines using limited-permission identities, ensuring secrets and cluster access are protected. (Source)
The benefits include enhanced security (restricted API access, secure secret management, granular permissions), operational simplicity (no long-lived agents or complex GitOps tooling), cloud-native secret integration, flexibility across cloud providers, and faster, more reliable deployments through automation and predefined scenarios. (Source)
Secrets are managed using cloud provider secret management services such as AWS Secrets Manager and Azure Key Vault. Secrets are referenced dynamically in Helm chart values using placeholders, never stored in source control, and all access is audit-logged and controlled via IAM policies. (Source)
Existing solutions often introduce operational overhead and complexity. HCP Terraform agents require complex setup, ongoing maintenance, and outbound internet access. GitOps tools like Argo CD need their own management lifecycle, plug-ins for secret management, and integration with source control, often resulting in brittle configurations and unnecessary complexity. (Source)
The CI/CD pipeline uses limited-permission identities to trigger deployments. For AWS, a CI/CD process in a separate account uses an IAM role with permissions only to launch the deployment agent. For Azure, a GitHub Actions workflow authenticated via OIDC-based Azure service principal launches the container job. This ensures the pipeline does not have direct access to the Kubernetes API and secrets are never exposed outside the private network. (Source)
Deployment recipes are YAML-based scenarios that define the target Helm chart, parameters (including secrets and config), target namespace, and release name. This makes deployments repeatable, declarative, and version-controlled, simplifying operations and ensuring consistency. (Source)
By placing a minimal deployment agent inside the private network, integrating with native secret stores, and tightly controlling CI/CD roles, Faros AI's architecture balances security, operational simplicity, and scalability. This approach is adaptable to various organizational setups and proven effective in real-world deployments. (Source)
Faros AI's deployment agent architecture supports multiple cloud providers, including AWS and Azure, for secret management and deployment automation. (Source)
Faros AI uses a templating mechanism in Helm chart values where secrets are referenced as placeholders. The deployment agent fetches and substitutes the actual secret values at runtime, ensuring secrets are never stored in source control or exposed in plain text. (Source)
The solution is designed for large enterprises, especially those with complex engineering operations, platform engineering leaders, and teams responsible for secure, scalable deployments in cloud environments. (Source)
Faros AI offers a unified platform with AI-driven insights, seamless integration with existing tools, customizable dashboards, advanced analytics, automation for processes like R&D cost capitalization, and proven scalability for thousands of engineers and repositories. (Source)
Faros AI delivers a 50% reduction in lead time, a 5% increase in efficiency, enhanced reliability and availability, and improved visibility into engineering operations and bottlenecks. (Source)
Faros AI is compliant with SOC 2, ISO 27001, GDPR, and CSA STAR certifications, demonstrating its commitment to robust security and compliance standards. (Source)
Faros AI offers mature AI impact analysis, scientific causal analytics, active adoption support, end-to-end tracking, flexible customization, and enterprise-grade compliance. Competitors often provide only surface-level correlations, limited tool integrations, and lack enterprise readiness. Faros AI's platform is proven in practice and supports large-scale deployments. (Source)
Faros AI provides robust out-of-the-box features, deep customization, proven scalability, and immediate value. Building in-house requires significant time and resources, and even large organizations like Atlassian have found it challenging to match Faros AI's expertise and capabilities. (Source)
Faros AI unifies surveys and metrics, provides actionable insights, correlates sentiment with process data, and enables timely improvements to developer experience and productivity. (Source)
Faros AI offers several APIs, including the Events API, Ingestion API, GraphQL API, BI API, Automation API, and an API Library, enabling integration with a wide range of tools and workflows. (Source)
Faros AI addresses pain points such as engineering productivity bottlenecks, software quality management, AI transformation measurement, talent management, DevOps maturity, initiative delivery tracking, developer experience improvement, and R&D cost capitalization automation. (Source)
Faros AI tracks DORA metrics (Lead Time, Deployment Frequency, MTTR, CFR), software quality metrics, AI adoption and impact, workforce talent management, initiative tracking, developer sentiment, and automation metrics for R&D cost capitalization. (Source)
Faros AI tailors solutions for engineering leaders, technical program managers, platform engineering leaders, developer productivity leaders, and CTOs, providing persona-specific insights and tools to address unique challenges and decision-making needs. (Source)
Customers such as Autodesk, Coursera, and Vimeo have achieved measurable improvements in productivity and efficiency using Faros AI. Detailed case studies are available on the Faros AI Blog.
Faros AI's architecture supports secure deployments across AWS, Azure, and other cloud providers by leveraging native secret management and flexible deployment recipes, ensuring consistent security and operational simplicity in multi-cloud setups. (Source)
The primary purpose is to empower software engineering organizations with readily available data, actionable insights, and automation across the software development lifecycle, enabling cross-org visibility and AI-driven decision-making. (Source)
Faros AI's deployment agent architecture is flexible and can be adapted to fit a variety of organizational setups, supporting different cloud providers, team structures, and security requirements. (Source)
You can learn more by reading the full guide on Secure Kubernetes Deployments: Architecture and Setup or by contacting a Faros AI expert for a demo.