Is GitOps still GitOps if you are not using a git repository as your deployment artifact? While git remains central to GitOps workflows, storing infrastructure definitions as Open Container Initiative (OCI) artifacts in container registries has seen a rise in adoption as the source for GitOps deployments. In this article, we will dive deeper into the ideas behind this trend and how GitLab features support this enhancement to GitOps workflows.
What is GitOps?
The OpenGitOps project has defined four principles for the practice of GitOps:
- A system managed by GitOps must have its desired state expressed declaratively.
- Desired state is stored in a way that enforces immutability and versioning, and retains a complete version history.
- Software agents automatically pull the desired state declarations from the source.
- Software agents continuously observe actual system state and attempt to apply the desired state.
An example of GitOps is storing the Kubernetes manifests for a microservice in a GitLab project. Those Kubernetes resources are then continuously reconciled by a controller running on the Kubernetes cluster where the microservice is deployed to. This allows engineers to manage infrastructure using the same workflows as working with regular code, such as opening merge requests to make and review changes and versioning changes. GitOps also has operational benefits such as preventing configuration drift and helps engineers audit what changes led to certain outcomes with deployments.
Benefits and limitations of git in GitOps workflows
While git is an essential piece of GitOps workflows, git repositories were not designed to be deployed by GitOps controllers. Git does provide the ability for engineers to collaborate on infrastructure changes and audit these changes later on, but controllers do not need to download an entire git repository for a successful deployment. GitOps controllers simply need the infrastructure defined for a particular environment.
Additionally, an important piece of the deployment process is to sign and verify deployments to assure deployment changes to an environment are coming from a trusted source. While git commits can be signed and verified by GitOps controllers, commits may also capture other details not related to the deployment itself (e.g., documentation changes, updates to other environments, and git repository restructuring) or not enough of the deployment picture as a deployment may consist of multiple commits. This again feels like a case this git feature wasn’t designed for.
Another challenging aspect of git in GitOps workflows is that it can sometimes lead to more automation than expected. Soon after merging a change to the watched branch, it will be deployed. There are no controls in the process outside of git. How can you make sure that nothing gets deployed on a Friday late afternoon? What if teams responsible for deployment do not have permissions to merge changes in certain GitLab projects? Using OCI images adds a pipeline into the process, including all the delivery control features, like approvals or deploy freezes.
OCI images
The Open Container Initiative has helped to define standards around container formats. While most engineers are familiar with building Dockerfiles into container images, many may not be as familiar with storing Kubernetes manifests in a container registry. Because GitLab’s Container Registry is OCI compliant, it allows for users to push Kubernetes manifests for a particular environment to a container registry. GitOps controllers, such as Flux CD, can use the manifests stored in this OCI artifact instead of needing to clone an entire git repository.
Often in GitOps workflows, a git repository can include the infrastructure definitions for all environments that a microservice will be deployed to. By packaging the Kubernetes manifests for only a specific environment, Flux CD can download the minimum files needed to carry out a deployment to a specific environment.
Security benefits of using OCI artifacts
As mentioned previously, signing and verifying the artifacts to be deployed to an environment adds an additional layer of security for software projects. After Kubernetes manifests are pushed to a container registry, a tool like Sigstore Cosign can be used to sign the OCI image with a private key that can be securely stored in a GitLab project as a CI/CD variable. Flux CD can then use a public key stored on a Kubernetes cluster to verify that a deployment is coming from a trusted source.
Using GitLab to push and sign OCI images
GitLab offers many features that help simplify the process of packaging, signing, and deploying OCI images. A common way to structure GitLab projects with GitOps workflows is to have separate GitLab projects for microservices’ code and a single infrastructure repository for all microservices. If an application is composed of n microservices, this would require having n +1 GitLab projects for an application.
The artifact produced by a code project is usually a container image that will be used to package the application. The infrastructure or delivery project will contain the Kubernetes manifests defining all the resources required to scale and serve traffic to each microservice. The artifact produced by this project is usually an OCI image used to deploy the application and other manifests to Kubernetes.
In this setup, separation of environments is handled by defining Kubernetes manifests in separate folders. These folders represent environments (e.g., development, staging, and production) that will host the application. When changes are made to the code project and a new container image is pushed, all that needs to be done to deploy these changes via GitLab’s integration with Flux CD is to edit the manifests under the environment folder to include the new image reference and open a merge request. Once that merge request is reviewed, approved, and merged, the delivery project’s CI/CD job will push a new OCI image that Flux CD will pick up and deploy to the new environment.
Signing an OCI image is as simple as including Cosign in your project’s CI/CD job. You can simply generate a new public and private key with Cosign by running the commands below locally. Just make sure to log in to your GitLab instance with the glab CLI and replace the [PROJECT_ID] for the Cosign command with your delivery project’s ID.
glab auth login
cosign generate-key-pair gitlab://[PROJECT_ID]
Once the cosign command runs successfully, you can see the Cosign keys added to your project under the CI/CD variables section under the key names COSIGN_PUBLIC_KEY and COSIGN_PRIVATE_KEY.
Example CI/CD job
A GitLab CI/CD job for pushing an OCI image will look something like the following:
frontend-deploy:
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
changes:
paths:
- manifests/dev/frontend-dev.yaml
trigger:
include:
- component: gitlab.com/components/fluxcd/oci-artifact@0.3.1
inputs:
version: 0.3.1
kubernetes_agent_reference: gitlab-da/projects/tanuki-bank/flux-config:dev
registry_image_url: "oci://$CI_REGISTRY_IMAGE/frontend"
image_tag: dev
manifest_path: ./manifests/dev/frontend-dev.yaml
flux_oci_repo_name: frontend
flux_oci_namespace_name: frontend-dev
signing_private_key: "$COSIGN_PRIVATE_KEY"
The GitLab CI/CD Catalog offers a GitLab-maintained CI/CD component for working with OCI artifacts and Flux CD. This component allows development teams to push Kubernetes manifests as OCI images to GitLab’s Container Registry or an external container registry, sign the OCI image using Cosign, and immediately reconcile the newly pushed image via Flux CD.
In the example above, the Flux CD component is included in a .gitlab-ci.yml file of a GitLab project. Using the component’s inputs, users can define what registry to push the image to (i.e., registry_image_url and image tag), the file path to Kubernetes manifests that will be pushed (i.e., manifest_path), the cosign private key used to sign images (i.e., signing_private_key), and the Kubernetes namespace and Flux CD OCIRepository name needed to sync updates to an environment (i.e., flux_oci_namespace_name and flux_oci_repo_name).
The kubernetes_agent_reference allows GitLab CI/CD jobs to inherit the kubeconfig needed to access a Kubernetes cluster without needing to store a kubeconfig CI/CD variable in each GitLab project. By setting up the GitLab agent for Kubernetes, you can configure all GitLab projects’ CI/CD jobs in a GitLab group to inherit permissions to deploy to the Kubernetes cluster.
The agent for Kubernetes context is typically configured wherever you configure the GitLab Agent for Kubernetes in your GitLab group. It is typically recommended that this be done in the project where Flux CD is managed. More information on configuring the agent for CI/CD access can be found in our CI/CD workflow documentation.
The variables $COSIGN_PRIVATE_KEY, $FLUX_OCI_REPO_NAME, and $FRONTEND_DEV_NAMESPACE are values stored as CI/CD variables to easily access and mask these sensitive pieces of data in CI/CD logs. The $CI_REGISTRY_IMAGE is a variable that GitLab jobs have available by default that specifies the GitLab project’s container registry.
Deploy OCI images
Using Flux CD with your GitLab projects, you can automate deployments and signing verification for your microservice’s environments. Once Flux CD is configured to sync from a GitLab project, you could add the following Kubernetes custom resource definitions to your project to sync your pushed OCI image.
apiVersion: v1
kind: Namespace
metadata:
name: frontend-dev
labels:
name: frontend-dev
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: cosign-public-key
namespace: frontend-dev
spec:
encryptedData:
cosign.pub: AgAKgLf4VbVzJOmr6++k81LlFayx88AELaUQFNOaXmBF4G+fBfBYeABl0skNvMAa1UrPVNSfMIHgFoYHoO96g576a+epk6V6glOI+++XvYbfsygof3GGxe0nL5Qh2b3ge0fNpyd0kTPSjTj0YUhRhKtMGMRSRw1jrwhNcGxCHK+Byibs52v8Np49KsIkeZKbzLdgYABkrv+k0j7hQM+jR180NpG+2UiRvaXpPuogxkbj61FEqWGrJHk8IVyfl3eh+YhoXxOHGDqko6SUC+bUZPDBlU6yKegO0/8Zq3hwulrSEsEjzRZNK+RFVMOLWWuC6h+WGpYhAMcsZPwjjJ/y29KLNa/YeqkN/cdk488QyEFc6ehCxzhH67HxIn2PDa+KkEOTv2TuycGF+Q00jKIizXF+IwLx/oRb3pTCF0AoAY8D8N3Ey+KfkOjsBON7gGID8GbQiJqX2IgIZxFMk0JRzxbRKOEqn+guLd5Shj7CD1a1Mkk0DxBdbqrGv2XNYUaFPI7xd3rZXUJZlnv+fsmwswsiGWRuXwim45HScWzQnfgLAe7tv3spVEGeaO5apl6d89uN21PBQnfE/zyugB//7ZW9tSp6+CSMyc5HynxI8diafqiwKPgvzLmVWRnkvxJijoXicRr3sCo5RudZPSlnjfd7CKdhwEVvLl7dRR4e/XBMdxCzk1p52Pl+3/kJR+LJii5+iwOpYrpVltSZdzc/3qRd19yMpc9PWpXYi7HxTb24EOQ25i21eDJY1ceplDN6bRtop2quzkjlwVeE2i4cEsX/YG8QBtQbop/3fjiAjKaED3QH3Ul0PECS9ARTScSkcOL3I00Xpp8DyD+xH0/i9wCBRDmH3yKX18C8VrMq02ALSnlP7WCVVjCPzubqKx2LPZRxK9EG0fylwv/vWQzTUUwfbPQZsd4c75bSTsTvxqp/UcFaXA==
template:
metadata:
name: cosign-public-key
namespace: frontend-dev
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: frontend
namespace: frontend-dev
spec:
interval: 1m
url: oci://registry.gitlab.com/gitlab-da/projects/tanuki-bank/tanuki-bank-delivery/frontend
ref:
tag: dev
verify:
provider: cosign
secretRef:
name: cosign-public-key
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: frontend
namespace: frontend-dev
spec:
interval: 1m
targetNamespace: frontend-dev
path: "."
sourceRef:
kind: OCIRepository
name: frontend
prune: true
The Kustomization resource allows for further customization of Kubernetes manifests and also specifies which namespace to deploy resources to. The OCIRepository resource for Flux CD allows users to specify the OCI image repository reference and tag to regularly sync from. Additionally, you will notice the verify.provider and verify.secretRef properties. These fields allow you to verify that the OCI image deployed to the cluster was signed by the corresponding Cosign private key used in the earlier CI/CD job.
The public key needs to be stored in a Kubernetes secret that will need to be present in the same namespace as the OCIRepository resource. To have this secret managed by Flux CD and not store the secret in plain text, you can consider using SealedSecrets to encrypt the value and have it be decrypted cluster side by a controller.
For a simpler approach not requiring SealedSecrets, you can deploy the secret via a GitLab CI/CD job using the kubectl CLI. In the non-sealed secret approach, you would simply remove the SealedSecret included above and run the job to deploy the public key secret before running the job to push the new OCI image. This will make sure the secret is stored securely in GitLab and make sure the secret can be accessed on the cluster by the OCIRepository. While this approach is a bit simpler, just note this is not a suitable approach for managing secrets in production.
The benefits of OCI, GitLab, and GitOps
OCI artifacts allow for GitOps teams to take deployments even further with added security benefits and allowing for deployments to be minimal. Users still gain all the benefits offered by git as far as having a source of truth for infrastructure and collaborating on projects. OCI images add a packaging approach that improves the deployment aspect of GitOps.
GitLab continues to learn from our customers and the cloud native community on building experiences that help simplify GitOps workflows. To get started using some of the features mentioned in this blog, you can sign up for a 60-day free trial of GitLab Ultimate. We would also love to hear from users about their experiences with these tools, and you can provide feedback in the community forum.
