GitOps by Courtesy of Jenkins-X

Gerd Aschemann

About me

  • Gerd Aschemann

  • Freelancer: Java, Groovy, Enterprise, CRM, CI/CD, Architecture, DDD, …​

  • Co-Organizer:

    • JUG Darmstadt

    • Javaland Conference

    • Open Source Conference Planner DukeCon

Agenda

  1. Recap: Kubernetes

  2. Helm Charts

  3. Dev vs. Ops → GitOps

  4. Jenkins X

Recap: Kubernetes (k8s)

k8s pods and services

K8S Building Blocks

  • Nodes

  • Pods based on Deployment(s)

  • Service(s)

  • Overlay Networks

  • Ingresses

  • Configmaps + Secrets

  • Volumes

  • Custom Resource Definitions (CRDs)

  • …​

Service Resource

Service Resource Definition
apiVersion: v1
kind: Service
metadata:
  name: my-service
...
spec:
  clusterIP: 10.19.255.213
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080

Deployment Resource

Deployment Resource Definition
apiVersion: apps/v1
kind: Deployment
...
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-service
...
    spec:
      containers:
      - image: hub.docker.io/ascheman/my-service:0.0.1
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /actuator/health
            port: 8080
...

kubectl

Apply the resource definition by calling

kubectl apply -f <myresources.yml>

kubectl for Ops

kubectl is the new ssh

(attributed to Kelsey Hightower)

kubectl in DevOps?

diag 9e53fa1202cd393037ef2975f99e90d8

Imperative + Environment specific

Problem 1

This is still an imperative definition of the Build + Deploy

Problem 2

Resource Definitions are environment specific

Solution

Wouldn’t a declarative way be better?

Approach

Tool to ensure desired state

Helm Charts to the Rescue

Solution 1

Helm maintains the state of a resource

Solution 2

Helm adds templating to resources

Helm file tree

Helm chart file tree
├── Chart.yaml
...
├── templates
│   ├── deployment.yaml
...
│   └── service.yaml
└── values.yaml

Helm Chart

Helm Chart file
apiVersion: v1
description: A Helm chart for Kubernetes
icon: https://raw.githubusercontent.com/jenkins-x/jenkins-x-platform/master/images/java.png
name: my-service
version: 0.1.0-SNAPSHOT

Helm Template(s)

Helm Template file
apiVersion: extensions/v1beta1
kind: Deployment
...
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ template "fullname" . }}
...
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
...
        livenessProbe:
          httpGet:
            path: {{ .Values.probePath }}
            port: {{ .Values.service.internalPort }}
          initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}

Helm Variables

Helm Variables file
...
probePath: /actuator/health
livenessProbe:
  initialDelaySeconds: 60
...

Running Helm

Install charts

helm install …​

Update charts

helm upgrade …​

Override variables per environment

helm install …​ -f myvalues.yaml

Composite Helm Charts

Helm charts can be composed of other Helm charts

Helm chart requirements
diff --git a/requirements.yaml b/requirements.yaml
@@ -6,9 +6,12 @@
 dependencies:
 - name: my-old-service
   repository: http://jenkins-x-chartmuseum:8080
   version: 0.7.3
+- name: my-new-service
+  repository: http://jenkins-x-chartmuseum:8080
+  version: 0.0.1

Helm for Environments

  • Compose your application from different charts

    • Use existing charts (Public Helm repositories)

    • Provide your own

  • Add configuration (aka value files)

→ Store environment composition in a Git repository!

Helm features

Helm is the Homebrew of k8s
  • Helm is a package manager

  • Helm Charts contain (Go) templates for k8s objects

  • Helm Charts contain (default) values for templates

  • Charts are versioned

  • Charts refer to Docker images

  • Charts are stored in repositories (Chart Museum)

  • Charts can depend on other Charts (Composition)

  • Deployments replace variables w/, e.g., environment specific values

→ Roll out / keep the desired state

Helm in DevOps?

diag 338c2841acc6dba169629595bc866567

Credential issues

Problem

The pipeline executor needs administrative privileges

Solution

Raise a wall between Dev and Ops!

Approach

Use Git + Pipeline to implement (versioned) changes

A wall? Come on …​

No one has any intention of building a wall (Walter Ulbricht, 1961)

GitOps

diag 8bd2dd25403a4cb54bc606f9ee970afa

Git flow: Checkout → Branch → PR → Test → Merge

GitOps Advantages

Improve Security
  • Separation of credentials

  • Git security

  • Compliance + Auditing

Increase Speed + Productivity

Implement Feedback + Control Loop

Improve Quality
  • Reduced MTR

  • Improved Stability + Reliability

→ Implement by Git + Pipeline Engine

GitOps with Jenkins(-X)

GitOps

GitOps: Processes + Environments

  • Build + Deploy

  • Combine

    • Infrastructure as Code

    • + (Opinionated) Deployment Workflow

  • Promote

    • To Staging (Default/Automatic)

    • To Production (on Demand/Manual)

    • To Preview (on PR/Branch)

    • To more environments if required (LuP, UAT, …​)

The JX Promise

  • JX is a full CI/CD GitOps stack

  • JX is (only) a command line tool

  • Set up k8s (optional)

  • Set up Jenkins / Tekton

  • Set up Everything

    • Tools

    • Repositories

    • Processes → GitOps

JX Overview

Overview

Start into DevOps

intro dev2jx

It is really simple!

intro ops simple

Use a Maven build

intro dev add maven

Maven Build Setup

  • How to write the pom.xml?

  • How to deploy the build artifact to the repository?

  • How to get the artifact to k8s?

  • How to set up a Maven repository?

Use a Docker registry

intro dev add docker

Docker Build Setup

  • How to write the Dockerfile?

  • Or use Maven, Skaffold, Jib, …​???

  • How to deploy Docker image to the registry?

  • How to get the image to k8s?

  • How to set up a Docker registry?

Use a Helm chart

intro dev add helm

Helm/k8s Setup (Test)

  • How to write the deployment configuration?

    • Deployment YAML?

    • Service YAML?

    • Ingress???

    • ConfigMap + Secrets?

    • …​

      → Helm Chart(s)

  • How to set up Chart repository?

  • How to set up Kubernetes?

Integration + Production

  • How to set up Integration or Production?

  • How to cope with

    • multiple (interdependent) applications?

    • different configurations?

    • other infrastructure (DBs, Messaging, UIs, …​)?

    • multiple Kubernetes clusters?

  • What is the driving entity?

Jenkins-X + Kubernetes

JX Architecture

Architecture

JX Architecture: Repositories

  • Github (External): Source Code Management

  • Nexus (in k8s): Maven Repository

  • Docker Registry (in k8s): Docker Images

  • Chartmuseum (in k8s): Deployment Configurations

Demo 0: Create JX

JX is an opinionated command line tool!
JX Create Cluster (GKE)
jx create cluster gke

or

JX Install Cluster (GKE)
jx install cluster gke

Check Platform

Check JX status
jx status
Check JX version
jx version

Kubernetes Providers

  • Google Kubernetes Engine (GKE)

  • local (Docker) machine: Minikube / Minishift

  • Amazon: Minikube, AWS native, Elastic Kubernetes Service (EKS)

  • IBM Kubernetes Services (IKS)

  • Azure Kubernetes Service (AKS)

  • Oracle Kubernetes Environment (OKE)???

  • OpenShift (Minikube)

  • On Premise (e.g. Hetzner Cloud w/ kubespray)

Demo 1: JX Create/Import App

JX Create Spring/Node/Go/…​
jx create spring

or

JX Import
# cd $dirWithApp
jx import
Checkout Spring Initializr: https://start.spring.io/

Check Application status

Check build activity
jx get activity
Get current applications (and environments)
jx get applications

Automatically Staged

Change to jx-staging environment automatically merged as PR:

Helm chart requirements
diff --git a/env/requirements.yaml b/env/requirements.yaml
@@ -6,9 +6,12 @@
 dependencies:
...
   name: exposecontroller
   version: 2.3.58
   repository: https://chartmuseum.build.cd.jenkins-x.io
   alias: cleanup
+- name: jx-demo-180906-1131
+  repository: http://jenkins-x-chartmuseum:8080
+  version: 0.0.1

And thereby promoted as new application in the env.

GitOps Overview (2nd)

GitOps

GitOps Steps

  • Promote deployments through opinionated workflow

  • Describe formally (e.g., Helm chart)

    • Single application

    • Environment composition

  • Cover application as well as configuration

  • File change request as pull request

  • Automatically merge (including tests) and roll out

  • Monitoring + Synchronisation of state

  • Roll back on demand

Demo 2: JX Promote

JX Promote (to production)
jx promote --version 0.0.1 --env production

JX Features/Roadmap

  • Client: OSX + Linux, Windows: In Progress

  • Dev + Build + Deployment: CLI, DevPod, Preview/Pull-Request → GitOps

  • k8s providers: GCP/GKE, AWS/EKS, Azure, MiniShift/MiniKube, OpenShift, Pivotal KS, IBM KS, Digital Ocean, Oracle

  • Tools: Helm, Skaffold, Prow, LetsEncrypt

  • Repositories: Chart Museum, Docker Registry, Nexus, Artifactory (Planned)

  • Issue Trackers: Github/GH Enterprise, Jira, Gitea (In Progress)

  • Multi-Cluster (In Progress)

  • Istio Canaries (In Progress)

Git providers

  • Working:

    • Github/Github Enterprise

    • Bitbucket Cloud

  • In Progress:

    • Gitea

    • Bitbucket Server

    • Gitlab

Build Packs

  • Working: C++, D, appserver, csharp, custom-jenkins, docker-helm, docker, dropwizard, environment, go, gradle, helm, javascript, jenkins, liberty, maven-java11, maven, php, python, ruby, rust, scala, swift, typescript

  • Planned: Kubeless, Serverless, .Net

Addons

  • Working: Prometheus, Gitea, Anchore, Theia IDE

  • Planned: Cloud9, Eclipse Che, SonarQube

State of the art: Jenkinsless Build

jx serverless
Figure 1. Jenkins-X without Jenkins
  • Prow: k8s reliable WebHook handler

    • Scalable / High Available

    • Configuration stored in Git(hub)

  • Tekton: Cloud Native Build Pipeline Engine

JX Overview (last)

Overview

Conclusion

  • CLI based setup: k8s, Jenkins, Nexus, Docker Registry, Chartmuseum

  • Misc. k8s providers

  • Fast evolving (Features, Repositories, Addons)

  • Strong support for GitOps

  • Open source/Community

  • Production ready?

The Developer Experience (Demo D)

intro devpod

Demo D1: JX DevPod / Sync

JX Sync (Run once)
jx sync
JX DevPod (Init / Run 1st)
jx create devpod --reuse --sync -l maven -s demo
# Then in the remote shell
cd $WORK_DIR
mvn spring-boot:run

Demo D2: Access DevPod (Default)

JX Port Forwarding
jx get devpod

ascheman-maven-demo maven 1h39m34s Ready

Get Devpod Ingress
# On localhost
kubectl get ingress
curl http://...-port-8080...

{"status":"UP"}

Demo D3: Access Exposed Service

Skaffold Deployment
# In devpod
./watch.sh
  • Builds and deploys application

  • Access via developer Ingress

Get Preview Ingress
# On localhost
kubectl get ingress -n jx-edit-ascheman
curl http://...jx-edit-ascheman...

Demo D4: Dev in DevPod

Add Spring DevTools to Dependencies
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <optional>true</optional>
        </dependency>
  • Restart mvn spring-boot:run

  • Then Open 2nd rsh and let Maven compile → Reload of Java code

JX DevPod Remote Shell / Login (2nd)
jx rsh -d
# Then in the remote shell
cd $WORK_DIR
mvn compile

Demo D5: Extend App

Add DemoController.java
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class DemoController {
    @GetMapping("/")
    public String getGreeting() {
        return "Hello World";
    }
}

Demo D6: Docker Build in DevPod

Docker build (via skaffold)
jx rsh -d
# Then in the remote shell
cd $WORK_DIR
VERSION=xxx skaffold build

Build / Push of Docker Image
Starting build...
Building [changeme]...
Sending build context to Docker daemon  17.55MB
Step 1/8 : FROM openjdk:8-jdk-slim
...
Build complete in 416.607605ms
changeme -> 10.31.249.194:5000/ascheman-jx/jx-demo-180811-1200:xxx

Thank You