From Monorepos to Google Kubernetes Engine

A full walkthrough of how I built a pipeline that deploys an app from a monorepo to GKE

Monorepos

Monorepos are great when working with lots of repos that rely on each other in some way or another. They solve a lot of problems that microservices bring along with them, such as

1. allowing sharing of code through libraries across all services, and making it a lot easier to upgrade or change libraries across the whole repo — try upgrading your logging library across all your services with polyrepos and see how laborious it can be;
2. making it easier to refactor code across multiple services;
3. making it easier to search through the whole code base.

I highly recommend doing some research into why and when monorepos are suitable for your needs — https://danluu.com/monorepo/

For personal projects it’s probably not so simple to argue why a monorepo is the way forward, but nonetheless that’s what I did. Apart from the above reasons I also wanted:

1. a way to quickly and easily deploy my small projects into the wild;
2. to get a better idea of monorepos (their pros and cons) first hand.

Build Process

Possibly the most difficult part of working with a monorepos is the build process. In polyrepos you can wire up pipelines that work well with that repo, but with monorepos we need a single method of deploying all our services and libraries. The one build tool that I’ve heard about the most is Bazel, which is Google’s open sourced version of their internal build tool. I think Bazel is for another day when I want to experiment with it after getting a better feel for monorepos. Bazel is just a build tool, and it’s not only for monorepos, but it does work very well with them.

I’ve forked off https://github.com/labs42io/circleci-monorepo to build my monorepo. It works with CircleCI and it does exactly what I need it to do for now — build go services from a monorepo.

Walkthrough

So here are all the steps I took to get a service from a monorepo into GKE. It’s going to be a bit dry and to the point, and if you want to follow along it will require some reading in the linked posts/articles:

1. Fork https://github.com/labs42io/circleci-monorepo
2. Change the name of the repo to Godzilla — to help signify the awesome size and power 🙃
3. Log into CircleCI with your github account
4. Setup a CircleCI job for the new monorepo
5. Add a personal token to the repo
   - https://circleci.com/docs/2.0/managing-api-tokens/#creating-a-personal-api-token
6. Add a CIRCLE_TOKEN to the new project
   - https://circleci.com/docs/2.0/env-vars/#setting-an-environment-variable-in-a-project
7. Change to using circleci/golang:1.14.3 to build the service
   - https://github.com/ankur22/godzilla/blob/master/.circleci/config.yml#L24
8. Add a linter step
   - https://golangci-lint.run/usage/install/ and https://golangci-lint.run/usage/quick-start/
9. Build static binary and disable cgo so that we can run the service in a barebones Docker container
   - https://medium.com/@diogok/on-golang-static-binaries-cross-compiling-and-plugins-1aed33499671
10. Create a Dockerfile which has a distroless base image, copies the binary, and defines the entry point for the app
    - https://github.com/GoogleContainerTools/distroless
    - https://medium.com/better-programming/how-to-harden-your-containers-with-distroless-docker-images-c2abd7c71fdb
11. Create a version file in the service that will be cat’d when creating a tag for the docker image
    - https://github.com/ankur22/godzilla/blob/master/packages/messenger-server/version.txt
12. Initially I wanted to work with Docker Hub
    - https://circleci.com/blog/build-cicd-piplines-using-docker/
    - https://circleci.com/docs/2.0/building-docker-images/ (use machine and don’t need a executor — https://github.com/ankur22/godzilla/blob/master/.circleci/config.yml#L164)
13. Once published to Docker hub, you should be able to pull and run it locally with:
    - docker pull <path to docker hub repo>
    - docker images | grep <name of image>
    - docker run <hash of image>
14. Create a hello world server, that can shut down and responds to GET requests
    - https://github.com/ankur22/godzilla/tree/master/packages/messenger-server
15. Once created and deployed run the following in the terminal:
    - docker pull ankura22/messenger-server:0.3.0–196
    - Find the image hash
    - docker run -p 127.0.0.1:8080:8081/tcp — env PORT=8080 0e72fae04e3c
    - curl localhost:8080
16. Deploy and run in Google Cloud
    - https://circleci.com/blog/simplifying-your-ci-cd-build-pipeline-to-gke-with-circleci-orbs/
    Use v1 instead of v1beta1 in the deployment yaml
    - https://cloud.google.com/container-registry/docs/pushing-and-pulling?_ga=2.83778515.-455075256.1582131774
    - https://circleci.com/orbs/registry/orb/circleci/gcp-gke#commands-rollout-image
    - Update to the latest orbs
    - To find the container name — kubectl get --output=wide deploy
    - Reduce the node count (defaults to 3) https://cloud.google.com/kubernetes-engine/docs/how-to/resizing-a-cluster
17. I changed circle_trigger.sh script to deploy the “latest” tagged docker image and deploy to k8s engine when merged into master
    - https://github.com/ankur22/godzilla/blob/master/.circleci/circle_trigger.sh
18. Using code coverage and codecov (this doesn’t work well with monorepos, but at least there’s another badge for the README):
    - https://circleci.com/docs/2.0/code-coverage/#using-a-code-coverage-service
    - https://circleci.com/orbs/registry/orb/codecov/codecov
    - https://docs.codecov.io/docs/flags
    - https://docs.codecov.io/docs/carryforward-flags
19. To get TLS to work I needed to work with helm charts
    - https://medium.com/bluekiri/deploy-a-nginx-ingress-and-a-certitificate-manager-controller-on-gke-using-helm-3-8e2802b979ec
20. To create the service file
    - https://cloud.google.com/kubernetes-engine/docs/tutorials/configuring-domain-name-static-ip
21. DNS:
    - https://cloud.google.com/dns/docs/quickstart#create_a_new_record
    - https://cloud.google.com/dns/docs/update-name-servers?authuser=1#console
22. TLS:
    - https://medium.com/bluekiri/deploy-a-nginx-ingress-and-a-certitificate-manager-controller-on-gke-using-helm-3-8e2802b979ec
    (Ensure Issuer is letsencrypt-prod not letsencrypt-production)
    - https://github.com/jetstack/cert-manager/issues/2393
    https://docs.cert-manager.io/en/release-0.11/reference/issuers.html
23. For wildcard you need to also follow:
    - https://cert-manager.io/docs/tutorials/acme/dns-validation/
    - https://cert-manager.io/docs/configuration/acme/dns01/google/
    - https://cert-manager.io/docs/release-notes/release-notes-0.3/#acmev2-and-let-s-encrypt-wildcard-certificates
    - https://john2x.com/blog/wildcard-certs-from-lets-encrypt-cert-manager-ingress-nginx-gke.html
    - https://andrewtchin.com/posts/2019/12/29/kubernetes-3.html — helped with kubectl commands
    - https://github.com/jetstack/cert-manager/issues/1557 — kubectl describe orders — all-namespaces helped me find an issue with wildcard certs
    secrets -> certs -> certrequest -> posts -> challenge -> ingress
24. Order of commands to verify wildcard cert issuance
    - kubectl describe issuer; kubectl describe secrets
    - kubectl describe certificates
    - kubectl describe certificaterequest
    - kubectl describe orders --all-namespaces
    - kubectl describe challenges --all-namespaces

The Monorepo

Here’s a link to the github page for my monorepo: https://github.com/ankur22/godzilla/