Kubernetes: your base setup towards container orchestration

December 16, 2016 in #docker #ansible #kubernetes #k8s #orchestrator #monitoring #logging | | | Share on Google+

The leading open source container orchestrator solution, with all necessary tools for success

Your takeaways from this blog

  • Get a full open source orchestrator solution to manage your containers
  • Easy deploy, hot version migration with kargo project
  • Understand and get running kubernetes and its monitoring and logging free tools
  • Be ready to build on top and deploy your own applications

Who should keep reading this post?

  • Devops and system infrastructure engineers, who are looking for solutions to manage 10+ containers on linux
  • IT engineers evaluating competitors like docker-swarm, mesos, nomad, openshift...
  • Anyone who loves containers and powerfull open source solutions!

Why kubernetes?

When you start working with more than 10 containers, you realize that managing each container on each server is not very optimal:

  • Containers usually don't use all server resources, and if a peak occurs, they are physically stuck on the same hardware
  • If your server crashes, your containers are not restarted on another host
  • Many manual steps are needed if you want to scale up/down similar containers, on multiple hosts

Kubernetes brings solutions to these problems with the following characteristics:

  • Portable: public, private, hybrid, multi-cloud
  • Extensible: modular, pluggable, hookable, composable
  • Self-healing: auto-placement, auto-restart, auto-replication, auto-scaling

Even if other solutions exist (docker swarm, mesos, nomad ...), kubernetes (or k8s) - an all star github google opensource project - is becoming the reference for scheduling and managing containers. Many companies which were already proposing their own orchestration solutions, are now integrating kubernetes as their container engine.

As a serious contender, we can cite for example openshift. This redhat product is leveraging kubernetes engine framework with additional layers to get to a full paas. While winning on ease of use (GUI to pilot the cluster, auto-deploy from registry), it is loosing on flexibility (slower kubernetes updates, integration with other solutions) and community.

Kubernetes alone is then very versatile and powerful. However, it requires a learning curve to understand the concepts and get it running. This post will help you get a base working setup, where you can start running your own containers on top.

More info on k8s: http://kubernetes.io/

Architecture and components of our base setup

This schema explains how it all works together. Let's break down each component:

  • Kargo: this project use ansible recipe to deploy and migrate kubernetes version on many types of servers (coreos, ubuntu, centos) and cloud (aws, azure, openstack, baremetal). So the choice is yours. In my doc, I will just deploy coreos servers manually, and give ip info to kargo. But you could use kargo to provision aws or google servers in 1 command line.

  • 1 master node running kubernetes components for container orchestration, it will pilot and gives work to the minions

  • 1 master/minion node: In order to have master redundancy and running containers too

  • 2(or more) minion nodes running the actual containers and doing the actual work

  • Coreos: this minimal and secure OS is perfect for running kubernetes masters and nodes.

  • Efk (logging): we will send all kubernetes container logs to an elasticsearch db, via fluentd, and visualize dashboards with kibana

  • Prometheus (monitoring) will check all this infra, with grafana dashboards

  • kubernetes dashboard addon (not efk dashboard), where you can visualize kubernetes component in a GUI

  • Service-loadbalancer: public gateway to access your internal kubernetes services (kibana, grafana). In the setup later, you will have 2 choices of lb, a static (haproxy) and a dynamic lb (traefik)

  • Registry: a private docker registry deployed in the kubernetes cluster

The heart of kubernetes machine

This schema represents kubernetes internal components after the kargo install. Nearly all of them are running as containers. You will be able to adjust the number of master, minion, etcd to fit your needs.

kubernetes-kargo

Whole status of your infra in few dashboards

Below is a visual preview of what you will get with this setup, using open source tools, which integrate perfectly with kubernetes.

Logging with efk:

First you will collect container logs with efk, so you can see who is very talkative, or which application is in pain and sending errors/timeout. Once setup, it is all automatic: any newly created containers will send logs to efk.

Monitoring with prometheus

Then we will dig deeper in the stats and counters (cpu, ram, disk, network), where we can investigation bottlenecks, memory leaks and plan for capacity management.
You can already enjoy the loaded detailed dashboards, and you can find a lot more online (thanks to the community!)

grafana-k8s-pod-resources1 grafana-k8s-pod-resources2 grafana-prometheus-stats

Ready? Get the code and deploy

Follow this github repo to get a full kubernetes stack running.

I added to this setup lots of explanations on how to launch services and access them. Plus an extra monitoring tool (heapster), a demo of gitlab CI/CD and some troubleshooting tips to fix some headaches you may have.

A previous setup for a deployment on cloudstack on exoscale is available here. Less flexible as you can't migrate kubernetes version (it doesn't use kargo), this version however takes care of firewalls.

Now you got a working setup, it is time to run your own containers in kubernetes.
Start building your yaml manifest file based on the examples provided. If you want to start from an existing docker-compose.yml file, use that fast/easy converter: kompose.

Some other cool kubernetes projects to try:

  • Kubernetes could run locally on your laptop with minikube
  • You can pilot it from your desktop (windows/mac) via a gui : skippbox
  • Even with your mobile phone: check cabin ;-)
December 16, 2016 in #docker #ansible #kubernetes #k8s #orchestrator #monitoring #logging | | | Share on Google+