When you start working with more than 10 containers, you realize that managing each container on each server is not very optimal:
Kubernetes brings solutions to these problems with the following characteristics:
Even if other solutions exist (docker swarm, mesos, nomad ...), kubernetes (or k8s) - an all star github google opensource project - is becoming the reference for scheduling and managing containers. Many companies which were already proposing their own orchestration solutions, are now integrating kubernetes as their container engine.
As a serious contender, we can cite for example openshift. This redhat product is leveraging kubernetes engine framework with additional layers to get to a full paas. While winning on ease of use (GUI to pilot the cluster, auto-deploy from registry), it is loosing on flexibility (slower kubernetes updates, integration with other solutions) and community.
Kubernetes alone is then very versatile and powerful. However, it requires a learning curve to understand the concepts and get it running. This post will help you get a base working setup, where you can start running your own containers on top.
More info on k8s: http://kubernetes.io/
This schema explains how it all works together. Let's break down each component:
Kargo: this project use ansible recipe to deploy and migrate kubernetes version on many types of servers (coreos, ubuntu, centos) and cloud (aws, azure, openstack, baremetal). So the choice is yours. In my doc, I will just deploy coreos servers manually, and give ip info to kargo. But you could use kargo to provision aws or google servers in 1 command line.
1 master node running kubernetes components for container orchestration, it will pilot and gives work to the minions
1 master/minion node: In order to have master redundancy and running containers too
2(or more) minion nodes running the actual containers and doing the actual work
Coreos: this minimal and secure OS is perfect for running kubernetes masters and nodes.
Efk (logging): we will send all kubernetes container logs to an elasticsearch db, via fluentd, and visualize dashboards with kibana
Prometheus (monitoring) will check all this infra, with grafana dashboards
kubernetes dashboard addon (not efk dashboard), where you can visualize kubernetes component in a GUI
Service-loadbalancer: public gateway to access your internal kubernetes services (kibana, grafana). In the setup later, you will have 2 choices of lb, a static (haproxy) and a dynamic lb (traefik)
Registry: a private docker registry deployed in the kubernetes cluster
This schema represents kubernetes internal components after the kargo install. Nearly all of them are running as containers. You will be able to adjust the number of master, minion, etcd to fit your needs.
Below is a visual preview of what you will get with this setup, using open source tools, which integrate perfectly with kubernetes.
Logging with efk:
First you will collect container logs with efk, so you can see who is very talkative, or which application is in pain and sending errors/timeout. Once setup, it is all automatic: any newly created containers will send logs to efk.
Monitoring with prometheus
Then we will dig deeper in the stats and counters (cpu, ram, disk, network), where we can investigation bottlenecks, memory leaks and plan for capacity management.
You can already enjoy the loaded detailed dashboards, and you can find a lot more online (thanks to the community!)
Follow this github repo to get a full kubernetes stack running.
I added to this setup lots of explanations on how to launch services and access them. Plus an extra monitoring tool (heapster), a demo of gitlab CI/CD and some troubleshooting tips to fix some headaches you may have.
A previous setup for a deployment on cloudstack on exoscale is available here. Less flexible as you can't migrate kubernetes version (it doesn't use kargo), this version however takes care of firewalls.
Now you got a working setup, it is time to run your own containers in kubernetes.
Start building your yaml manifest file based on the examples provided. If you want to start from an existing docker-compose.yml file, use that fast/easy converter: kompose.
Some other cool kubernetes projects to try: