By Adam Hawkins, IOD Expert
I’ve worked in tech for the last 10 years, mainly building, deploying, and running backend systems. I started out building PHP web services. Next, thankfully, came Ruby on Rails. I took the plunge into single page applications with Ember.js a few years later, but soon went running back to backend and service work.
The company I previously worked for went all in on Docker shortly after version 1.0, so the team and I experienced all the changes that went along with it. My last big project at the company was replacing our hand-rolled Docker orchestration system with Kubernetes. It was a huge challenge, but an impactful learning experience. Now, I want to pass some of that knowledge onto you by sharing my story of adopting, developing, and running production containers.
My adoption story starts with a confluence of two factors: first, internal reorganization and a ground-up product redesign; second, Docker hits 1.0. We decided to go with Docker because it supported our polygot system and standardized operational tooling. We rolled out the new system to production nine months later. Creating the deployment infrastructure was the most difficult part since there were no available orchestration systems. Around this time, the Docker community mainly focused on improving development workflows and not yet on production operations. Our setup had many warts, but we learned to live and work around them. The infrastructure ran well enough for two years until things changed.
What changed was a need for a less expensive system. However, no one on the team was confident enough to hack on our custom infrastructure. Luckily for us, there were more options available at that time. Container orchestration technology like Kubernetes, Docker Swarm, and Mesos were now viable production. We undertook a major engineering effort to build a Helm chart build pipeline on Kubernetes. Our system rolled out to production six months later. Decommissioning the old system was a huge accomplishment, but not without fault.
Nothing in this story is without fault. In fact, there were massive faults, engineering compromises, successes, and outright fails. However, some decisions were outstanding, and the resulting systems still run well in production. My intention, though, is not to convince you of anything. Instead, I simply want to share an honest retrospective on the journey of adopting containers since Docker 1.0 and how close or far off the mark it was. I’ll break my points up into three areas: adoption, workflows, and production operations.
Adoption
Hindsight shows we chose Docker for the right reasons at the wrong time. Same goes for our move to Kubernetes: right reasons, wrong time. We did not do adequate research into the supporting tools for either. Sure Docker and Kubernetes worked well enough, but we failed to consider all the upstream and downstream technologies they’d need to interact with along the way. Early adopters must carefully consider these trade-offs.
I’d been an early adopter before so I had some expectations. Adopting and working with Docker stands between adopting Rails and Ember.js. Adopting Rails was a new approach to the CRUD application world. Adopting Ember.js required a new approach to a new single page application paradigm. That was much more difficult. Adopting Docker required a new strategy since it created a new paradigm.
I would definitely not repeat the Kubernetes decision without better due diligence next time. I consider it best to wait until there is a supporting cast around the core technology. Consider the current conditions. There are tools like Docker Compose, Docker Machine, Kubernetes, Helm, and hundreds of other tested tools. If you’re an early adopter, then you’ll need to create and battle test those tools yourself. Ask yourself: do you want to put in the extra effort? And really, is that what’s actually needed?
Development
Adopting containers changed my development workflow more than anything since discovering TDD. It was a surprise then, but no surprise in hindsight. The mental shift came when I started doing all development inside containers. This was required because different services use different languages and tools.
Containerizing the development process created marked improvements across the development workflow. I was able to apply TDD and high impact black box smoke tests to any project/framework/language. I had a way to put projects through an entirely new battery of tests. Now I could start the server in one container and a test client in another. Many production regressions disappeared as my focus shifted from testing code to testing processes.
The shift happened over the course of a year because I completely adopted containers and continually brought them into more and more areas. It did not happen in teams that only built images at the end of a build pipeline. So if you’re just coming to containers, then dive right in–and strive to–create a fully containerized workflow. Ask yourself: how can I containerize this? This question teases out software and workflow assumptions. Containerize, Containerize, Containerize!
IOD is a content creation and research company working with some of the top names in IT. Our philosophy is experts are not writers and writers are not experts, so we pair tech experts with experienced editors to produce high quality, deeply technical content. The author of this post is one of our top experts. You can be too! JOIN US.
Production Operations
Containers have held up their core value promise over time in my experience. Containers have made it drastically easier to put a myriad of different applications, languages, and frameworks into production with less effort than before. They’ve succeeded so well, that I cannot imagine a production environment without containers. Kubernetes is becoming the standard platform for distributed applications. The big three cloud providers offer various hosted orchestration tools. It’s wonderful to see the focus shift from solved pre-production workflow problems to production problems. Things are getting better each and every day.
That said, it’s still the early days for running containers in production on these tools–a matter of a few years. Compare containers to the JVM and you’ll see what I’m talking about. My advice is to assume the technologies will work for you, but try to refute your assumptions with prototypes and small-scale projects. Running containers in production using orchestration requires understanding and adapting to distributed systems whether you like it or not. Be sure you’re ready for paradigm shift.
Looking Forward
Deciding to go all-in with containers ranks with my top career decisions. I’m happy because I learned an immensely powerful upcoming technology and am more prepared now to research, choose, and implement the next immensely powerful upcoming technology. I’m definitely going conversative on the next up and coming technology. That’s not because of technology, but because I’m guessing I’ll have other work to do. Being an early adopter is fantastic if you have infinite time, less so if you’re under the gun.
I only see sunshine on the horizon. Deployment and pipeline management tools like Helm or Spinnaker are the next wave. I’m keen to let those wave settle before jumping back in. There’s no rush for me. Their and my time will come and we’ll both be better for it. I’m sure their maturity will empower a new generation of teams to experience continuous deployment and polygot environments with less pain. That pushes more teams towards a DevOps value stream which is where the real value lies. They did for me, and can for you as well. So good luck out there and happy shipping!
= = =
Adam is backend/service engineer turned deployment and infrastructure engineer. His passion is building rock solid services and equally powerful deployment pipelines. He has been working with Docker for years and leads the SRE team at Saltside. Outside of work he’s a traveller, beach bum, and trance addict.
