On May 2nd – 4th, 4 members of the Daysha team attended the first European Atlassian Summit in Barcelona. This blog is a collection of information gathered throughout the summit whether it be by attending various talks or speaking to other attendees.
The Daily Telegraph
This presentation was made at the Atlassian Summit conference in Barcelona May 3rd 2017 by Carol Johnson. of the Daily Telegraph.
What is immediately clear is how business culture drove IT to be lean. The Daily Telegraph has been around since 1850 but recently has won a series of firsts as a digital product and in doing so has had to adapt its content production and business models. This business culture made it easier for IT leadership to succeed as innovators on their DevOps journey.
The next most striking feature was the persistence and patience required to get to ‘what good looked like’. There were several cul-de-sacs and while they are still not complete one of the most significant and recent breakthroughs was empowered teams. Teams who owned the delivery of their code into production and its support through initial production release.
The newspaper business has been digitally disrupted since 2009. Back then at DT there was an enterprise focus from an op’s perspective which meant clunky change processes which led to poor communication with the business and a product backlog that was First In First Out … in effect chaotic. There was no sense of the cost or value of a feature but a reaction to the business owner shouting loudest. There was a one year backlog.
The First Digital Strategy – GIOT
This was to start moving some app’s towards a SaaS model. They also started moving VM’s into their own data center. The catch phrase at the time was GIOT – Get It Out There.
They built fast, reduced costs and used self healing and auto scaling/load balancing systems. However this resulted in snowflake configuration problems leading to fragility in applications. They experienced significant problems through poor test automation. Operations and service management bore the brunt of these failings.
The Second Digital Journey – More Engineers.
This led to more Dev capacity, mode products and faster delivery. There were more teams including offshore but there was still a hand over to Op’s in the process via ‘DevOps’. Op’s teams did not see any benefit from this approach. There were further challenges in the form of website speed. So how did they get to faster releases, and improved response time on their site.
One of the learnings from this phase was around solution architectures based on their 3 R’s design principle. So, when building and releasing software they design so that they can recover in the following sequence
1. Restart the app – this is the fastest
2. Reboot an instance – this is fast
3. Relaunch the service – this is slowest but fixes most things
This phase led to 25% more releases, 10% less failures on new features and 7% less downtime.
The Third Stage – Working Together.
This stage was the most arduous for all concerned and in the end they arrived at a good solution, somewhat by chance but mostly from good management. They felt they were still unclear on the meaning of DevOps. They had read the books and done the research but it still didn’t feel right and further change was needed. They looked at what other companies were doing and felt they were off.
In all earlier phases Dev and Ops were silo’d and this had to change if releases were to be delivered more quickly with higher quality and lower downtime. There was something of an accident of birth about how this finally came together.
Dev teams were working on a series of API projects which were described as POC but out of the blue they explained they would like to put these into production. This made Ops folk very nervous but Dev stood up to the plate and said they were prepared to see these through to production.
They agreed that the Dev who wrote the release notes would be on call. So this was the start of integrated teams. They had ‘DevOps’ guys (release automation) in phase 2 but they became a bottleneck so they started to assign a release lead to each Dev stream and this emerged as an optimal structure.
Today the DT teams say ‘we build, we support, we own’. Their rules are
- Fast build, fail and fix
- Self select tools and guilds
- Seamless and clean processes
Ops keep systems healthy and they rotate around the Dev teams. Ops folk are trained in the guilds (in house experts who train others to their own level so there are fewer points of failure). Op’s own logging, monitoring and security for example improve error messaging, scripting and in the future there is a thought that Ops will code.
Changing change control was a big job as there was a culture of clunkiness in how this was done but it did provide the business with a level of confidence in how production was progressing. Starting in 2009 the CAB took 8 days through a 4 step process to approve change. The next iteration was a twice weekly meeting with a 2 step approval process and it took 5 days on average. Now the CAB is daily and if its automated it doesn’t go through CAB otherwise there is a 1 step process which takes one day to pass/fail.
Today there are daily stand ups and feedback loops, fortnightly sprints and daily releases.
Carol’s parting advice
Integrate the team’s early – start with experiments, short feedback loops and fix what hurts. Don’t be afraid to change process but verify new process is working through strong metrical measurement.
Above all else empower teams – they want to be empowered more than you want them to be.