At the end of January 2020, we hosted the second in our series of DevOps Leaders Roundtable events. The team at Puppet co-hosted the event in their modern Aldgate Tower office. It was another incredibly successful evening where some of the best minds in DevOps joined forces to discuss some of the industry’s hottest topics.
The aim of this event was to provide a platform for leaders in DevOps to discuss their views surrounding implementing security models, managing implementation and communication and the key to building a successful team.
Once again Director of DevOps at Zoopla and Author of Next Gen DevOps, Grant Smith, joined us to chair the event and got the evening off to a start by opening the conversation around security model implementations.
Implementing a security model
One approach to implement a strong security posture that was popular with the group was implementing a test-driven security approach. Implementing security tests in the pipeline, initially non-blocking, with warning messages to educate engineers as to the standards expected. It was the experience of some members of the group that engineers will resolve these issues themselves even if the tests don’t block the deployment.
This led neatly to a discussion about classifying security tests by risk categories and threat. Some tests should block deployment when they fail, once the standards have been well defined and agreed between security and product engineering teams. Foundational concepts such as firewall rules are the most powerful. Vaccination is cheaper and easier than a cure.
It was noted that without a defined set of security standards there is no baseline and there’s a good chance that the security measures in place will be more about theatre than risk and threat. If a set of security standards are chosen then there are Git repos that can provide a lot of standardised security tests. These can take a lot of the toil out of creating a security culture in engineering. Testing is approximately similar to monitoring. Security testing should naturally morph into monitoring. Dashboards, monitoring and logging capabilities are key enablers for security.
The group found that generally smaller companies don’t tend to hire security professionals. They focus on simply upskilling devs. It seems that there are usually more devs than ops and more ops than security. This leaves a gap as even very security conscious product engineering teams often can’t spend the time to learn about new threats.
Managing implementation and communication
In order to be continuously learning and innovating an organisation’s security posture requires:
• Curating a security culture
• A process for upskilling engineers
• A security organisation providing interactive support and guidance to engineers
• A focus on learning and innovating and continuously improving security
One member of the group proposed an interesting idea: as security risks are discovered they could be raised in Jira and assigned to product teams. This could be part of an engineering excellence workstream.
It was also noted that it’s necessary to leverage the support provided by Cloud providers, pen-testers, and single-sign on organisations. These organisations are experts in security and can provide a great deal of support and guidance.
Security wargaming was another interesting topic that came up. Several members of the group have experience with this and felt that was a great tool for helping to create a security culture. Amazon’s Well-Architected Review process was also spoken about as another useful way to highlight security risks and escalate them to leadership.
Finally, some members of the group had experienced push back from addressing obvious security problems. This was because it was seen that doing so would impede development of an MVP. As a group, we felt this was nonsense as security breaches are far bigger impediments. In-fact MVPs present a great opportunity to iterate on security implementation.
How to build a team
Interviewing
On building successful teams, one of the techniques that the group spoke about was behavioural interviewing. Interviewing for the behaviours that will add to the culture of an organisation. This is most valuable if we recognise that skills can be learned if an engineer has the right core capabilities and a willingness to learn.
Another interesting interview technique that came up was bringing interview candidates in to work in sprint teams. Pairing them with a mature member of the team for a couple of hours. The feedback from existing team members about the candidate can be much more enlightening than any technical test.
Saying that it should be noted that the group does think tech tests are valuable. In particular, it was felt that basic tech tests were more valuable than very complex technical challenges. Tech tests can be a useful tool to gain an understanding of the candidates thought processes. However, it was also felt that conversational questions are more valuable than submitted tech tests. The conversational questions provide deeper insight into the candidates technical capabilities, their behaviours and their attitude and passion.
One member of the group described an interesting approach to interviewing structured around seeing how people deal with conflict. In a previous interview, they had created a situation in which two interviewers presented two different perspectives on a feature and then assessed how the candidate attempted to resolve the conflict.
Team behaviours and dynamics
When discussing admirable behaviours some members of the group were passionate that our teams need more people with soft skills. The kind of skills that can help create positive interactions with the product development teams that are often our customers and peers.
It was also felt that we needed to make a point of showing appreciation for what our engineers do, the evenings and weekends they work particularly to liberate high performance. A point was raised that we need to not be afraid of performance management. Generally the group was pretty comfortable with this, we’re all pretty experienced after all. It was raised that performance management is a great deal easier if the environment supports high performance.
We then got on to the subject of whether the team necessary to build a new capability were, in fact, the right ones to then manage that capability over the long term. It was felt that the type of people who step into a new problem domain and engineer a solution from the ground up, were not necessarily going to enjoy optimising and managing that capability.
Hiring people with the temperament to optimise and manage a solution and introducing them gradually to the team. Then moving the original team members out to work on the next big challenge would lead to a sustainable set of teams capable, not only of solving difficult problems but also to manage and evolve the solutions to those problems over the long term.
Further, it was felt that this would provide benefits to new, less experienced and potentially less skilled team members who could upskill. As well as to the existing team who develop their skills and behaviours as they mentor and train the new team members.
The group strongly agreed with hiring for diversity in skillset, background, perspective and behaviours. In our experience the highest performing teams, over the long term, are diverse. They have a balance of technical and soft skills and different levels of seniority and experience. Big picture thinkers and detail-focused people.
Poor performers and people who we can’t be trusted can be a bad influence on a team. Particularly on younger team members who will see their behaviours and either be disillusioned or be turned into a low trust individual.
The group were all in agreement that enablement, capability, communication and responsibility are a great description of DevOps. As well as a great description of what we’re looking for in our teams. We also felt When building a team every individual has different needs and management is about serving the individuals. Doing the right thing for the people is almost always the right thing for the organisation.
Finally, it was observed that long running teams need experiences that ebb and flow, they need crises and victories, life and work can’t be a constant grind.
Thank you
Our second roundtable event was a great success and we’re already in the process of planning number 3. Alongside our co-hosts at Puppet and roundtable Chair, Grant Smith, we were able to provide another fantastic opportunity for these DevOps leaders to learn from one another about the current trends in tech. We’d like to thank everyone who attended for contributing to such an insightful and fun evening.
If you’re interested in speaking at our next event or even attending one, please don’t hesitate to contact Third Republic or even the host Adam Elliott-Smith directly.
