Agile as a working practice
Agile originates from the US tech industry where the waterfall approach to software development once dominated. This followed classic product development practices where a concept was translated into a technical specification, teams focused on meeting that spec, then delivered a finished product at the end of a given period of time. Feedback and reworking was carried out after the initial product had been delivered.
Dissatisfied with both the lengthy process involved and its inability to respond to changing conditions (such as competitor activity, customer feedback or technical problems), a group of software developers created and published the Agile Manifesto. At its heart, this has four values and 12 principles (see Appendix 1).
The four values of agile:
1. Individuals and interactions over processes and tools
With a waterfall approach, it is the process being used which keeps a project moving forward. With agile, the individuals within the development team and their engagement with the business (or customers) become the motor for development. Organic, ongoing and constant communication by team members ensures this alignment between needs and outputs.
2. Working software over comprehensive documentation
One of the reasons for the lengthy cycles in software development in waterfall was the emphasis on capturing and approving each step, from specification through testing to delivery. Agile focuses on the creation of a minimum viable product as rapidly as possible. Documentation during this process becomes a set of user stories – these are one of the biggest changes in product development culture and hold a particular appeal for analytics.
3. Customer collaboration over contract negotiation
Negotiating the nature of the product at the outset and then waiting for it to be delivered was central to the waterfall approach. While this was intended to create the product exactly as the customer wanted it, the effect was to lock the customer out of the development process. Agile brings the customer into that process, demonstrating the prototype at regular stages and allowing for ongoing changes in its specification.
4. Responding to change over following a plan
When developing software using waterfall, changes could prove costly and create long delays. As a result, multiple, parallel components were worked on, leading to inter-dependencies and a required order for delivery – if one of these elements fell behind, everything would be delayed. Agile acknowledges change as a constant and a potential source for added value by recognising the importance of context and shifting intentions and adopting a system to cope with them, typically in the form of sprints.
Applying agile to data and analytics
Agile began its migration into the data and analytics space during the early Noughties when the rise of interest and investment in big data was bringing about rapid changes to the nature of the tasks these functions were being given. Just as digital transformation has focused on deploying modern channels and technologies which are customer-focused, so datafication has concentrated on giving decision-makers the evidence and tools they need to respond more rapidly to opportunities and threats.
Early advocates of agile analytics tended to focus on how it could be used to speed-up the deployment of technology within the D&A space, as with “Agile analytics: A value-driven approach to business intelligence and data warehousing” by Ken Collier. Given the rate of advancement in new data and analytical tools, from Hive and Pig through to R and Python, this is perhaps understandable.
However, the true potential for agile in this domain comes from adopting it as the operating model for the function itself, rather than for its platform. To do this, the four values of agile as a software development process need to be translated into values that apply to data and analytics:
1. Use cases over tools and techniques
Agile analytics (and data) should be driven by a close understanding of what the business needs – finding problems that need solutions – rather than what the D&A function already knows which might be of value – having solutions that need problems.
Ongoing capture of possible use cases creates the pipeline for the function. This then needs to be prioritised and the leading candidates tackled in a series of sprints. By rapidly prototyping and testing, the potential for a solution will soon be realised – or unworkable, low value solutions discarded before significant investment is made. Both analytics and data teams can work on the same use case provided they have an aligned timeline and milestones. Short sprints (ie, two weeks) ensure high dependencies do not build up as obstacles or delays are identified early.
2. Collaboration over ownership
Given the pace and iterative nature of agile as a methodology, it is essential that all stakeholders are involved throughout. This means that any changes in the requirement are communicated rapidly, early prototypes can be validated by the business quickly, and quality controls or data governance applied throughout each project. Collaborative technology (such as internal communications and project management platforms) and an important ingredient in ensuring each step is shared and understood. They also help to overcome the issue of one function taking ownership of the solution until it is handed to its customer.
3. Cross-functional teams over specialists
A key feature of agile analytics is the nature of the teams used to tackle each project. Scrums of cross-functional representatives from data management, data engineering, data science, lines of business, devops and governance come together to develop a specific solution. Sometimes, this happens within a data lab environment as described by McKinsey (see Figure 1) which allows it to operate without disrupting other programmes or business-as-usual operations. Some organisations operate with multiple scum teams within a data factory to allow for overlapping solutions to be progressed. In pursuit of a minimally-viable product, each solution is developed and tested at every stage before a final operationalised version is released.
4. Value over validation
One of the most challenging aspects of agile analytics is the constant need to ensure solutions will deliver value into the business and to shut down any project where this will not happen. Data science teams can find this part of the process hard to accept, particularly if a solution they have developed has been validated at a theoretical level. Despite the robustness of a model, it may be that the organisation is incapable of putting into practice through the absence of sustainable data supplies, governance issues or incompatible systems. Or it may be that the model does not offer sufficient commercial return compared to other options or within its own terms. Either way, the function must be willing to “kill its babies” when necessary in order to demonstrate its commercial mindset and stakeholder alignment.
Implications of agile analytics
Adoption of agile analytics is likely to drive greater value out of the D&A function – and therefore the organisation’s investment – but it does not happen from a blank sheet. Transformation of existing working practices, both within the function and across its shareholders, is likely to be required. To achieve this successfully, a number of approaches can be adopted:
1. Agile analytics leadership style
The characteristics of a leader of this type of methodology are somewhat different from conventional data and analytics, especially those of a chief data officer v1. According to Gartner, there are four effective behaviours:
- Relate: Share business insights that build team trust;
- Scout: Seek information from other stakeholders;
- Persuade: Engage, support and encourage the team;
- Empower: Delegate, coach and support team members.
The goal is to create a self-organising team which embodies critical thinking, strong communication, cross-functional working and a focus on value, all with minimal hands-on, top-down intervention, but with maximum support and recognition.
2. Hybrid agile
Among DataIQ Leaders members, there is a consistent view that pure agile working is not appropriate to the data and analytics domain. Instead, a variety of hybrids are seen as more effective. One is where an organisation has a waterfall culture that may be hard to change – here, pursuing analytics tasks in parallel within the framework of the planning of a project can help to speed up the move into testing and development. Another is to use the rigour of waterfall to plan, specify, budget and develop, but switch into sprints as soon as there is sufficient scope and support to do so.
Appendix 1
12 Principles of Agile:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity–the art of maximising the amount of work not done–is essential.
- The best architectures, requirements, and designs emerge from self-organising teams.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.