Tuesday, September 4, 2018

Agile and Scrum Basics

I have about 15+ years of Software Development Experience (2018) and have been working on Agile Projects since about 2006. My First Formal Introduction into Agile was through a Training by Thoughtworkers (Thoughtworks is a Leading Agile Company). This was while I was a Senior Software Engineer at Huawei, Bangalore, India. I have worked on Agile/TDD/Pair Programming (Various Variants) in multiple companies including Huawei, Symantec, Yahoo, Finastra*, Oracle*, OpenText*. Recently and Once Again, I attended a Formal Classroom Training (Company Internal) on Agile. I jotted down the most important points and now am presenting it in this Blog. I hope it helps and becomes a Ready Reckoner for Understanding/Learning the Agile Basics (Needs, Motivations, Practice and Evolution Story).

* [Original Product Firms were Acquired by these Current Companies]

We will start with Dilbert!

[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]
 


 
 The Waterfall Accident 
Formal Software Development, specifically for who started into the field in 1990s or even 1980s was essentially a very Heavy Plan Driven Approach. The name given to it was Waterfall Software Development Process (As a Result of Being a Set of Successive Steps akin to a Waterfall). Even Academia captures it in textbooks with the name as the Waterfall Process. The proponent of this approach was an American Computer Scientist by the name Winston Royce. In his 1970s paper, he first formally defined the Stage Wise Waterfall Process (Though he is not the one who named it 'Waterfall'). The sequence wise phases that were identified by him are as follows:
  1. Requirements (System)
  2. Requirements (Software)
  3. Analysis
  4. Design
  5. Development (Coding)
  6. Testing
  7. Operations

[CREDITS BEGIN: WIKIPEDIA/OTHERS]
Royce called them "Implementation Steps to Develop a Large Computer Program for Delivery to a Customer".  
[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]


Royce actually foresaw a major shortcoming in this methodology, which he described as:
The testing phase which occurs at the end of the development cycle is the first event for which timing,storage, input/output transfers, etc., are experienced as distinguished from analyzed. These phenomena are not precisely analyzable. They are not the solutions to the standard partial differential equations of mathematical physics for instance. Yet if these phenomena fail to satisfy the various external constraints,then invariably a major redesign is required. A simple octal patch or redo of some isolated code will not fix these kinds of difficulties. The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated.

According to Royce; In the Process Model "The Design Iterations are never Confined to the Successive Step", and for that model without iteration is "Risky and Invites Failure". As alternative Royce proposed a more Incremental Development, where every next step links back to the step before.
[CREDITS END: WIKIPEDIA/OTHERS]

 Sometime Later and Ignoring the Shortcoming stated by Royce, the American Department of Defense accepted this "Waterfall Model" as the Standard for Defense System Software Development.  It promoted the One-Pass Document-Driven Waterfall process. It contained sentences like “The Contractor shall establish the Top-Level Design of each CSCI by Allocating Requirements from the SRS and, if applicable, IRS to the TLCSCs of each CSCI.” This was the beginning of the real widespread usage, application and practice of the "Waterfall Model". This model was drafted under the document DoD STD 2167. The Principal Author of this document later regretted promoting such a rigid process. He also agreed that he was not familiar with a Iterative, Evolutionary Requirements Based or Incremental Development Process. Even though, In 1987 DoD started warning against the "Waterfall Model" - It was in such widespread usage that Organizations all over the World refused to update their Process in their Software Development Documents and/or in Real Practice. The above events related to the Waterfall Process or the Original/First Formal Software Development Process are often termed as the The Waterfall Accident.

Shortcomings of Waterfall (Visualization) 
As Everywhere Else, I am representing the Real-World Situation in Usual Software Project Development (Aggravated by The Waterfall Model). 


[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]

There is this one more diagram, that also emphasizes the reason as to why the above situation is created. It is primarily due to the rigidity in the process and also the barriers to move back and forth between various phases. The diagram is good in proving that without iterative back and forth movements between the core phases, neither changing requirements can be incorporated reliably nor can any newly identified constraints.
[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]


 
Agile Manifesto (What is Agile?)
In 2001, group of 17 Software Thinkers and Practitioners met in Utah to arrive at a set of four common values that led to the creation of the Agile Manifesto (http://agilemanifesto.org/). I am quoting from this site directly, 

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and Interactions over Processes and Tools
Working Software over Comprehensive Documentation
Customer Collaboration over Contract Negotiation
Responding to Change over Following a Plan

That is, while there is value in the items on
the right, we value the items on the left more.

This was the beginning of the Agile Movement / Revolution, that now is the Software Development Process of choice across the world.

Twelve Principles of Agile
The above mentioned group of 17 people had also chalked out the Twelve Driving Principles for Teams that Adopt Agile. You may find the actual 12 principles at the following link: http://agilemanifesto.org/principles.html

  1. Customer Satisfaction by Early and Continuous Delivery of Valuable Software
  2. Welcome Changing Requirements, even in Late Development
  3. Working Software is delivered Frequently (Weeks rather than Months)
  4. Close, Daily Cooperation between Business People and Developers
  5. Projects are built around Motivated Individuals, who should be Trusted
  6. Face-to- Face Conversation is the Best Form of Communication (Co-Location)
  7. Working Software is the Primary Measure of Progress
  8. Sustainable Development, able to maintain a Constant Pace
  9. Continuous Attention to Technical Excellence and Good Design
  10. Simplicity— the Art of Maximizing the Amount of Work Not Done—is Essential
  11. Best Architectures, Requirements, and Designs emerge from Self-Organizing Teams
  12. Regularly, the Team Reflects on how to become More Effective & Adjusts Accordingly
Visualizing Agile
Since we were discussing about Waterfall, you should first understand how the Agile Process can/should be Visualized. This will allow you to understand the Flexibility, Adaptive, Iterative and Incremental Nature of Agile Methodology.
[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]



Also, from the perspective of the Stakeholder(s) Developing and Delivering the Software, It can/should be visualized as the following. This is to understand that the Chunks of Work are better manageable than getting bogged down by the Big Block of Work. It also should allow to Increase Quality, Manage Timelines, Continuous Delivery and  Achieve Greater Customer (For Whom the Work Is Done) Satisfaction.



[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]



Original Agile Signatories (Who Started Agile?)
I am also mentioning the names of these original 17 Software Developers who were the original signatories of the Agile Manifesto. They should continue to find special mention in years to come, as they were the ones instrumental in Formalization of such a  game-changing Software Development Process. Together, they published the Agile Manifesto for Software Development. 

Kent Beck
Mike Beedle
Arie van Bennekum
Alistair Cockburn
Ward Cunningham
Martin Fowler
James Grenning
Jim Highsmith
Andrew Hunt
Ron Jeffries
Jon Kern
Brian Marick
Robert C. Martin
Steve Mellor
Ken Schwaber
Jeff Sutherland
Dave Thomas


Types (Practice, Frameworks) of Agile (How to Agile?)
Since Agile only provides Values and Principles to guide Development Teams, there are many Methods or Methodologies through which Agile is Implemented. Each of them may require an article of their own. Also, I am familiar with only about 5-6 of these Methodologies to really be able to write about them comprehensively. I am enlisting first the Software Development Frameworks through which Agile can be Implemented. I am providing only the Topmost Implementation Methods, as per my selection. There may be many more ways to Implement Agile, with some or the other Innovative Value Add being done to the Agile Process every now and then. I have also provided the Wikipedia Links so that you may directly read about them.


Apart from the above mentioned Agile Development Methods, there are multiple keywords and practices that are associated with Agile Development. I am only mentioning the ones that I am most familiar with, along with their direct Wikipedia Links.



 Scrum Basics
The term Scrum was first mentioned in 1986 Harvard Business Review Article by  Hirotaka Takeuchi and Ikujiro Nonaka. They compared High Performing Cross-Functional Teams to Scrum Formation in Rugby. Scrum is a way to Implement Agile and teams working are called as Scrum Teams. The Five Values that should Drive Scrum Teams are Below:

Focus: Because we Focus on only a Few Things at a time, we work Well Together and produce Excellent Work. We deliver Valuable Items sooner. 
Courage: Because we work as a Team, we feel supported and have more resources at our Disposal. This gives us the courage to Undertake Greater Challenges.
Openness: As we Work Together, We Express How We're Doing, What's in Our Way? and Our Concerns, so they can be addressed.
Commitment: Because we have Great Control over our Own Destiny, we are more Committed to Success.
Respect: As we Work Together, Sharing Successes and Failures, We come to Respect each other and to help each other become Worthy of Respect. 
[ https://www.scrumalliance.org/learn-about-scrum/core-scrum-values-roles ]


 Three Pillars of Scrum that are Fundamental to Scrum Include the Following:

Transparency: Advocates that the Significant Parts of the Process to be Visible to All.
Inspection: Scrum Artefacts Constantly Inspected as also Progress towards Milestones.
Adaptation: Deviation of any Process Aspects Outside Acceptable Limits must be Adjusted.

[ http://www.agilistix.com/scrum/three-pillars-of-scrum/ ]



 Scrum Roles  
The real world Implementation of Agile through Scrum has Three Important Roles. 

Development Team/Member
Takes on Tasks and Delivers Chunks of Work, In Frequent Increments

Scrum Master
Protects the Scrum Process and Prevents any Distractions.

Product Owner
Determines what Needs to be Done and Sets Priorities to Deliver the Highest Value

[ https://www.scrumalliance.org/agile-resources/scrum-roles-demystified ]

Companies Adopting Agile
I myself have worked for Top Software Companies of the World including Yahoo (Altaba), Symantec, Huawei, Siebel (Oracle), GXS (OpenText) and Misys (Finastra). Also, I have worked for some well known IT Services/Consulting Companies like Infosys, Headstrong (Genpact) and also relatively smaller Product Brands like Persistent and Aptean. Since I started my career in 2003, I saw the move towards Agile Adoption (Variants, Loose Variants) in Various Companies throughout my Experience - It was really exciting for me as a Software Developer since the greatest crib that I ever had was Excessive Software Documentation and the Lack of Energy and Excitement. The idea of Regular Delivery of Working Software (Demos) was inherent and natural to me as a Software Developer. It is the one that brings the Greatest Joy (Apart from Everything, as Discussed in Earlier Sections) in Software Development. I am enlisting few of the Other Companies that have Adopted Agile (or are Agile Proponents), across Business Lines.

  • Google
  • Microsoft
  • ThoughtWorks
  • CA Technologies
  •  Barclays
  • Ericsson

 From my Own Experience, I can comfortably say that almost all companies in the world now use Agile/Scrum (and/or Variants) as one the Software Development Methodology of Choice. Also, Most Software Development Companies (Read Product Software) were always almost Agile, but now may be using the Formal Agile Principles as their Driving Factor to achieve Greater Efficiency. Many Companies use Variants of Agile or Mix of Agile with Other Processes for their Teams. Agile is also not restricted to Software Development - There are Industries and Functions that now use Formal Agile Methods for their Deliverables and Daily Tasks.


 
Popular Tools for Agile/Scrum
You will come across these Tools and Technologies that are used to Implement or Drive Agile/Scrum Methods and Practices in Organizations Worldwide. There may be many more Tools, Frameworks, Technologies - But I am only enumerating either the ones that I have come across or the Ones that are Popular. They may not be directly Agile Tools, but ones that either Accelerate Agile, Used for Agile Project Management, Agile Planning, Agile Task Management and Continuous Integration/Delivery.
  • Rally Agile Platform
  • Atlassian JIRA
  • (Microsoft) GitHub
  • Apache Jenkins
  • Zoho Sprints
  • Visual Paradigm
  • Thoughtworks Slack
  • Thoughtworks Mingle
  • Microsoft Team Foundation Server



Agile Pitfalls (Reasons for Failure)
Since we are now familiar with the very basics of Agile and Scrum, It is time to move on and find out the Statistics in Real-Time. These statistics reveal that even though Agile is no silver bullet, but it does provide a very high project success rate. Usually, Companies mix Agile with Other Existing or Newer Development Processes to try and increase success rate, as per the Context. 

 [CREDITS: IMAGE IS COPYRIGHTED TO VITALITY CHICAGO]


 This article will try and cover the reasons for the failure of Agile Projects - From the above statistics we can easily observe that the Project Success Rate of 42% in Agile is almost 1.6 times that of Waterfall. At the same time the ratio of Failure or Challenged Projects is 1 : 1.27 (Agile : Waterfall).
 
[CREDITS: IMAGE IS COPYRIGHTED TO VITALITY CHICAGO]
 

The Best Candidates for Waterfall are Large to Very Large Projects which have very Static Requirements and with little or no External Factors or Systems impacting them. In such cases, Waterfall has greater chances of yielding Higher Quality Output in possibly  than Agile. 

Next, I am enumerating the reasons for Failures of Agile Projects, themselves. Since, It is a Highly Dynamic and Adaptive Process, I am always a strong believer that Lack of Highly Skilled Workers as one Top Reason for Failure of Agile Projects. With Agile, the Time in Utilizing Current Skills and Experience is the Most Important Differentiating Factor in Success of a Project. There is a Constant Focus and Pressure of Constant Delivery that the Holistic Skill Gain from Agile has to be through Individualistic Discipline and Practice. The Outcome of Agile in Short Term is again, creation of Lowly Skilled or Semi Skilled Workers. The Task of Gaining In-Depth Knowledge or creating Absolute Experts is no longer Automatically Supported. This may Impact Quality of the Deliverables and Software itself. But then, through Individual Practice, Character and Discipline, the Individual Engineer or Developer may be different at a Skill Levels from others, even at the Same Experience Level. So, I am only enumerating the reasons here for failure, as an In-Depth discussion is beyond the Scope of this Article. You may read these reasons, more in detail at the following link: https://www.agilealliance.org/8-reasons-why-agile-projects-fail/

  1. Insufficient Training / Low Skill Level
  2. Unwillingness of Team to Follow Agile
  3. Broader Organizational or Communications Problems
  4. Lack of Support for Cultural Transition
  5. Pressure to Follow Traditional Waterfall Process
  6. Lack of Management Support (Transitioning to Agile)
  7. Company Philosophy at Odds (Versus Agile)  
  8. Lack of Experience in Agile (Transitioning to Agile)


Agile/Scrum in Real-World (2018)
So, The thoughts presented in this write-up are an Introduction to Agile and Scrum. In real-time, What is it that is really followed/practiced in Agile Projects? I am mentioning this from my Own Experience of Working in Agile Projects (and Variants). If you have never worked in an Agile (or Variants) Project, you may find the following in an Agile Project. It is not that all of this never existed in Waterfall, but mentioning this in a (best-effort) sequence that you may come across in an Agile Development Project.

  1. Project Kick-Off
  2. Project Milestones
  3. Sprint Planning
  4. Backlog
  5. Acceptance Criteria
  6. Story (T-Shirt) Sizing 
  7. Fibonacci Scale 
  8. Sprint Tasks
  9. Capacity
  10. Daily Scrum
  11. User Story
  12. Technical Story
  13. Technical Spike
  14. Research Spike
  15. Architectural Spike
  16. Refactoring Spike
  17. Burn- Down
  18. Burn- Up 
  19. Velocity 
  20. Sprint Demo
  21. Sprint Review
  22. Sprint Retrospective
  23. Project Retrospective
  24. Go-Live

I myself discovered these two separate manifesto's that are directly aligned to Agile Movement - One of them provides guidelines for Software Developers and other for Project Management. I am mentioning few salient points and the links where you can read more on these. In the real-world, as per me, you may not come across these as being used directly - but possibly being practiced without knowledge of such a Formal Manifesto.
Software Craftmanship Manifesto (Software Developers/Craftsmen)
I am quoting from the site directly, http://manifesto.softwarecraftsmanship.org/ - You may visit the link and also sign the Software Craftmanship Manifesto.

As Aspiring Software Craftsmen we are raising the bar of Professional Software Development by practicing it and helping others learn the craft. Through this work we have come to value: 
Not Only Working Software, But also Well-Crafted Software
Not Only Responding to Change, but also Steadily Adding Value
Not Only Individuals and Interactions, but also a Community of Professionals
Not Only Customer Collaboration, but also Productive Partnerships  

 That is, in pursuit of the items on the left we have found the items on the right to be indispensable.


After reading this manifesto for the first time, I strongly feel that it has taken the Agile Manifesto rightfully ahead and should be a mandatory read for every Software Developer. The folks who are being introduced to Agile as Software Developers, should thoroughly note the points and make it a part of their Daily Activities and Tasks.
Declaration of Interdependence (Project Management)
Continuing on the Agile Manifesto to align Management Principles that are required to achieve an Agile Mindset in Product and Project Management. The Six Principles that are essential to Modern Project Management. You may read more on this Declaration at the following link: https://en.wikipedia.org/wiki/PM_Declaration_of_Interdependence
Increase Return on Investment by making Continuous Flow of Value our Focus. 
 Deliver Reliable Results by engaging Customers in Frequent Interactions and Shared Ownership. 
 Expect Uncertainty and manage for it through Iterations, Anticipation and Adaptation. 
 Unleash Creativity and Innovation by recognizing that Individuals are the Ultimate Source of Value...
Boost Performance through group accountability for Results and Shared Responsibility for Team Effectiveness. 
 Improve Effectiveness and Reliability through situationally Specific Strategies, Processes and Practice

The above Declaration of Interdependence outlines the Leadership Approaches used to manage the Inter Dependency of People, Processes and Value to enhance the Performance of Work. The Six Key Areas are:
  1. Focus on Value
  2. Engage Customers
  3. Expect Uncertainty
  4. Unleash Creativity
  5. Group Accountability
  6. Improve Effectiveness

 
Happy Agile Development to All! 
 
at
Labels: , , , , , , , , ,
Location: Manila, Metro Manila, Philippines

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)