Software Engineering Processes

In software engineering, there is no silver bullet in how to go about developing and maintaining software. Elegant theories of computer science are rarely relevant to large complex problems that require a software solution. However, knowledge of mindset and practices can help in practice.

Central Software Engineering Activities	Description
Software Specification	The functionality of the software and constraints on its operation must be defined.
Software Development	The software to meet the specification must be produced.
Software Validation	The software must be validated to ensure that it does what the customer wants.
Software Evolution	The software must evolve to meet changing customer needs.

Software Requirements Document

Central document: software requirements document (or software requirements specification or SRS). It is an official statement of what the system developers should implement, and may include both the user requirements for a system and a detailed specification of the system requirements.

Software Process Models

A software process is a set of related activities that leads to the production of a software system.

A software process model is a simplified representation of a software process (e.g. sequence of activities).

Waterfall Model

The waterfall model is a simple linear plan driven software process model, with one phase after another. Output of one phase feeds into the next. Development of this plan-driven model starts only after requirements engineering and design.

Use Cases

• Embedded systems where software has to interface with hardware systems.
• Critical systems with need for extensive safety and security analysis.
• Large software systems part of broader engineering systems.

However, cons include inflexibity to change, and that propagated errors may require workarounds in the next one, and issues might be discovered only late.

Incremental Development

Incremental development model interleaves the activities of specification, development and validation. The system is developed as a series of versions (increments) with each version adding functionality to the previous version.

Pros: Cost of change and response is lower, and more rapid delivery of software. Cons: Process is less measurable and may have a degrading system structure (with refactoring required).

Integration and Configuration

With many existing open-source systems, libraries and frameworks, we can focus the process on re-using them rather than develop from scratch.

Pros: We reduce amount of software to be developed, advantage in cost and managing risk. Faster delivery of software. Cons: Requirements compromises and control over system evolution is lost.

Overall

Overall, all three listed are general models that specify the execution of individual software engineering activities.

(Agile) Software Engineering

Agile as both a mindset and concrete management and engineering frameworks

• Focus on delivering value in increments and/or avoiding waste
• Feedback loops and customer involvement
• Adaptive to changing requirements
• Lightweight processes
• Empowering teams and people (Software Engineering as a craft rather than industrial process)
• Inspired by lean manufacturing principles
• A lot of buzzwords and certifications

Agile Development: Historic Development

• 1980s and early 1990s
  During this period, software engineering focused heavily on structured, formalized approaches. The goal was to reduce risk and improve predictability, relying on:
  • Careful project planning.
  • Formalized quality assurance processes.
  • Use of analysis and design methods, often supported by advanced software tools.
  • Controlled, rigorous software development processes like the Waterfall method.
• Characteristics of Software Projects
  • Large, long-lived systems such as aerospace or government projects.
  • Geographically dispersed teams working on projects for extended periods (e.g., control systems for modern aircraft, taking up to 10 years).
• Late 1990s
  Dissatisfaction with traditional methods led to the development of agile methods. Issues included:
  • Delays of months or even years in delivering software.
  • Unpredictable processes that often resulted in products that failed to meet customer needs.
  • Overly rigid planning, outlined in tools like Gantt charts, which reassured management but failed to adapt to reality.

---

Agile Manifesto (2001)

• Written by representatives from frameworks like Extreme Programming, Scrum, DSDM, Adaptive Software Development, Crystal, and others.
• Drafted in an informal setting: The Lodge at Snowbird Ski Resort, Utah.
• Focused on four key values:
  1. Individuals and interactions over processes and tools.
  2. Working software over comprehensive documentation.
  3. Customer collaboration over contract negotiation.
  4. Responding to change over following a plan.
     • Note: This does not mean no documentation or planning; it prioritizes adaptability and collaboration.

Read the Agile Manifesto

---

Agile Principles in the Agile Manifesto

1. Customer satisfaction through early and continuous delivery of valuable software.
2. Welcome changing requirements, even late in development, to provide a competitive advantage.
3. Deliver working software frequently, with a preference for shorter timeframes (weeks to months).
4. Foster daily collaboration between business people and developers.
5. Build projects around motivated individuals with the environment and support they need.
6. Prioritize face-to-face communication as the most efficient way to convey information.
7. Measure progress by working software.
8. Promote sustainable development with consistent pacing for all stakeholders.
9. Emphasize technical excellence and good design to enhance agility.
10. Value simplicity by maximizing the amount of work not done.
11. Encourage self-organizing teams to produce the best architectures, requirements, and designs.
12. Regularly reflect and adjust behavior for continuous improvement.

---

Agile: Modern Reasons for Adoption

• Yearly surveys reveal increasing adoption of agile methods across industries.
• As of 2023, agile continues to gain traction due to its flexibility, emphasis on collaboration, and ability to deliver value incrementally.

• Agile principles align well with modern business demands for adaptability, efficiency, and customer satisfaction.

• Lean manufacturing inspired “lean software development”
• Concept of limiting or avoiding “waste”
• Kanban has been adopted as an agile method
• Introspection and continuous improvement is part of agile processes as well as DevOps

“Waste” in Software Engineering

• Modern Interpretation of Waste
  In software engineering, “waste” refers to unnecessary hardship or inefficiencies in daily work.

1. Partially Done Work
   • Incomplete tasks or features that are not yet providing value.
2. Extra Processes
   • Processes that do not add value to the customer, such as redundant approvals or documentation.
3. Extra Features
   • Features that are unnecessary for the organization or the customer, leading to overengineering.
4. Task Switching
   • The inefficiency caused by individuals working on multiple projects simultaneously.
5. Waiting
   • Delays caused by waiting for inputs, approvals, or other dependencies required to complete work.
6. Motion
   • Inefficiencies from handoffs between teams or individuals that require excessive communication or rework.
7. Defects
   • Issues such as software bugs, unclear requirements, or incorrect information that disrupt progress.

Agile Methods

Scrum

• General Framework
  • Lightweight framework for project management.
  • Does not prescribe specific approaches for requirements engineering, architecture, etc.
  • Commonly used synonymously with agile approaches due to its popularity.
  • Popularity: According to the Digital.ai 17th Annual State of Agile Report, 63% of agile teams use Scrum as their methodology.

Key Characteristics

• Prescriptive Rules: Clear rules to follow.
• Focus: Delivering value through increments in sprints.
• Foundation: Empiricism and lean thinking.
  • Empiricism: Decisions based on experience and observation.
  • Lean Thinking: Reduce waste and focus on essentials.

Scrum Pillars

1. Transparency: Work and process visibility for all stakeholders.
2. Inspection: Frequent artifact and process reviews to identify problems.
3. Adaptation: Adjustments made when outcomes are unacceptable.

---

Scrum Team

• No hierarchies: Self-managing teams.
• Team Size: Around 10 or fewer people.
• Roles:
  • Scrum Master: Facilitates the Scrum process, coaches the team, and removes impediments.
  • Product Owner: Maximizes product value, manages the Product Backlog, and communicates the Product Goal.
  • Developers: Create increments and are responsible for the Sprint Backlog.

---

Scrum Events

1. Sprint:
   • The “heartbeat” of Scrum.
   • Goal: Create a usable Increment.
   • Includes Sprint Planning, Daily Scrums, Sprint Review, and Retrospective.
2. Sprint Planning:
   • Define the Sprint Goal and lay out tasks.
   • Key Questions:
     • Why is this Sprint valuable?
     • What can be done this Sprint?
     • How will the work be done?
3. Daily Scrum:
   • A 15-minute stand-up meeting to focus on the Sprint Goal.
   • Adjust plans and refine the Sprint Backlog if needed.
4. Sprint Review:
   • Present results to stakeholders and discuss progress toward the Product Goal.
   • Collaborative working session, not just a presentation.
5. Sprint Retrospective:
   • Reflect on the Sprint and plan improvements for quality and effectiveness.
   • Results may include improvement items for the next Sprint Backlog.

---

Scrum Artifacts

1. Product Backlog:
   • An ordered list of work needed to improve the product.
   • Product Goal provides long-term objectives.
   • Items refined and broken down by developers for Sprint Planning.
2. Sprint Backlog:
   • Consists of:
     • Sprint Goal (why).
     • Selected Product Backlog items (what).
     • Actionable plans for delivering the Increment (how).
   • Updated during the Sprint as more is learned.
3. Increment:
   • A concrete step toward the Product Goal.
   • Must be usable, verifiable, and meet the Definition of Done.

---

Timeboxing in Scrum

Phase/Event	Duration
Sprint	1 month or less
Sprint Planning	Up to 8 hours
Daily Scrum	15 minutes
Sprint Review	Up to 4 hours
Sprint Retrospective	Up to 3 hours

• Source: The Scrum Guide

Kanban

Overview

• Kanban: A visual system designed to manage workflow efficiently.
  • Uses Kanban Boards with:
    • Cards: Represent work items.
    • Columns: Represent stages in the process.
  • Origin: Toyota Production System.
  • Focus: Optimize flow by reducing waste and enforcing Work-In-Progress (WIP) limits.

---

Kanban Board

• Structure:
  • Visual representation of the workflow.
  • Cards move across columns from “To Do” to “Done.”
  • Physical or digital boards can be used (e.g., GitHub Projects, Atlassian).

---

Key Concepts

1. Flow:
   • Represents the movement of value through the system.
   • Optimized by visualizing and tracking tasks.
   • Waste includes incomplete features that do not provide value.
2. Work In Progress (WIP) Limit:
   • Restricts the number of work items in progress at any stage.
   • Prevents bottlenecks and ensures steady progress.
3. Definition of Workflow (DoW):
   • Shared understanding of how work flows through the system.
   • Components:
     • Work Items: Units of value (e.g., user stories).
     • States: Defined stages (e.g., To Do → In Progress → Done).
     • Policies: Rules governing how work transitions between states.

---

Kanban vs. Scrum

Aspect	Kanban	Scrum
Workflow	Continuous flow	Time-boxed activities (Sprints)
Prescriptiveness	Less prescriptive	More prescriptive
WIP Limits	Enforced	Not explicitly enforced
Task Board	Visual task board is core	Often used but not mandatory
Usage	Flexible, can be combined with Scrum (“Scrumban”).	Focused on team roles and artifacts.

---

Combining Kanban and Scrum (“Scrumban”)

• Take a flow-oriented perspective in Scrum.
• Limit WIP to enhance flow efficiency.
• Measure and improve flow over time.
• Useful for teams transitioning between methodologies or looking for flexibility.

---

DevOps

Overview

• DevOps: Combines “Development” and “Operations” to emphasize collaboration across teams for delivering value efficiently.
  • Emerged around 2010.
  • A mindset, set of practices, and tools that bridge the gap between Development and Operations.
  • Focus: make production deployments routine and predictable.

---

Core Principles

1. Flow:
   • Reduce batch sizes for faster delivery.
   • Minimize handoffs to reduce delays.
   • Continuously identify and remove constraints in the delivery process.
2. Feedback:
   • Fast feedback loops from all stages to catch and address issues early.
   • Use automated processes (e.g., builds, integration, testing) to provide immediate feedback.
   • Promote shared learning across teams to improve resilience.
3. Continuous Learning & Experimentation:
   • Encourage a culture of experimentation to find better solutions.
   • Use standardized processes to document and share successful outcomes.
   • Ensure a blameless safety culture to support learning from incidents.

---

DevOps vs. Agile

Aspect	DevOps	Agile
Focus	Automates delivery and operations processes.	Deliver value frequently and iteratively.
Collaboration	Strong emphasis on Dev + Ops collaboration.	Cross-functional collaboration.
Metrics	Measures operational performance (e.g., DORA metrics).	Iterative delivery of value.
Parallels	Logical continuation of Agile practices.	Core methodology.

---

DORA: DevOps Research and Assessment

• Measures performance through four key metrics:
  1. Deployment Frequency: How often new changes are deployed.
  2. Lead Time for Changes: Time taken to go from code committed to production.
  3. Change Failure Rate: Percentage of deployments causing failure in production.
  4. Time to Restore Service: Time taken to recover from a production failure.
• Continuous Improvement: Use these metrics to optimize processes and enhance performance.

---

Key Practices

1. Continuous Integration and Deployment (CI/CD):
   • Frequently integrate changes, test them automatically, and deploy to production.
   • Requires automated tests and pipelines.
2. Infrastructure as Code (IaC):
   • Manage infrastructure configuration as code using scripts or configuration files.
   • Follow best practices like version control and CI/CD pipelines.
3. Value Stream Mapping (VSM):
   • Visualize the sequence of activities delivering value.
   • Identify inefficiencies and optimize processes.
4. Observability and Monitoring:
   • Monitor running code and infrastructure to ensure smooth operation.
   • Use insights to refine processes and improve the value stream.
5. Learning from Incidents:
   • Blameless postmortems: Investigate and address incidents without assigning blame.
   • Document findings and implement measures to avoid recurrence.

---

DevOps Lifecycle

• Often visualized as an infinite loop, representing continuous feedback between Development and Operations.
  • Plan → Develop → Build → Test → Release → Deploy → Operate → Monitor → Back to Plan.
  • The exact phases vary based on implementation but emphasize iterative improvement.