Ultimate Guide to Quality Requirements for Software Architects: Meeting Stakeholder Expectations

Table of contents

• A funny story as an introduction to a supposedly boring topic
• An introduction to the ISO standard and a practical look at it
• The alternative: Q42 - the quality model of arc42
• An example non-functional requirement from practise
• Introduction to quality scenarios
• Documenting non-functional requirements
• Further techniques for working with non-functional requirements

Every stakeholder has them: Non-functional requirements or at least (non functional-) expectations for the next software initiative.

• "It has to be fast."
• "It should be easy to maintain."
• "It should be scalable."
• "It should offer a good user experience"
• "A new feature can be developed quickly and easily."

But often no one can explain what they mean by performance efficiency, scalability, usability, coupling, maintainability and so on.

An essential task of a software architect is to ensure that the quality goals of a particular software system become concrete and, hopefully, measurable.

So let's start with an experienced ("funny") story from the real world as an introduction to a supposedly boring quality topic. 👇

A funny story as an introduction to a supposedly boring topic

I am often on the road as a consultant for various CTOs of SMEs. A few years ago, I supported an SME with an initiative for a new software product.

He proudly showed me the list of "non-functional requirements" of this particular software system and said

"Look, we've already identified the future non-functional requirements that we need to fulfil with the new system."

What he showed me was a list of about 30 (!) quality characteristics.

The list looked something like this:

• Scalable
• Reliable
• Usable
• Secure
• Maintainable
• Portable
• Efficient
• Compatible
• Accessible
• Testable
• Flexible
• Interoperable
• Performant
• Resilient
• Modular
• Intuitive
• Customizable
• Responsive
• Consistent
• Fault-tolerant
• Reusable
• Sustainable
• Robust
• Upgradable
• Integrable
• Localizable
• Concise
• High-capacity
• Traceable

This list or these "quality characteristics" were of course completely useless.

Because

1. The entire development team could not even name five quality attributes from this huge list.
2. In the unrealistic case that someone consults the Confluence page with these 30 quality attributes, they would not know what is meant by 'resilient' or these other quality attributes, for example.

I know that this non-functional topic can be boring or annoying because at first glance it doesn't contain any tech stuff.

But as a good software architect - and of course as a good requirements engineer / product owner - you have to do this work and make them concrete and priorisable.

Why?

Because these - and especially these - are your drivers and, more importantly, the reasons for almost all of your software architecture decisions.

But the good news is that in addition to all this boring quality stuff, there are also good and modern techniques for developing and maintaining explicitly non-functional requirements.

An introduction to the ISO standard and a practical look at it

Let's start with the probably boring stuff, the official ISO/IEC 25010 standard.

This quality model is the cornerstone of a system for evaluating system quality. The quality model defines which quality characteristics should be taken into account when evaluating the properties of a software product.

The product quality model defined in ISO/IEC 25010:2023 comprises the eight quality characteristics shown in the following figure:

Here's a brief summary of the ISO/IEC 25010:2023 quality characteristics:

• Functional suitability: Measures the ability of the software to fulfil stated and implied needs.
  • Functional Completeness
  • Functional Correctness
  • Functional Appropriatness
• Performance efficiency: Evaluates performance in relation to the amount of resources consumed.
  • Time Behavior
  • Resource Utilization
  • Capacity
• Compatibility: Evaluates the ability of the software to coexist and exchange information with other products.
  • Co-existence
  • Interoperability
• Interaction capability: Focuses on how easily and satisfactorily users can interact with the software.
  • Appropriateness recognizability
  • Learnability
  • Operability
  • User error protection
  • User engagement
  • User assistence
  • Self-descriptiveness
• Reliability: Examines the ability of software to perform under certain conditions over a certain period of time.
  • Faultlessness
  • Availability
  • Fault Tolerance
  • Recoverability
• Security: Refers to the ability of the software to protect information and data in such a way that persons or systems are only granted access that corresponds to their type and authorization.
  • Confidentiality
  • Integrity
  • Non-repudiation
  • Authenticity
  • Accountability
  • Resistance
• Maintainability: Considers how easily the software can be changed to correct faults, improve performance or adapt to changes in the environment.
  • Modularity
  • Reusability
  • Analysability
  • Modifiability
  • Testability
• Flexibility: The ease of adapting to changes in requirements, context or system environment
  • Adaptability
  • Scalability
  • Installability
  • Replacability
• Safety: Avoid a situation in which human life, health, property or the environment are endangered
  • Operational constraint
  • Risk identification
  • Fail safe
  • Hazard warning
  • Safe integration

Two areas recognizable

The ISO/IEC 25010 standard provides an overview of possible quality characteristics with a useful grouping.

At a high level, we can divide these quality characteristics into two areas:

• the quality of the system at runtime, i.e. the observable behavior. This has a significant influence on the quality of use by users.
• the quality of the system at development time, which has a significant influence on the quality of use for the developers or maintainers of the system.

An opinion

The currently published standard (ISO/IEC 25010:2023) lacks pragmatism and practical applicability, as it omits key terms such as deployability, energy efficiency or code quality and instead focuses on system properties that are described with less intuitive nouns than adjectives. This linguistic choice makes it difficult to discuss and understand quality characteristics in everyday working life.

Shortcomings of ISO 25010

Abstract: Published in 2011, the ISO 25010 standard on software product quality lacks pragmatism and practical applicability. Terms like scalability, deployability, energy efficiency, safety, or code quality are missing. This article explains these shortcomings and shows that even the (draft) update from 2022 still needs polishing… Before we dive into the details, please allow me some nitpicking. Adjectives versus Nouns The ISO standard (and most of the historical quality models too) is writing about the “properties of a system or product”. Such properties are crucial in everyday life, therefore our (natural) language provides adjectives as a specific category of words: In real life, people call things delicious, fast, usable or secure. Product owners want or require systems to be responsive, reliable, suitable, or easy to change. Too bad, computer scientists plus the ISO 25010 standard committee, as well as the inventors of traditional quality models, preferred to use nouns to describe properties: They use “flexibility” instead of flexible, “time-behaviour” instead of fast and many more “…ilities” instead of the beautiful, pragmatic, and understandable adjectives our languages provide. You might consider this nitpicking, but it’s definitely a linguistic downside, which makes the practical day-to-day work with quality characteristics and quality-requirements more difficult than it needs to be. ISO 25010 - An Overview Due to ISO 25010, software product quality is composed of eight top-level characteristics, which in turn consist of several sub-characteristics. While such a hierarchy is structurally simple, it gives rise to serious disadvantages - but more on that in a moment. Altogether, the standard attempts to establish a widely usable (“generic”) taxonomy of almost 40 terms in total. Presumably, all stakeholders should be able to recognize their goals and requirements represented there. In reality, this is not the case, as you will see right away. During development of a specific system or product, we have to refine that generic model for our system anyhow: We need to describe or demand those characteristics (aka qualities) that are relevant for our specific system. The diagram shows the top-level characteristics of ISO 25010 from the 2011 version, which is still the official version. Holes in the conceptual carpet Developers of embedded or real-time systems in medicine, aerospace, the automotive industry, chemistry, mechanical engineering, and others will immediately notice an omission: “Safety” (safety of life and health, physical integrity, etc.) is missing from this overview. And no, it does not appear as a subfeature in the ISO standard either. Now, for the teams that develop modern web or mobile applications: They will be surprised that operational or DevOps topics like deployability are missing from the standard. Oh yes, scalability, which is usually a challenge in online applications, is missing too. Well – if there are any product owners among you, they will likely miss “Time-to-Market” (TTM) because you surely want to bring your product to the market or to your users in a timely manner. If you think that something like TTM is not a quality attribute at all: It is, however, an attribute that some stakeholders want from a system, and thus it counts as a quality attribute for them. My conclusion so far: A few topics relevant from a practical standpoint are missing in the current version of the ISO 25010 standard. This is certainly not dramatic because we can add them to our system-specific model. However, in a standard document as detailed as ISO 25010 presents itself in many parts, there certainly would have been room for those missing pieces. Confusion under the Hood Let’s dive into the details, and head over to some of the about 40 so-called sub-characteristics of ISO-25010. Figure 2: ISO 25010:2011 detailed view Their sheer number might seem overwhelming at first, but let’s ignore this for now1. Functionality, no lo comprende2 Let’s start on the left of figure 2: The functional suitability subtree causes some confusion, at least in my head: At first, this term is quite uncommon, and definitely exceeds my everyday vocabulary. For me, “suitability” refers to the entire system, not just its function. A system needs suitable performance, availability and so on, not just suitable functionality. It goes further: One of the subfeatures is functional adequacy. I, at least, have failed to understand the difference between “suitability” and “adequacy”. Adapt, Maintain, Change? Let’s look further, to maintainability and transferability: conceptually, I see no real difference between “changeability” (subcharacteristic of maintainability) and “adaptability” (subcharacteristic of transferability). I want to adapt systems to changed requirements, not only adapt them for transfer to another operating system. This distinction might lead to confusion in practice. Limits of the Strict Hierarchy From my perspective, a tree structure is not suitable to structure a multi-faceted term like “quality”, as a strict hierarchy has its limits. Consider some examples: “Testability”: ISO positions testability under maintainability, which is understandable: When maintaining or changing systems, testability is an important property, to reduce risks. But: From my experience, people also consider testability an indispensable part of reliability. Therefore, testability should occur in both subtrees, not only under maintainability. “Availability” is located below reliability. However, if a system is not available, it is not usable. Therefore, availability also belongs to usability. “Adaptability” belongs to both portability and maintainability, but ISO decided to put it below portability only. A strictly hierarchical placement of system qualities will always be disputable. Another approach would be to allow detailed qualities to be “taggable”, so a specific quality might be tagged (or labelled) with more than just one top-level quality property. The recently published arc42 quality model allows exactly that (see [2]). My conclusion so far: A lot of mass, but important nutrients (e.g. conceptual consistency and clarity) are missing. These and other problems with ISO 25010 have initially been pointed out by Bass+2021. Does the 2022-Draft Improve the Situation? When ISO announced an update to the 2011-version in a mailing list in mid-November 2022, my expectations were high. After all, ISO has had 11 years to come up with an improved proposal. At first glance, it is noticeable that the top-level features have grown from 8 to 9, see the diagram below. Safety has made it into this draft. Definitely a massive step forward. Still, I see overlapping terms in the areas of “maintainability” and “flexibility”. This remains dubious even when considering the ISO definitions: Flexibility: ability to meet different or extended requirements, the ease with which changes can be adapted to requirements or the system environment. Maintainability: ability to be changed effectively and efficiently. From my perspective, these two sentences are too similar, and leave way too much room for interpretation. That will lead to useless discussions in practice. Below the usability category, aesthetics and accessibility are replaced by engagement and assistance. Again, we could argue about this — but in my perception, the term “accessibility” has become established in the UI/UX field, which is why this replacement rather bothers me. In the new version, I consider it positive that: scalability has been added, as a subcharacteristic of flexibility, and maturity has now become_faultlessness_. Isn’t Software made up from Code? Although internal structure, legibility and conciseness have been part of previous quality models (Boehm+1976 mentioned these qualities in his model from 1978!), the ISO refused (or forgot?) to include such properties in their model. In my opinion, development teams are important stakeholders, and structure and readability of code will often be essential goals for them. Ignoring code quality and component structure as quality requirements is simply not acceptable from my experience, and I consider their absence a serious deficit in the ISO 25010 standard. Practical Use? Although one can use the ISO 25010 standard as a checklist for specific quality requirements, a smaller but well-chosen selection of terms would be sufficient for this. For example, the arc42 quality model (Q42) consists of just eight key properties, instead of the 35 terms from ISO 25010. Maybe the most serious disadvantage of ISO 25010 is the complete lack of specific examples of how to apply the model to practical, real-world systems. That leaves development teams and other system stakeholders out in the cold. Bass+2021 have proven in their book that such examples can be systematic and useful. arc42 followed suit, providing numerous tagged and cross-referenced examples of quality requirements. Another potential source of unwanted complexity is the ISO metamodel, which distinguishes Quality-in-Use, Data Quality, Service Quality and Product Quality. (The metamodel is part of the official documentation in the “ISO/IEC 2501n Quality Model Division”, explained within the non-public parts of 25010. It refers to additional standards 25011 (IT service quality model), 25012 (Data quality model) and 25019 (Quality-in-use model), cf. ISO+25011). In my opinion, this makes the ISO 25010 model overly difficult to digest for real-world projects. My final complaint is the paywall: The price of almost 140 Swiss Francs (approx. €140 and US-$140) might prevent people from reading the complete document. Fast Food or Gourmet Dinner? ISO 25010 is neither. Even the 2022 draft update of ISO 25010 doesn’t go beyond canteen food: Versatile, but difficult to access due to the multitude of overlapping and sometimes confusing terms. Formality-lovers will adore the metamodel, but practitioners will lack practical guidance. The often abstract definitions of terms build up superfluous hurdles. Specific, usable examples of practical applications are missing (again, arc42 to the rescue). In my opinion, less would have been more: Instead of nearly 40 terms at various levels, a few top-level features would be sufficient, coupled with suggestions or examples for each of them. In this respect, Len Bass and Co. have done an outstanding job with their fundamental work Bass+2021. They had already invented the concept of “quality scenarios” many years ago. In their fourth and latest edition of their book, they followed up with a multitude of practical examples. You can find a brief overview of their work in another article. Acknowledgements Thanx to “m” , Anja Kammer, Daniel Lauxtermann, Joachim Praetorius and Ben Wolf for bug fixing and dramatically improving the quality of this post. Thanx to Eberhard Wolff for encouragements. The 40 qualities from ISO 25010 might seem a large number, but one can easily identify way more potential qualities, when considering a broader range of stakeholders. Wikipedia lists nearly 90 “system quality attributes”, the collection on quality.arc42.org even more. ↩ For readers who, like myself, lack Spanish language skills: “I don’t understand that.” ↩

arc42 Quality Model

Shortcomings of ISO 25010

Published in 2011, the ISO 25010 standard on software product quality lacks pragmatism and practical applicability. Terms like scalability, deployability, energy efficiency, safety, or code quality are missing. This article explains these shortcomings and shows that even the (draft) update from 2022 still needs polishing…

INNOQDr. Gernot Starke

The alternative: The quality model of arc42

As ISO 25010 lacks practical guidance and pragmatism, there is an alternative approach proposed by arc42 - the arc42 quality model "Q42".

The arc42 Quality Model "Q42" is a straightforward and practical method for assessing product and system quality.

It starts with understanding stakeholder expectations and requirements to identify 8 key system properties. These properties are designed to encompass most of the 100+ traditional quality attributes that are required, desired or expected.

Eight typical stakeholder groups in a software initiative

In this approach, the focus is initially on the stakeholders.

They have identified the typical stakeholder groups of a software initiative, which often have different expectations of the quality of a system:

• Users expect the product to be user-friendly, reliable, readily available and safe.
• Management strives for efficiency in development and operational costs and for reliability.
• Domain experts demand that the system is reliable and efficient in its core functions and can be flexibly expanded with new functionalities.
• Product owners look for flexibility to incorporate new features.
• Developers want a system that is flexible and reliable in terms of code maintenance and is easy to test.
• Testers need a system that is easy to test and delivers consistent results, and value flexibility in automation and instrumentation.
• Administrators value ease of operation and maintenance, integration with other systems, security, flexibility in the execution environment and overall safety.
• Others may have additional expectations that are not explicitly listed here.

Eight typical qualities of a system as #

The quality model proposes eight tags that cover the qualities of a system.

• #reliable: The system should be consistently available, strong, trustworthy, and have the ability to tolerate faults and recover from them, ensuring safety and security.
• #flexible: It should be adaptable during development and in different infrastructure and runtime environments, allowing changes, extensions and integration.
• #efficient: The system is expected to perform quickly and make efficient use of memory, CPU, threading, network, and other resources, and should be maintainable and operable with efficiency.
• #usable: It must be user-friendly, visually appealing and geared to the needs of the user.
• #operable: The system should be easy to install, operate, monitor and maintain, with reasonable operational costs.
• #suitable: It should provide the required features, functionalities, and responsibilities.
• #secure: It should ensure the integrity, confidentiality and accountability of data and protect against unauthorized access.
• #safe: It should be designed to prevent and manage failures, provide timely warning of potential risks and detect them early.

Each of these tags covers a group of related quality attributes. E.g. #reliable covers qualities like

• Accuracy
• Availability
• Backward compatibility
• Capacity
• Clarity
• Compatibility
• Compliance
• Correctness
• Dependability
• Fail safe
• ....

The quality model provides a good scheme to capture relevant qualities of a system by going through these typical eight quality groups and going through the potentially relevant qualities from the perspective of each important stakeholder group.

Based on the identified qualities, you can concretize these qualities with suitable quality scenarios.

An example of a non-functional requirement from practise

Some time ago the PO came to me and said:

"When the user opens the frontend, it must be fast. So it has to be there immediately so that the user can start filling out the form."

If we take a look at Q42, this non-functional requirement can obviously be assigned to the top-level quality property #efficient. If we look at the qualities currently assigned to this top-level group, we find efficiency, which we can probably assign to this "high-level" non-functional requirement.

But let's break down this high-level non-functional requirement that was expected by the product owner.

When I look at the statement from the PO, I notice the following parts:

1. [...] opens the frontend, it must be fast. [...]
2. [...] must not be slow [...] be immediately there [...] that the user can start filling up the form.

Let's create a quality scenario for this example in the next chapter. 👇

Introduction to quality scenarios

Quality scenarios are a great and proven technique to make these lofty and imprecise expectations clear.

Quality scenarios were introduced by Len Bass et al. in the great software architecture book Software Architecture in Practice.

A quality scenario consists of the following elements:

• A stimulus source: a stimulus must have a source. It must come from somewhere. E.g. from a person, a system or another actor
• A stimulus: An event that arrives at a system, e.g. a user action, an attack, a request, ...
• An artifact in a specific environment:
  • The stimulus encounters a specific target - the artifact.
  • The environment is the set of circumstances in which the scenario takes place.
• A response: The response is the activity that takes place as a result of the arrival of the simulus
• A response measure: When the response occurs, it should be measurable in some way so that the scenario can be tested - that is, so that we can determine whether the product team has achieved it.

An example quality scenario for "opening fast the first view of a frontend"

Ok, let's try to write a quality scenario for the PO expectations mentioned.

Defining the stimulus source

The stimulus source in this scenario is the "web app user".

Defining the stimulus

The stimulus or event is "Loading the web app via https://my-web.app".

Defining the artifact and its environment

The artifact is the "web frontend" and the environment is the target browser (e.g. a Chromium based browser)

Defining the response

The response is "The browser loads and renders the web frontend"

Defining the response measure

The response measure is (and this is where explicitness and measurability come in): "The main content of the web front end within the view port is loaded in 2.5 seconds or less."

The resulting quality scenario

The whole resulting quality scenario is pretty cool to discuss with the Product Owner and / or other stakeholders about their expectations.

The quality scenario can be documented in tabular form.

Btw. these quality scenarios can be very easily linked to a developer task or a specific user story.

The great thing about this response measure is that it can be tested automatically via Lighthouse. This metric is called Largest Contentful Paint (LCP).

The LCP and other metrics can be easily measured via Lighthouse.

Tip: If you want to learn more about frontend architecture and quality goals, take a look at the following blog post.

Which web frontend architecture fits best?

This post will help you find the right web frontend architecture that best fits your specific quality goals.

workingsoftware.devPatrick Roos

Documenting non-functional requirements

If we look at the arc42 software architecture documentation template, there are two chapters in which the quality goals are mainly documented.

Chapter 1: Introduction and Goals

In this chapter, you document the three (maximum five) top quality goals for the architecture, the fulfillment of which is of utmost importance for the most important stakeholders.
Again, I often refer to the top-level quality groups of Q42 and describe the most demanded quality expectations of the different stakeholders

Chapter 10: Quality Requirements

This is the main chapter in which you document the quality requirements in detail. You can get a good overview with a quality tree, followed by the specifically described quality scenarios.

You can find a good example of a quality tree with a link to the quality scenarios here in the good arc42 example dokchess.

Further techniques for working with non-functional requirements

There are other techniques for working with non-functional requirements in the context of non-functional requirements.

Architecture Characteristics Worksheet

The Architecture Characteristics Worksheet is a tool for identifying and documenting the key characteristics that define the architecture of a system, focusing on aspects such as scalability, performance, reliability and security. It helps stakeholders understand and prioritize architecture requirements by breaking down complex systems into manageable components, each with defined attributes and evaluation metrics. This worksheet is a critical step in the architecture design process, facilitating discussions and decisions that ensure the system achieves its intended goals and performs effectively in its operational environment.

Architecture Inception Canvas

The Architecture Inception Canvas, which is ideal for new software initiatives, has a special section to capture or elaborate the quality goals of a new software initiative.

Architecture Inception Canvas

An Efficient and Collaborative Way to Define your Software Architecture Playground

Software Architecture CanvasGernot Starke, Patrick Roos and arc42 contributors.

Architecture Communication Canvas

If you are working out an existing architecture of a system and want to communicate it, you should take a look at the Architecture Communication Canvas. The Architecture Communication Canvas has an extra section for quality requirements, in which you enter the elaborated quality goals on which the software architecture is (probably) based.