Summary

Many developers don’t speak to users or collaborate with the business, instead they take direction from email threads and poorly written tickets. Remote, outsourced and offshore developers know this only too well. Even with better access to people, these developers would struggle to navigate organisational complexity, political decision-making, and siloed systems.

Being on the receiving end of a poorly performing development team is upsetting, particularly if you’ve tried everything to fix the situation but low throughput and poor workmanship remain. The business pays the hidden price for this; individuals least well-placed to know what’s going wrong. Unfortunately, these experiences are only too familiar.

This handbook is written for development teams and their managers who need to write excellent software and delight users. It explains when software development works best, what happens when your team can’t work like that, and provides practical advice to address common pitfalls. Teams that adopt the practices of this handbook can share their experiences directly with the author, who will be happy to respond.

Questions and feedback: info@bettersoftware.uk


Table of Contents


Introduction

When software development works best

Software development works best when developers interact directly with users, hearing about their experiences with the software and discussing what should come next. Conversations replace the need for requirements, and ideas and concepts are quickly put together and pushed out the door. Developers confidently release to real users because high-quality processes guard against adverse effects. Feedback happens in near real-time; a simple comment, under-the-breath muttering or facial expression is often enough to gauge reception. It sounds like fantasy, but commercial software really was built like this circa year 2000, and still is in many startups and small companies.

This approach works because developers know multiple technologies and can work up and down the stack, delivering entire working features themselves. Developers create database tables and stored procedures, develop middle-tier code and author front-end interfaces (often not very good ones, I admit). They may have strengths in particular technologies, but what’s most important is delivering whole solutions. Additionally, these developers converse with real-life humans, sketch out rudimentary designs, and even test finished code, a far cry from most developers today.

Why software development goes wrong

The distance between the end user and software developer is the biggest indicator of whether your software product will delight users, or be a poor proxy of misunderstood needs. The further you stray from close collaboration between users and developers, the greater the likelihood of things going wrong, and the greater the need to manage the software development process carefully. Developers not sitting with actual users, or user advocates, break the close collaboration and direct feedback loop that avoids the need for formal software requirements and software specifications in the first place.

Technical silos

Modern software development is very different from the early 2000’s. Technical complexity means developers specialise in one or two technologies or frameworks, teams are distributed globally, ‘product ownership’ and ‘user experience’ are entirely new functions, and DevOps sits outside the development team. I’m not against any of this, nor am I advocating a return to an earlier time; rather, I’m pointing out how things have changed in the software development industry.

Large corporations develop software in ways that definitively break the link between users and developers, leaving a communication gap of monumental proportions. Users and developers no longer collaborate; sometimes, they have never even spoken to one another. Remote development teams, outsourced and located offshore in supplier ‘centres of excellence’ is standard practice. The user feedback loop is broken too, needing to traverse through layers of product owners, user researchers, designers and business analysts, before ending up in ticketing systems like Atlassian Jira and Azure DevOps.

Poor management

Someone needs to tell the developers what to do, otherwise certain chaos ensues. Good software developers can tolerate a measure of ambiguity in their work; many enjoy putting their skills to good use and resolving unknowns. What developers cannot do, however, is make decisions on behalf of a business that doesn’t know what it wants or navigate political environments and organisational complexity from behind their development IDE. Developers struggle to do this.

Someone, somewhere, needs to convert ideas and concepts into instructions for the software developers, but it’s hard to do well because you really need to understand the business problem you are trying to solve. Then, you must communicate effectively. Not knowing what to tell the developers isn’t an option, unless you want poorly built software, idle hands, or conversations and ideas that never become a reality. Each developer needs a prioritised work stack and sufficient work to do; large development teams with broad skill sets make this particularly tricky.

High complexity

Some software projects are more complicated than others. The risk of unexpected problems and total project failure is higher when one or more of the following are present:

  • New proposition and product development
  • Bespoke and custom software development
  • Integration of multiple backend systems
  • System rationalisation
  • Retiring end-of-life technology
  • Several suppliers present
  • Outsourced and offshore development
  • Poor quality or inexperienced developers
  • Absence of critical project roles
  • Operating in regulated environments
  • Developing safety-critical applications
  • History of failed projects and poor delivery

My software development is going wrong

You’ll attend a product demo and be either thoroughly underwhelmed at the progress made or simply furious at the sloppiness you see, or you’ll lie awake at night worrying that you don’t know what is being delivered, and when, even approximately or on a prioritised basis. These are major indicators that something is wrong with your software development process.

Regularly cancelled product demos because of overrunning development. Infrequent production releases to actual users. Production releases that go wrong; unexpected outages, data corruption and spiralling bug counts. Unhappy users when they finally get new software versions. Frustrated developers working in a team with low morale and high staff attrition. These are other indicators to look out for.

What developers need

Developers need to know what to do and be able to make the right decisions. Some developers tolerate ambiguity better than others, however they still need a clear understanding of what to do. Experienced developers in small teams with good access to users require less prescriptive requirements; junior developers in remote teams building enterprise software require more. Some folk believe modern agile practices remove the need for software requirements, but this couldn’t be further from the truth.

The further developers are away from end users and business stakeholders, the greater their reliance on written communication. Developers who work without direct, personal user feedback depend heavily on written requirements as their primary source of guidance. By all means, bring users and developers closer together, but when you can’t, you need software requirements. Software requirements are far from perfect, often only a poor approximation of close collaboration and direct feedback. Still, they are necessary to bridge the monumental communication gap that exists in modern software development. This is particularly true in large corporate and enterprise settings.

Occasionally, the exact thing to build is known but the software requirements lack detail, such as acceptance criteria, business rules, testing scenarios, and other important things. You’ll see developers becoming blocked in their work, needing conversations to iron out the details, and then waiting for system access and environments to be made available. Better software requirements, adequate time to review them as a team, and consideration for third-party dependencies are usually the solution.


Better software requirements

Better software requirements are a team sport that requires the participation of each team member. The idea that someone, single-handedly, can write down exactly what the developers should do to create a market-leading software product is unrealistic. It also shows contempt for your teammates who have much to offer in the process.

Many concepts, ideas, and opinions feed into the early stages of requirements gathering, each involving deliberate sense-making and consideration before informing the technical team’s work. It’s a messy and complicated process, but software requirements are a human affair. Read on for some practical advice about how to better navigate this endeavour.

Gathering software requirements

“Have you got the requirements yet?” is a well-intended but misguided question anxious stakeholders ask when faced with an urgent need to deliver some kind of system, unrealistic deadlines and the desire for progress. If only the requirements were complete, then the real work could commence, goes their thinking.

Unfortunately, software requirements aren’t necessarily known in advance, they aren’t located in the heads of the stakeholders, and they aren’t solicited in back-to-back Zoom meetings, nor are they lying in hallways waiting to be collected like eggs from a chicken coop. Equally, going around saying “Tell me what you want” is a pretty terrible approach and certainly won’t provide the answers needed or hoped for. You wouldn’t need to hire a business analyst if it really was this easy.

Requirements gathering isn’t a simple linear process, and unfortunately, it often can’t be planned and executed in a deterministic manner. Sure, you can schedule two weeks of stakeholder interviews on a project plan, and you will find out a heap of stuff, but you may not get the answers you need to build the kind of software the business demands, and users want. Interestingly, IT system builds often represent a deep, unmet organisational need and there is immense value in accurately clarifying and surfacing it.

Quickly capturing enough of an initial backlog for the early sprints demonstrates progress, aligns developers and allows work to commence. Everyone starts to feel better. The risk of rework and the ability to pivot is considered when compiling the initial backlog, given the high chance of this early work being off the mark. Importantly, commencing development secures the time and remit to commence the real work of understanding deeper organisational needs through immersion in day-to-day interactions with users and stakeholders over time.

Writing software requirements

Expecting developers to magic something from a vague idea or a one-sentence user story isn’t a particularly helpful approach. Technical staff are great at solving relatively well-bounded technical problems but dire at stepping into the customer’s shoes.

Unfortunately, many software requirements are just too vague and open to interpretation. Select a few requirements from your team to inspect. Are they clear and effective? Do they convey the outcome required? Please tell me what you find.

What’s usually missing is all the context. Here are some helpful things to consider including alongside each software requirement, mindful not to slip into ‘solution mode’ when doing so:

  • Background information
  • Acceptance criteria
  • Alternative scenarios
  • Business rules
  • Error handling
  • Illustrations, UI mockups and designs
  • Research data and user feedback
  • Related product features
  • Clear directions stating what technical approaches are acceptable
  • Assumptions, constraints
  • Anything else relevant

Performance attributes and non-functional requirements (eg. accessibility, interoperability, redundancy) should also be defined. It is usually best practice to document these centrally for an overall product or product sub-system, referencing them from individual requirements. Technical details such as API endpoints and message schemas should only be included if they are significant or enhance readability.

Read more: Painless Functional Specifications

Reviewing software requirements

Software requirements aren’t something that get written and handed over to the technical team, never to be discussed again. Rather, they are actively ‘developed’ over time in a process best seen as an ongoing conversation between users, developers and testers (nb. often a business analyst or product owner is the proxy for the user in this arrangement).

The technical team should be involved fairly early in reviewing the emerging software requirements and consulting on the feasibility of building such things. Their involvement helps form an opinion on the classic ‘iron triangle’, ie. time, cost and features. Management are usually very happy for an informed view, and any estimates can be revisited as the software requirements become better known.

Software requirements are best reviewed as a team in frequent, regularly scheduled review sessions. Once or twice a week usually works well for a team of up to 5 developers. The idea is to ensure each software requirement has been reviewed before development commences so that significant clarifications have been bottomed out. Incomplete, ambiguous and poorly written requirements should be identified quickly and marked as unsuitable for development.

Practically, here’s how to conduct a good review meeting. Open each software requirement and have someone read it aloud to the team. Each person should use their professional expertise to assess what they hear critically. Is the requirement clear and unambiguous? Are business rules, edge cases and third-party dependencies known? Do we know enough to build and test it? If not, are we still happy for a developer to commence working on it? These are all good questions to ask. Developing a standard set of review questions together as a team is often a helpful exercise.

Unfortunately, some developers can quickly become bored when asked to review requirements that have little relevance to their specialised skillset. The solution is to triage all new software requirements as a team, identifying what technical skills will likely be required to build each requirement. Reviews are then performed in smaller breakout sessions with only the relevant parties. Sometimes, these reviews are performed ‘offline’ or asynchronously through ticketing systems and collaboration platforms. Either way, ensure that testers are always involved, irrespective of how reviews are conducted or the technical skills required by the software requirement.

Often, the same software requirement goes through several rounds of reviews, successively clarifying and elaborating it each time as more becomes known. The result isn’t a detailed specification, unless you are manufacturing an aeroplane engine; rather you are gathering together enough information for each requirement so that developers can resolve the remaining details themselves and work without becoming blocked. Better software requirements are best described as ‘developed’ rather than ‘written’.

Read more: Card, Conversation, Confirmation

Implementing software requirements

Developers prefer to solve technical problems rather than hash out incomplete or poorly formed requirements. They want to apply the breadth of their technical knowledge to design appropriate solutions and learn new skills along the way.

Developers also enjoy working in a “flow state” for extended periods of time, focusing on solving the problem at hand. Every software developer has experienced being so immersed in their work that hours literally flew by without being noticed. This is where the real enjoyment of software development lies.

Picking up some work to find the underlying need hasn’t been fully articulated is annoying, although often easily addressed with a quick conversation. Continuing work to hit another blocker that can’t be as easily fixed is more irritating, requiring emails and perhaps more conversations. Sometimes, the work becomes completely blocked as the need for further analysis emerges. Seeking further clarification is required.

While some developers can perform the analyst role, they really shouldn’t have to. Instead, developers are most effective when they can resolve the right amount of detail upfront or early in their work, before proceeding with the software development relatively unhindered. Developers thrive when they can determine their own work schedule.

Working in a focused manner is not a ‘perk’ of being a software developer; instead, it’s an inherent part of working effectively on complex problems in complex domains. One unscheduled phone call or interruption, even just the threat of that, can be enough to break concentration. Being able to focus is a reasonable expectation for any knowledge worker.

The measure of good software requirements is whether developers can schedule their own day and work in a focused manner without unexpected blockers or the need for unplanned conversations. Ceremonies, appropriate conversations and alignment of work should only happen because they are valuable to the software developer, rather than imposed by a framework or overbearing agile coach.


Special considerations

Remote development

Developers struggle when the people who can answer questions are not readily accessible or available. Remote developers struggle more due to their lack of co-location and increased reliance on written communication. Difficulty getting answers in real-time, waiting on emails, and language barriers are all things a local development team would never face.

Collaborating effectively across time zones is possible, but shifting to asynchronous communication and mostly written forms of communication is required. Remote teams working similar hours to the local business can more readily act as an extension of the local workforce, whereas other offshore teams perform best when responsible for end-to-end work, avoiding part-finished work bouncing between distributed teams.

Regardless of the approach, the remote team must work with you rather than against you. Accommodate their needs as best as possible and avoid the temptation to impose local processes without assessing their suitability first. Meet up in real life, at least once, to put names to faces and start developing some human relationships.

Neurodiverse developers

‘Can’t you just collaborate for once’ said the frustrated manager, perhaps not an uncommon experience for the developer it was directed at. But maybe they really could not collaborate, at least not in the way expected of them. Noisy office space, unstructured meetings, verbal instructions, business jargon and lack of written documentation might have been factors at play.

Around 1 in 6 people working in technology are believed to be autistic, a much higher rate than the general population (Tony Atwood, 2022). Neurodivergent traits can be incredibly valuable for technology workers, hence the increased prevalence; however, these individuals see, hear and process the world differently, sometimes drastically so. ADHD and other conditions often co-exist, adding to an already complex situation. Difficulties can arise, both interpersonally and at work, such that autism is a recognised disability under UK law.

Professional understanding, diagnosis and access to support are relatively recent occurrences, not widespread but certainly more prevalent than they were 20 years ago. Many neurodivergent individuals have lived their entire adult life flying under the radar, undiagnosed, unsupported and coping as best they can. Can you really be sure your ‘difficult developer’ is wilfully acting this way? You can’t.

Certainty, structure and routine are everyday needs amongst neurodiverse individuals, coping mechanisms for high levels of hidden anxiety. By their very nature, software requirements reduce ambiguity and clarify what’s required. Good software requirements do this to a level sufficient for the developer to work without worry. A welcome relief for many individuals.

Read more: The lost years of a young software engineer

Agile transformations

Some organisations don’t want agile. They say they do, but shop floor behaviour indicates otherwise. Unless the CEO personally spearheaded agile ways of working; small, autonomous, cross-functional product teams who release software frequently and practice continuous improvement are aspirational at best.

Agile remains one small initiative in the product or technology department, ready to be overridden, deprioritised or ignored when something more important comes along. This is the reality of building software in large enterprises, particularly if they aren’t in the business of technology.

Developers find this incredibly confusing, particularly when an agile coach espouses individual empowerment and bottom-up transformation despite an organisation that acts otherwise. The internal conflict of trying to work one way versus being inadvertently prevented from doing so can cause irritability, tiredness, fatigue, burnout and poor mental health. High levels of staff sickness and turnover are good indicators of this.

Software requirements are the antidote to confusion, ambiguity and undue personalisation. Slice the requirements thin, call them user stories, practice story refinement, get a solid definition of ready, adopt a rigid Scrum cadence and ask the agile coach to protect your boundaries.


Conclusion

Software development works best when developers interact directly with users. Conversations replace the need for requirements, and ideas and concepts are quickly pushed out the door. However, the further you stray from close collaboration between users and developers, the greater the likelihood of things going wrong.

Large corporations develop software in ways that definitively break the link between users and developers, leaving a communication gap of monumental proportions. Developers who work without direct, personal user feedback are heavily dependent on written requirements as their primary source of guidance. By all means, bring users and developers closer together if you can, but when you can’t, you need software requirements.

Software requirements are a messy and complicated process, but they are also very much a human affair. This guide presents practical advice based on many years of personal experience that your team should consider adopting. Please write and tell me about your experiences with better software requirements.

Questions and feedback: info@bettersoftware.uk


About Frank

Frank Ray

I’m Frank, a former Microsoft C# .Net software developer and owner of Better Software UK, a consultancy specialising in software requirements for agile development teams.

Remote, outsourced and offshore development is difficult to do well, and my time in enterprise software development teams, firstly as a hands-on developer and then as a business analyst, clearly showed me this.

I’ve written extensively about my workplace experiences and dedicated my professional life to improving software development. I discovered that I am autistic in my mid-40s and learned that it’s not uncommon for engineers. It’s also why I understand and communicate with other developers so well.


Frequently Asked Questions


What are software requirements?

Software requirements describe what a software product should do. They represent the needs and wants of product users and broader stakeholder groups and, as such, should be clearly written down and prioritised.

Many formats exist to capture requirements, including use cases, agile user stories, business requirement documents, and requirements catalogues. All are valid and have their place, depending on context and environment.

Software requirements are easy to do poorly and notoriously tricky to get right. Business analysts and requirements engineers are responsible for gathering and managing software requirements; an interesting but specialised job role.

Are software requirements important?

Yes, because without them, chaos happens. Your developers won’t know what to do, so they get stuck and frustrated, or spend their time day trading instead of working. Others may work it out for themselves, building what they think is needed and making important decisions as they go. Even the most ‘agile’ of developers still need a way to clarify what’s required.

Sadly, the business pays the hidden cost of not having good requirements, the very individuals least well-placed to know what’s going wrong. The result is poor quality software, delivered slowly, not working as expected, by frustrated or demotivated developers. Only to be sent back for reworking. Nobody wins, and it doesn’t feel good.

How long do software requirements take?

A good rule of thumb is that for every 3 to 5 developers, you need one full-time business analyst to take care of the requirements. ie. for every month of software development, the software requirements take a month of effort to prepare, which is a 20% overhead.

The software requirements work needs to commence ahead of the development, but just enough to get the developers started. Avoid the temptation to define all the requirements in advance. Instead, most of the software requirements should be worked on in parallel with software development, incorporating learnings and emergent requirements as they become known.

The 20% overhead rule of thumb is completely unscientific, but it has held up as a heuristic pretty well over the years. Mileage will vary; more complex products and bigger development teams require greater coordination; simpler products and smaller teams require less.

What is the difference between functional and non-functional requirements?

Functional requirements describe what the software should do; non-functional requirements, often abbreviated to NFRs, describe how it should do it. Non-functional requirements are also known as ‘quality attributes’ because they relate to the standard of service a user should expect from the software.

Here’s an example to better illustrate the differences. ‘I want to make a payment by credit card’ is a functional requirement; it describes a user action that is part of achieving a bigger goal. ‘The payment should be secure’ is a quality attribute that explains how the payment should be performed ie. securely. Further elaboration is possible by specifying implementation-specific details, eg. ‘256-bit encryption for card details’ and ‘compliance to PCI-DSS’, an industry standard for card payments.

Non-functional requirements often apply to entire software products or whole sub-systems, unlike functional requirements that describe individual functionality. ‘I want to make a payment by PayPal’ is a second payment requirement, but the expectation of doing it securely still applies. For this reason, NFRs are often specified at the product level and then referenced within individual software requirements or inferred as generally applicable. ‘Cross-cutting concern’ is a commonly used term to describe this.

What is a solution design?

Software requirements describe what the software should do; a solution design defines precisely how it will be done. Software requirements avoid specifying solutions and implementation details where possible, allowing the requirements to proceed without undue concern for solution options too early in the process. However, once the software requirements have been defined and agreed upon with the stakeholders, then a solution design can be produced.

Developers won’t need a solution design for trivial requirements because they can simply extend the existing codebase in a familiar pattern and work out the minor details themselves. Existing development standards and design patterns are leveraged. Slightly more complicated changes can still be delegated to the technical team, often overseen by a technical lead or performed in collaboration with another developer. Writing documentation and noting key architectural decisions is a pragmatic way to proceed.

Starting a new software project, substantial changes to an existing application and complex functionality will require a more detailed consideration of the software design. Usually, a dedicated technical architect or an entire architecture function will be responsible for this. Several implementations may be possible, but the solution design clarifies which one to use. Large development projects, junior developers and remote development teams especially value having detailed designs to follow.

What is agile software development?

Agile development builds software in small, frequent increments. Instead of waiting six or more months for a finished product, agile development typically ships a new version every two to four weeks. The most important or valuable features are identified and worked on in each increment, and users come to expect new and improved features on a regular basis. It’s not hard to see the popularity of agile software development.

However, short development cycles and frequent releases bring challenges that require careful management. Better software requirements and solution designs are essential, along with very high testing rigour to ensure software works as intended and without unnoticed regressions.

Unlike waterfall approaches, requirements gathering is limited to the minimum required for each increment, avoiding wasted effort and future rework. Agile teams often favour user stories, a particular format of software requirement, for communicating what the software should do. Stories should be prepared in advance of each increment commencing.

Are software requirements still agile?

Software requirements and solution designs are even more critical for good agile software development, not less. The confusion about this comes from the idea that all requirements and design should be done up front and without the involvement of the technical team, neither of which an agile team should aspire to. Instead, just enough detail at the last possible moment is the right approach, supported by a product roadmap and overarching high-level design to guide the team in their daily decision-making.

Better software requirements in the form of agile user stories are incredibly valuable, and having these prepared ahead of developers picking them up prevents their work from becoming unexpectedly blocked. Not bothering with user stories only works when you have a well-rounded developer physically sitting next to the product owner, both happy to have conversations throughout the day as the work progresses; an excellent arrangement if that’s your situation.

What is the difference between outsourcing and offshoring?

Outsourcing is when you rely on a 3rd party supplier to develop your software, whereas offshoring is when you locate your software developers in a different country. In reality, though, it’s a bit more nuanced. Outsourcing typically occurs when you contract most or all of your development to a 3rd party strategic partner, deciding they can develop software better than your own company, or deciding that you don’t want to be in the software industry yourself. The cost of engaging a strategic supplier should be more than compensated by acquiring access to their expertise and avoiding the opportunity cost of trying to develop that yourself. The outsourcer may, in fact, offshore their own development, in which case you have actually outsourced and offshored your software development. You may not even know this has happened.

When does outsourcing make sense?

Anyone can hire a developer to build some software, but good software development is complex and costly, something many businesses are better off not doing themselves. Outsourcing your technical decisions and software development makes sense when the specialist skills you need are missing internally, particularly if you need them quickly or the nature of your business is not technical. Many companies are not geared up to hire developers and build their own software. You can, however, build strategic and competitive advantage by having your outsourced provider develop market-leading software on your behalf, avoiding the technical risks and expertise required to do so yourself. Strategic partnerships are well placed to advise your business in product decisions and technical matters, whilst agency outsourcing lets you scale development capability. Long-lasting, mutually beneficial relationships can occur in both forms of outsourcing.

Should I offshore my software development?

Certainly, if you want to consciously support developing countries as part of your corporate and social responsibility. Choosing an offshore location with lower living standards and a favourable exchange rate can have a material impact on individuals, families and potentially whole communities.

However, most offshoring is driven by budget holders seeking to reduce development costs, but hiring offshore developers is a risky business with hidden costs that often overshadow the end result. Firstly, know that good-quality developers in a similar time zone and who speak the same language as you will usually cost the same or similar to what your local market already demands. So, the economic case isn’t made like this.

Lower development costs need compromises to happen. Whether it’s far away time zones, junior boot camp developers, language differences, cultural challenges or political instability – these result in attractively low rates. Your economic case for offshoring revolves around being confident in delivering on par with a locally sourced team, despite these compromises. Otherwise, you face hiring local staff to manage offshore staff, tolerating poor quality and longer release cycles whilst feeling stressed, all for a higher cost than expected.

Offshore development can work really well. Just be adequately informed in your decision-making and realistic about what to expect.

What are your favourite offshoring locations?

We favour Eastern Europe, particularly Poland or Romania. The tech calibre is very high, there is a good grasp of English, and working hours are similar to the UK. Perhaps most importantly, Eastern Europe seems to have a culture of excellent software craftsmanship. However, that also means the cost of Eastern European developers is no longer much cheaper than the local UK market, although I would still prefer to hire remotely given the choice.

If you are looking for more significant cost savings, look to Southeast Asia, places like Vietnam and the Philippines that have decent emerging offshore software development industries. Favourable currency rates and lower costs should more than compensate for increased challenges compared to their Eastern European counterparts. Of course, there are no guarantees, and further afield comes with higher risks of things going wrong. Better software requirements can help. There are many Indian offshore development firms; however, they would not be my first choice.