Knowledge areas BABOK Guide v3, chapter 8
Solution Evaluation
In short This knowledge area covers how you check whether a solution, or part of one, is delivering the value it was meant to, what is holding it back inside the solution or in the wider organization, and what should be done to get more value from it.
Based on: IIBA, BABOK® Guide v3, chapter 8 “Solution Evaluation”; Watermark Learning, BABOK v3 Study Tables; Yu. A. Maglinets, Analysis of Requirements for Information Systems (in Russian), lecture 9; D. Paul, D. Yeates, J. Cadle (eds.), Business Analysis (BCS), 2nd ed., ch. 13.
On this page
Purpose#
A delivered solution is a hypothesis about value. The BABOK® Guide uses this knowledge area to test it: you analyze the value the solution really provides, identify the constraints that stop it from providing more, and recommend actions. The evaluation usually focuses on one component of the enterprise (a system, a process, a product), not the whole enterprise. It can be applied to a solution in production, a pilot, or a prototype.
Through the lens of the Business Analysis Core Concept Model, you recommend changes to the solution or the enterprise to realize the potential value, assess how well the solution meets the need, measure performance and analyze why value falls short, elicit performance and value information from stakeholders, and consider the context when choosing measures and identifying limitations.
Tasks#
The BABOK Guide defines five tasks.
8.1 Measure Solution Performance. You decide how to measure the solution’s performance, including how it supports business objectives, and you collect the measurements. You produce: solution performance measures.
8.2 Analyze Performance Measures. You interpret the measures to understand the value the solution delivers and whether it still meets the current business need. You produce: a solution performance analysis.
8.3 Assess Solution Limitations. You look inside the solution for problems that prevent it from meeting the need, such as defects, missing functions or poor fit. You produce: a description of solution limitations.
8.4 Assess Enterprise Limitations. You look outside the solution, at culture, structure, processes, skills and other systems, for factors that stop the enterprise from realizing the solution’s value. You produce: a description of enterprise limitations.
8.5 Recommend Actions to Increase Solution Value. You identify and recommend what the enterprise can do to get more value: change the solution, change the organization around it, or retire or replace it. You produce: recommended actions.
Key inputs and outputs#
The Watermark study tables summarize the flow as follows (simplified):
| Task | What goes in | What comes out |
|---|---|---|
| 8.1 Measure | Business objectives from strategy analysis; the implemented solution | Performance measures |
| 8.2 Analyze | Potential value from strategy analysis; performance measures | Performance analysis |
| 8.3 Solution limitations | The implemented solution; performance analysis | Solution limitations |
| 8.4 Enterprise limitations | Current state description; the implemented solution; performance analysis | Enterprise limitations |
| 8.5 Recommend | Solution and enterprise limitations | Recommended actions |
Techniques you will use most#
- Metrics and KPIs and Acceptance and Evaluation Criteria: define what “performing well” means.
- Data Mining, Survey or Questionnaire, Observation, Interviews and Focus Groups: gather performance data and user experience.
- Benchmarking and Market Analysis: compare performance with others.
- Root Cause Analysis and Risk Analysis and Management: explain gaps and judge their consequences.
- Organizational Modelling, Process Analysis, Roles and Permissions Matrix and SWOT Analysis: examine enterprise limitations.
- Lessons Learned and Item Tracking: capture and follow up issues.
- Decision Analysis, Financial Analysis and Prioritization: choose and justify recommended actions.
Practical tips#
Separate solution limitations from enterprise limitations#
When value falls short, the instinct is to blame the software. Always check both sides. A booking system that works perfectly will still fail if staff are not trained to use it, or if the process around it was never changed.
Measure against the objectives, not against the requirements#
Requirements describe what was built; business objectives describe why. A solution can meet every requirement and still miss the objective. Set measures that trace back to the objectives defined in Strategy Analysis.
Treat benefits as something to manage#
The BCS book describes benefits management and realization: map how each enabling change leads to business changes and then to benefits, review expected benefits at each decision point, and report on realized benefits after delivery, sometimes months later. This is the same concern as solution evaluation, seen from the business case side; see Building a Business Case and Financial Justification.
When the question is “buy, commission or build?”#
Maglinets (lecture 9) adds practical guidance for evaluating packaged versus custom solutions. He names three options rather than two: buy from a vendor, commission a custom solution from a development firm, or build in-house. The main criteria are total price (purchase or development, implementation and ownership), how unique the business is, and the level of service available. In practice, organizations often combine options, for example buying a package and then maintaining it with an internal team.
He describes three strategies for choosing a solution:
- Requirements analysis: write requirements for the product you will select, not only for one you will build, and ask vendors to demonstrate how their product meets them. Decide the scope, breadth and depth of the requirements and the resources the selection needs.
- Gap analysis: model the current state (including existing automation), the target state and the transition, then compare candidate products with the target.
- Best-practice approach: skip detailed modelling of the current state and start from the practices built into the candidate product, adapting the product as little as possible.
He also outlines a fast selection process from a published case: a small selection team with mixed skills; well-prepared demonstrations from a limited number of vendors, who get no access to the team beforehand; a shortlist of three; and a final vote on two factors, functional fit and the quality of the implementation team, using a simple three-point scale. See also Vendor Assessment and Requirements for Off-the-Shelf (COTS) Solutions.
Example: six months after launch at The Workshop#
The Workshop (a fictional chain of 12 bike-repair shops) launched online booking and status notifications. The evaluation finds that online bookings grew as planned, but status calls to the front desk barely dropped.
- Solution limitation: the “repair finished” message is sent, but no message is sent when a repair is delayed waiting for parts.
- Enterprise limitation: mechanics update status only at the end of the day, because the only terminal is at the front desk.
- Recommended actions: add a “waiting for parts” notification; give each workshop a tablet at the bench; make status updates part of the repair checklist; re-measure after two months.
Sources and further reading#
- IIBA. A Guide to the Business Analysis Body of Knowledge (BABOK® Guide), version 3. International Institute of Business Analysis, 2015. Chapter 8 “Solution Evaluation”.
- Watermark Learning. BABOK v3 Study Tables. 2016. (study aid for the BABOK Guide): task tables for chapter 8.
- Yu. A. Maglinets. Analysis of Requirements for Information Systems, lecture notes (in Russian). Siberian Federal University, 2007. Lecture 9 (buy, commission or build; three selection strategies; fast selection case).
- D. Paul, D. Yeates, J. Cadle (eds.). Business Analysis, 2nd ed. BCS, The Chartered Institute for IT, 2010. Ch. 13 (benefits management and realization).