Improving User Experience Through Acquisition

By Colt Whittall, Chief Experience Officer, Department of the Air Force. January 2023.

I speak with a lot of organizations across the Air Force, Space Force and other DOD organizations. Nearly every audience includes acquisition professionals in information technology and/or “owners” of systems. Most of them, if not all, acknowledge that our software and other information technology should provide a better user experience. But one of the challenges they raise is that the technology they deliver is provided by contractors and they are bound to the scope and requirements of the contract. Sometimes they also raise that they don’t have the solicitation language or evaluation criteria to select the contractors capable of providing the best user experience. Then they don’t have the necessary language to include that scope of work under the contract and factor it into performance evaluation and incentives. It is my view that this represents both a blocker to good user experience in government and, if we can solve it, a tremendous opportunity.

The DAF’s SAF/CN organization recently posted a set of Tools, Tactics and Procedures (TTPs) for the Acquisition community to use in information technology related solicitations. These are available now through SAF/AQ’s TTP site on the DAF’s Teams platform and will be updated periodically by SAF/CN, based on actual contracting experience. They were developed based on input over two years from various experts in both government and industry and discussions in the User Experience Community of Interest. This is an informal network of User Experience Professionals from across the DAF that meets monthly and works to improve User Experience across the DAF’s technology portfolio. The language was then refined through coordination by SAF/CN’s strategy team and submitted to SAF/AQ for inclusion as a TTP.

The user experience TTPs consist of sample language for the Performance Work Statement (PWS) and evaluation criteria used in a typical solicitation. The concept is simple. The PWS requires the offeror to include a plan to deliver a user experience that meets mission expectations. This must include how they will measure, monitor and continually improve user experience. Then the evaluation criteria specify how this plan is to be evaluated and weighted in the selection. This approach puts the onus on the offeror to “show their cards” and convince the government they can deliver a mission capable user experience. It empowers offerors to differentiate themselves based on their understanding of user experience. It likewise empowers the evaluators to consider the offeror’s proposed user experience plan in the selection and give appropriate weight to this part of the proposal.

Without this language, acquisition professionals find it difficult to define requirements in a PWS that lead to a mission capable user experience. When the requirements are spelled out, contractors tend to find ways to check all of the boxes, whether they are truly capable in these areas or not. This is clearly not desirable, especially when there is a better option.

That said, other government agencies have produced useful content to guide acquisition professionals and system owners regarding user experience. The best is produced by GSA. It is called the “CX Evaluation and Buying Guide” and can be found below. This is focused on “Customer Experience” more broadly but it applies to User Experience fairly well. For acquisition professionals and/or system owners seeking guidance on how to structure their acquisition strategy, I refer you to this document because it is the best I have seen. The TTP language above is consistent with GSA’s guidance in this document. See CX Evaluation and Buying Guide: https://www.performance.gov/cx/assets/files/CX-Evaluation-and-Buying-Guide.pdf

If you are a government acquisition professional or a system owner, feel free to copy/paste the TTP language into your solicitation. SAF/CN and I would appreciate your feedback, revisions, etc. since we plan to refine this product and improve it.

If you are a government contractor, or an aspiring one, and want the opportunity to differentiate your proposal based on your user experience expertise and capabilities, then feel free to submit this TTP language to your government contracting contacts. Many government solicitations put our Requests for Information and Draft RFPs. These are opportunities to suggest this TTP language be included in the final solicitation when it is released.

User experience of information technology in the DAF will not improve until we select contractors with the capability to deliver the best experience and get those requirements into contracts and performance evaluations. This TTP language is the beginning of that journey. But to scale this up we need your help.

Acquisition Contracting Solicitation Language on User Experience

Purpose

The purpose of this document is to provide Tactics, Techniques and Procedures (TTP) for the Acquisition and Contracting Solicitation Language on User Experience.  This TTP includes a sample template of Warfighter Effectiveness Acquisition and Contracting Solicitation Language that shall be leveraged for solicitations of information technology, including development and sustainment of both IT infrastructure (networks, computing, cloud) and software applications.

Background

Currently, software user experience varies widely among the Department of Air Force’s 1,000 + applications. Many of both legacy software and modern applications fail to meet the desired mission requirements and the lowest standards of commercial application usability. Recent DAF IT Pulse Surveys show Airmen reflect poorly on DAF’s IT systems, despite generous funding to meet capability-based mission requirements. Delivering software of such a low standard in user experience reduces the productivity of Airmen, threatens to undermine the warfighter capability, and introduces unacceptable risk in time-sensitive use cases.

This document provides guidelines on the solicitation language to improve user interfaces in software/applications. Compliance will increase Airmen’s capabilities and efficiencies, save time and future costs for the DAF. These guidelines also help to reduce the risk of developing poor software products that do not appropriately meet the Airmen’s mission requirements.

Content      

Solicitations of information technology, including development and sustainment of both IT infrastructure (networks, computing, cloud) and software applications shall include the following Measures of Merit language under Section L, as applicable.  Section M provides guidance on evaluating and determining the most appropriate software program.

Sample Warfighter Effectiveness Acquisition and Contracting Solicitation Language

Section L: The offeror shall propose a program to drive measurable improvements in the User Experience. This “UX Program” will address the approach to UX in terms of user’s needs, user experience, offeror’s experience and skills, deployment environment, security management, and metrics and tools for evaluating effectiveness. If incentives are to be used in the acquisition, UX elements of the proposal shall be included clearly in the award factors and shall be incentivized/compensated in the acquisition.

Measures of Merit: 

·      Offeror proposal demonstrates an understanding of how UX will enable the delivery of the core requirements

·      Offeror demonstrates how they will deliver the proposed UX improvements as defined and measured in the program

·      Offeror demonstrates that UX targets are integrated into CDRLs and/or the incentive plan and that the incentive plan is aligned properly with UX objectives of the program

·      Offeror demonstrates appropriate UX skills and experience by submitting UX team’s resume and a portfolio of past UX designs developed by the team for previous clients

·      Offeror involves future users in initial and iterative UX discussions to gather improvements and/or functionalities, and reflects these inputs in creating UX mockup(s)

·      Offeror proposes UX mockup(s) to be reviewed by future users/UX specialists (TBC)

As part of this program, the offeror will, to the extent applicable, maximize the utilization of user feedback and telemetry to optimize the development and enhancement of the system.

Measures of Merit:

·      Offeror shall propose target metrics that are aligned with the mission, context and user needs of the system

·      Offeror shall propose a plan to deliver their product(s) or service to meet those targets

·      Offeror shall propose a plan to gather user feedback and use that information to both design the system and continuously improve the system while in operation

·      Offeror shall propose tools and/or techniques to gather user feedback and telemetry, leveraging the existing Air Force feedback and telemetry platforms where appropriate

As part of this program the offeror will, if applicable, maximize the utilization of Human Centered Design (HCD) concepts outlined in DoD Instruction 5000.95 Human Systems Integration in Defense Acquisition that put the Airman/Guardian at the foci of the development effort. The approaches, tools, and methods of the HCD concepts and Section 508 of the Rehabilitation Act shall be applied across the appropriate stages of the systems development.

Measures of Merit:

·      Offeror shall demonstrate within their proposal their ability to join human-centric qualitative insights with deep analytics to shape the desired end-state and value potential for the warfighter.

·      Offeror shall demonstrate their ability to use HCD to confirm desirability, feasibility, and viability to plan the execution of prototypes, concepts, blueprints, and roadmaps for business cases.

·      Offeror shall reflect how they can rewire mindset with agile execution of their roadmap to deliver seamless implementation and adoption while detailing value tracking and continual customer feedback.

·      Offeror proposal shall demonstrate an understanding of HCD and Section 508 of the Rehabilitation Act (Section 508), and how these concepts will be applied to the effort 

·      Offeror shall demonstrate how they will leverage HCD and Section 508 concepts to improve UX, efficacy, and the overall adoption of the program

·      Offeror shall reflect how HCD and Section 508 is applied and ensure pricing addresses these areas

Section M: Evaluation Criteria. The factors that will be used in the integrated assessment to determine best value include:

·      Technical factors and the relevance and score of the past performance determined in part by the delivery of UX and HCD services similar to that in the offeror’s proposed approach. This is to include references to specific UX process, skills, tools, deliverables, methods, target metrics, measurement tools and methods.