Beyond metrics? Utilizing ‘soft intelligence’ for healthcare quality and safety

This was a really interesting study that explored the collection and use of ‘soft intelligence’ in healthcare. Soft data (similar in principle to the small data / big data dichotomy) is the type that derives from sources beyond the conventional metrics and formal knowledge-sharing and management systems.

I can’t do this study justice, so if you’re interested in the topic then grab the full paper.

For context, they highlight a couple of tragic hospital events which involved care-related deaths. In one example the inquiry remarked that “High-level metric … may not be sensitive to the underlying risks. For that reason, it is important to understand what is happening in clinical services themselves” (p20).

In another example, hospital leaders sought “information that appeared to reflect well upon the organization and a corresponding tendency to discount discomfiting data” (p20) and with the corresponding inquiry remarking that “Statistics and reports were preferred to patient experience data” (p20).

One way of moving forward is a change from comfort-seeking approaches to problem-sensing. Problem-sensing approaches should be directed towards identifying fallibilities in organisational systems, structures, routines and norms.

While hard data will continue to be invaluable, they “may not yield full insight into the range of vulnerabilities and fallibilities in organizations” (p20). One reason why is that social practices influence the production of data; noting that what gets reported isn’t the outcome of a neutral scientific process but instead reflects what is observed, how it is observed, and how the data is translated.

This study therefore drew on data from 107 in-depth interviews with senior leaders (managers & clinicians) in the English NHS for insights into the use of soft data. In noting the context, the authors quote a report in the aftermath of the Stafford tragedy “leaders need first-hand knowledge of the reality of the system at the front line, and they need to learn directly from and remain connected with those for whom they are responsible” (p20).

Results

WAY too much to cover in this paper, so the below is a sample only.

Key findings included:

·        Participants readily appreciated the value of soft data and especially for identifying troubling issues that could be obscured by conventional metrics

·        Most struggled, therefore, not in the value of this data but how to access it and turn it into a useful form

·        Some of the dominant methods of using soft data “risked replicating the limitations of hard, quantitative data” (p19)

·        Methods included aggregation and triangulation methods (discussed later) that preferenced reliability of results, or instrumental use of soft data to animate the hard metrics. Problematically, the “unpredictable, untameable, spontaneous quality of soft data could be lost in efforts to systematize their collection and interpretation to render them more tractable” (p19)

·        A more difficult but rewarding use of soft data was how it could disrupt taken-for-granted assumptions about safety and quality rather than aiming for reliability and quantity of findings

·        Thus, “Using soft intelligence this way can be challenging and discomfiting but may offer a critical defence against the complacency that can precede crisis” (p19).

Participants emphasised the challenges leaders had in forming accurate pictures of quality of care at the sharp end. Conventional approaches of revealing the sharp end to the blunt end were seen to be “sometimes slow and cumbersome, and prone to distort or diminish potential hazards” (p21).

Leaders remarked that what happens day-to-day at the ward level tend to be “invisible” to leaders. Some ways they tried to overcome this issue was via use of varied hard metrics – but noting limitations of hard metrics, like adding further noise without amplifying the signal due to an already data-rich system.

Noted was that responses to issues tends to involve adding new initiatives or metrics without anybody asking or demanding that something else be removed or consolidated.

Many agreed that soft data offered a “critical counterpoint to the hard metrics yielded by audits, surveys and performance monitoring, and incident-reporting systems” (p22). Many agreed that there was no substitute for walking around and talking to people and stakeholders directly.

Soft data was believed to offer rich, detailed and specific insights into real or perceived issues and “in at least some organizations, these insights were taken seriously at senior levels in some organisations” (p22; emphasis added….which is quite an alarming comment that in only *some* organisations was this data apparently taken seriously).

Recognising the benefits of soft data also made conscious the limitations and challenges of using soft data. That is, how do leaders make soft data intelligible?

Participants had concerns to the reliability, validity and evidential standing of soft data insights. Serious issues raised during ward visits may be simply one-off and atypical “blips”, or may relate more to the perceptions, motives or dispositions of the individual, e.g. moans and gripes; which according to the participants, may turn out to be actually trivial issues for the organisation.

While leaders did trust staff to accurately relay they clinical concerns, patients and carers were conversely, also seen to “lack such a means of calibrating their concerns, and accordingly the pertinence and reliability of the insights they provided were seen as particularly variable” (p22).

Leaders believed that insights from the sharp end “could not be taken at face value” and instead required active interpretation to assess the validity, scope and importance.

That is, soft intelligence according to the authors is not about data per se but rather a set of processes and behaviours. Indicative of this was how leaders saw collecting soft data as not enough but requiring means to convert it into intelligence; to instil it with meaning.

The authors categorised the methods used by leaders as:

1.      Aggregation. This involved grouping similar reports/issues/insights into the same categories, thereby concluding that the issue was more than a one-off blip and thus may require real investigation.

2.      Triangulation. This involved the use of soft data in tandem with harder metrics to use one to validate the other. For example coupling quantitative temporal trends of quality of care with qualitative insights. Importantly, “the role of soft data was generally subordinate rather than having standing in its own” (p23).

3.       Instrumentalization. Leaders often saw soft data as “poorly calibrated” for diagnosing problems of care. Thus, soft data was used by some more to “add emotional force to an argument”. Soft data was used “instrumentally, as a ‘technology of persuasion” (p23).

Through these means, participants gave a wide number of examples of making soft data “meaningful”. The strategies were not mutually exclusive and two or more were often present in participants’ accounts of the strategies they use. While aggregation “implied that soft data could become useful independently, through the accumulation of multiple soft data, Triangulation saw generating soft intelligence only as a complement to or means of validating data derived from conventional sources” (p23).

An example of soft data is provided via a patient complaint of poor quality of care. The authors note that the complaint provided intel that couldn’t have properly been reduced to metrics. Thus “the idiosyncratic, uncalibrated views of patients or their carers became instead a fresh and untainted source of insight not simply—as in Instrumentalization—a way of adding colour and human interest to dry numbers” (p23).

In order to overcome some perceived limitations of soft data, participants before interpreting the data use aggregation etc., used a process of systematisation. Systematisation sought to become more proactive in the capture of soft data, for instance at the point of discharge via surveys (and not waiting for complaints). This was seen as a way to introduce random sampling and inherent biases of self-selection of data, where people get a voice who may have otherwise not voiced it.

It was believed that “Systematizing approaches like this could ‘tame’ soft data at the point of collection, making them less idiographic and minimizing the risk of bias so that data were more readily useful” (p24).

However, importantly, these translation processes could “tame” soft data too much and thereby lose their valuable and raw insights. Thus, ways to better capture soft data “in the wild” were needed. Participants had varied means. One was to encourage staff, patients and carers to speak out.

In discussing the findings above, it’s said the methods to tame and translate soft data also may introduce some disadvantages of hard metrics. Like, views held more frequently were given more credence than the exceptional views by the few.

Leaders also equated reliability of insights across multiple people as evidence of validity, which may have neglected “rarely articulated” insights. These broad-brush approaches may have omitted important insights that were not as amenable to measurement in summary statistics.

They argue that “An unchecked drift into failure (Dekker, 2012) might therefore occur not necessarily through failure to seek out soft data (though that may happen too), but rather because of defects in the processes and behaviours involved in generating soft intelligence” (p24, emphasis added).

Quoting Weick, they argue that sensemaking isn’t about finding truth and getting things right, but instead “it is about continued redrafting of an emerging story so that it becomes more comprehensive, incorporates more of the observed data, and is more resilient in the face of criticism” (p25)

Use of aggregation and triangulation risks reinforcing bias towards congruity and consensus and resists challenge and disruption. By trying to translate soft data into more broadly applicable knowledge “these approaches give precedence to commonsense views that are plausible and broadly acceptable, over the difficult, counterintuitive, foreign—but potentially very useful—insights presented by a few iconoclasts” (p25).

They note that the “unpredictable, untameable, spontaneous quality of soft data is what gave them their value, but it could be lost in efforts to systematize their collection and make them more tractable” (p25).

They argue that it isn’t the process of formal knowledge management systems collecting insights that corrupts the data, but rather the desire to turn the data into a form that managers can use.

Thus, a strong value of soft data may not necessarily be a diagnostic of sharp end reality and to generate clarity, but rather disruption to “to create a space for multiple knowledges and marginalized voices [ … ] and to deconstruct self-evident concepts” (p25).

In wrapping up the paper, they conclude that:

·        Deriving optimal value from soft intelligence requires more than accessing it; and trying to understand it may be misleading especially if treated similarly to hard data

·        What matters isn’t s much the scope, detail, reliability and clarity of soft data, that is, confirmation is not so important as is disruption

·        Data needs to be seen as much as a way that reveals as much as to hide real work and situations, based on the interpretive lenses

·        Further, “Where soft intelligence challenges the dominant picture, this should be valued as an opportunity rather than dismissed as an anomaly” (p26)

·        Used “intelligently and sensitively”, soft data will be discomforting and disruptive and will often “introduce greater doubt rather than greater certainty” (p26).

Link in comments.

Authors: Martin, G. P., McKee, L., & Dixon-Woods, M. (2015). Social Science & Medicine, 142, 19-26.