This post can be downloaded as a pdf here.
Background
Work is currently going on to produce a health and social care plan for Cornwall and the Isles of Scilly. It is officially known as a Sustainability and Transformation Plan (STP). A draft Outline Business Case (OBC) was published in October 2016, followed shortly after by what was described as a summary of the OBC, entitled Taking Control, Shaping our Future. In that document some elements of the OBC don’t appear, while others appear to have been added. The latter publication incorporated a questionnaire-based survey, the STP survey.
The STP survey was the second to be undertaken within the space of a year. In January 2016 NHS Kernow (Kernow Clinical Commissioning Group) and Cornwall Council published a health and social care integration questionnaire. As a survey questionnaire this was seriously defective in a number of ways, as I described in a report for West Cornwall HealthWatch entitled How not to run a health and social care survey.
The recent STP survey questionnaire put serious problems in the way of anyone trying to respond to it. In a post in January this year I set out the reasons why I was finding it so difficult to answer the questions in it. Last month (March 2017), Cornwall Council’s sub-committee on the STP, on the basis of the evidence they received, said they found the survey to be ill-conceived’ and ‘unprofessional’. So what can be done with the responses received to it? In March the STP team produced two reports on the engagement process, an interim ‘update report’ and a ‘final report’, which drew heavily on the responses to the STP survey. In the following sections of this report I examine the final report and ask what credence can be attached to its conclusions, given the defects of the survey.
Nearly half the working-age respondents were health and care professionals or support workers
The update report tells us that 1896 people completed the survey. But the final report reveals that 30% of them – i.e. around 570 – were a ‘health and care professional or support worker’. So only 1330 or so were likely to approach the survey primarily as members of the general public. Of the 1896, 1258 were of working age (19-65). These working-age respondents will have included nearly if not all of the health and care professionals/support workers. So that group will have contributed around 570 to the 1258 working-age respondents. In other words, nearly half of the working-age respondents (45%) were health and care professionals or support workers.
People employed in health and social care amounted to around 13% of the working-age population of Cornwall in 2011. So they were considerably over-represented among respondents to the survey. By no stretch of the imagination, then, can the survey be said to have been a survey of a representative sample of the general public.
The response to the survey was tiny
To get a sense of scale: the figure of 1896 is about one-third of one per cent of the current resident population of Cornwall and the Isles of Scilly (around 550,000) so it’s a tiny sample. Notably, the figure of 1330 non-health or social care workers who responded to the survey is little over half the 2450 who completed the previous health and social care survey in early 2016, further evidence of the off-putting nature of this latest survey.
The reports present very little quantitative data
We see that in the final report there is a remarkable dearth of quantitative data. Other than a breakdown of respondents by age, and an unclear map showing responses by postcode, a pie chart on p.13 showing ‘First choice priority’ is the only significant quantitative evidence presented. We are told: ‘Overall, people responded favourably to the priorities set out in the Shaping our Future documents and many thought it was hard to argue with the positive approach.’ Since those priorities, as set out in the survey, mostly took the form of ‘motherhood and apple pie’ aspirations, difficulty in arguing against them should not be surprising.
Very little analysis is put forward
Even with the limited data collected, some search for correlations would have been possible. Were there significant differences in responses by age or geographical location? Asking such a question could have generated useful information about the likely impact of future ageing of the population or difficulties with physical access to healthcare facilities in certain parts of Cornwall. And were there significant differences between health and care professionals or support workers and the general public? The ‘top priorities’ could have been very different for the two groups, and indeed it is hard to see the man and woman in the street spontaneously saying what is needed is ‘System reform to achieve better care’ or ‘Preventing and improving population health’ (sic) as in the pie chart below. It is a matter for regret that no search for correlations appears to have been carried out.
This approach lends itself to ‘cherry picking’
There is a striking illustration of the way that this approach allows the writers of reports to pick and choose what they include. The final engagement report says: ‘Respondents said the top priorities should be ‘Prevention and improving population health’ and ‘Integrated care in the community’’. If you look at the pie chart below you will see that another priority, ‘Transforming urgent and emergency care’, tied for second place (at 19%).
Evidently the author of this document felt able to pick and choose for presentation the information that supported a particular case. That is not an acceptable way to present the results of a survey.
Impressions are no substitute for facts
Instead of figures, what we find – on almost every page of the engagement report – are impressions: ‘Many people commented …’, ‘Many said …’, ‘People wanted …’, ‘Others thought …’, ‘People felt …’, ‘Many respondents said …’,‘An overwhelming majority of people agreed …’. We are very rarely given the actual numbers. We are never told how many is ‘many’. We must depend on the author’s predilections and judgment, his interpretation of responses and their impression of how close together these responses are. This is what allows them to be lumped together. The language used and the instance of cherry picking from the pie chart, described above, must raise doubts as to the quality of analysis and interpretation.
Using impressionistic language like this allows cherry picking of survey responses that support a particular view. Considerable passages of the survey questionnaire and the reports are couched in ‘management-speak’. This language may be appropriate shorthand for internal documents, but it is not appropriate for those that are public facing. The use of this style may in itself influence the choice of responses to be quoted in the report, making it more likely that those using a similar linguistic style will be chosen. At the time of writing it is understood that the completed questionnaires will be made available for independent scrutiny, so it may be possible to discover whether this was indeed the case.
The University of Exeter: a key player?
The introduction to the final report has this to say about its authorship:
This report has been written by Garth Davies, Associate Director, Communications and Engagement for the Shaping Our Future programme but with special thanks to the Social Innovation Group at the Centre for Geography, Environment and Society at the University of Exeter for their data analysis and independent view on the findings. Dr Michael Leyshon at the University of Exeter has confirmed that the report is a ‘fair account of our analysis’.
Suffice it to point out (1) that the report does not actually present any data analysis; (2) that Dr Leyshon, a social and cultural geographer whose academic profile does not mention expertise in the NHS or social survey research methods, was not involved in the design of the survey questionnaire; and (3) that the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care, South West Peninsula, which includes the Universities of Plymouth and Exeter as well as local NHS organizations, and undertakes high-quality applied health research focused on the needs of patients and supports the translation of research evidence into practice in the NHS, was not involved in any aspect of the STP survey process.
Evidently what we have here is an attempt to give the STP survey an academic stamp of approval which it does not deserve.
‘Engagement’: At best nothing more than a case of unsystematic communal brainstorming
The process by which those in charge of the STP attempted to engage with the public took two main forms: a series of meetings and the STP survey. The meetings provided an opportunity for local people to give vent to their feelings about health and social care issues, such as the closure of community hospitals, and these have been reported, albeit in a general and impressionistic fashion, in the final engagement report.
As a survey, this exercise has been seriously deficient. At best it has amounted to nothing more than a kind of unsystematic communal brainstorming. Such an exercise may be productive in terms of ideas and highlighting matters that have been overlooked, but it is not a serious tool of social investigation. That much could have been predicted from the questionnaire itself, composed as it was of unanswerable questions, leading questions and questions that look like multiple choice but don’t actually offer a choice (such as questions which ask ‘To what extent …?’ but don’t offer a scale on which to register ‘extent’).
Governance issues
Work on the STP is in the hands of a team of officers under the supervision of a Transformation Board. The Board has four leading members: the Chief Executives of Cornwall Council (who is in the chair), Royal Cornwall Hospitals NHS Trust and Cornwall Partnership NHS Foundation Trust, together with the interim Chief Officer of NHS Kernow (Kernow Clinical Commissioning Group).
There are two important questions to be asked about this organizational structure:
(1) How was it that the design and implementation of the STP survey came to be entrusted to a specialist in public relations, self-identified as Associate Director, Communications and Engagement, and an employee of the Royal Cornwall Hospitals NHS Trust, rather than to someone with experience in social surveys and research methods?
(2) Over the past year West Cornwall HealthWatch has pointed out the inadequacies of the surveys that are supposed to underpin the Sustainability and Transformation Plan, and in an attempt to be constructive we have actually produced a set of guidelines for managers on the design and use of survey questionnaires. Why has the Transformation Board shown no sign of taking on board what has been spelled out to them?
We don’t know what the process was that led to the assigning of a social research project to a public relations specialist. But the failure to learn from the experience of the earlier health and social care survey is seriously worrying. If there is one thing that an innovative project, such as transformation, calls for it is the ability to learn.
The inability of an organisation to learn has a number of unhappy consequences. Mistakes are repeated, with a consequent waste of time and money. Communication channels become narrowed or blocked off, creating a hierarchical structure that supports the status quo. This prevents learning from other people engaged on a similar task elsewhere who are doing things differently. Addressing the public inappropriately with ‘management-speak’, using words such as ‘theme’, ‘vision’ and ‘priorities’, may lead to further spiralling misunderstanding and frustration. We have seen all of these in the present case.
When it comes to public engagement, we need to ask whether the four leading official bodies actually ‘get it’. NHS Kernow’s Governing Body includes a Lay Member, who is supposed (by law) to help to ensure that, in all aspects of its business, the public voice of the local population is heard and that it responds in an effective and timely way to feedback and recommendations from patients, carers and the public. The individual concerned worked for 31 years in the NHS, and at the point when he retired was Chief Executive of two primary care trusts. While his experience may well make him a valued member of the Governing Body, we may legitimately question whether it enables him to speak with authority for the public.
Recently the Director of Healthwatch Cornwall (no connection with West Cornwall HealthWatch!) has been invited to join in the deliberations of the Transformation Board. While this arrangement may be of value, the fact is that she has spent 18 years in the NHS in a variety of senior management and leadership roles. Again, we may legitimately ask whether such experience is an unmixed blessing when it comes to listening to people’s experiences of publicly funded health and social care services and thereby contributing these to the decision-making process.
To negotiate between the institutional world of health and social care provision and the ‘real’ world in which we all live requires an openness of mind along with the ability to learn and the ability to mediate between very different cultures. It is not immediately apparent that long experience in the institutional world confers and demonstrates such openness and abilities.
Postscript
The final engagement report has a message on branding. It says: ‘To give our plan a strong sense of local identity going forward we are referring to our five year plan as ‘Shaping our Future’ rather than the STP.’ We may justifiably wonder how excluding any reference to ‘health’, ‘social care’, ‘plan’ and Cornwall and the Isles of Scilly is calculated to impart ‘a strong sense of local identity going forward’.
Peter Levin
9/4/2017
Update
We have recently learned that since the final engagement report was published, the STP team has taken on additional expertise in the form of Dr Lou Farbus. She is the NHS’s Regional Head of Stakeholder Engagement and is currently on secondment to the Cornwall and Isles of Scilly STP team. She has many years’ experience of teaching questionnaire design and applying this skill specifically to service redesign programmes. She is currently in the process of putting together a package of engagement related training modules, to include questionnaire and survey design, which will be delivered to relevant members of the STP team.
West Cornwall HealthWatch warmly welcomes Dr Farbus in her new role, especially in the light of our experience with STP engagement so far, and we shall be glad to cooperate with her in any way we can. It is heartening to learn that someone in NHS Kernow has – albeit belatedly, we have to say – taken our point about the need to strengthen the STP team to handle engagement and surveys.
13/4/2017