This post, which is a revised version of one originally posted on April 17th, 2016, can be downloaded as a pdf file here.
Introduction
In January 2016 Cornwall Council and NHS Kernow (KCCG), partners in the ‘devolution deal’ for Cornwall, published a ‘Health and Social Care integration questionnaire’ under the headline ‘Have your say on health, care and wellbeing’. Both the design of the questionnaire and the administration of the survey left a great deal to be desired. This report presents a critique of both. It also draws on findings from the public events that were held in conjunction with the survey, and offers some lessons for future exercises of this kind.
The questionnaire
A detailed analysis of the Health and Social Care integration survey can be found below. In a nutshell, the findings include the following:
- The survey was not based on a systematic sample of any kind. (It was, however, addressed to all age groups, including children under 11.)
- All the questions were addressed to people receiving care: there were no questions that you could answer as a parent or carer.
- There were three versions of the questionnaire (on-line, paper and easy-read): they were in material respects different from one another.
- The questionnaires were not tried out among the general public, and a number of the most important questions were difficult to make sense of.
- When the questionnaires were published no arrangements had yet been made for a proper analysis of the responses.
And while Council officers will doubtless express their satisfaction with the 2000 or so responses received to the survey, this will represent only 1 in 250 of Cornwall residents.
Who takes responsibility?
This incompetently designed and administered survey appears to have been largely the work of Cornwall Council officers, with NHS Kernow people playing a peripheral role, as witness the answer to a question from the public at the March 2016 meeting of NHS Kernow’s Governing Body:
NHS Kernow colleagues leading on engagement and communications commented on early drafts of the survey. It was then shaped by input from the Single Cornwall Plan steering group, this includes the CCG’s Director of Strategy as the representative for NHS Kernow. The final sign-off came from the Joint Strategic Executive Committee of which the Managing Director and Chair of the CCG are members.
It is not a straightforward matter to discover who among Cornwall Council’s officers was responsible for producing the survey. At the public events (daytime drop-in sessions and evening question-and-answer meetings) members of the Council’s communications team and the Communities and Organisational Development Directorate were seen. It is not apparent from the Council’s website how the latter might have been involved, but the domain of the communications team, according to its web page, includes not only public relations, brand management and leaflet design, but also ‘consultation and engagement’ with local people, to allow them to ‘give their views’ and ‘influence decision making’.
‘Engagement’ is a new name for public participation, which has been around since the 1970s. What seems to have happened here is that the leadership of the communications team successfully staked a claim over ‘engagement territory’, presumably on the basis that they possessed the requisite professional expertise. Evidently this claim went unchallenged: the survey’s many defects did not prevent it from being signed off by senior people in Cornwall Council and NHS Kernow. But practitioners of consultation and engagement need to have some grasp of and training in social research, especially social survey methods, where matters as personal and complex as health and social care are concerned: unfortunately the skills associated with public relations and brand management are very different.
Three different questionnaire formats
The questionnaire was available in three different formats: on-line (with completed questionnaires forwarded to a website in Sweden!), paper (as a pdf to be printed out and filled in by hand if downloaded), and ‘easy-read’. There were significant differences in the questions that they asked, so there were actually three different questionnaires, which will inevitably hinder analysis of the responses.
The survey was manifestly defective in other respects too. It was addressed to people who are receiving services (or who think they might be in future), but not to people looking after those who cannot express their needs for themselves. So the parents and guardians of young children, and carers – looking after people with disabilities, or who have dementia, or who are housebound and lack internet access – were effectively denied the opportunity to ‘have their say’.
And the language of the questionnaires presented problems. Members of a local Patient Participation Group in West Cornwall spent time in the waiting room of their GP practice encouraging patients to complete the questionnaire: they discovered that many patients found the questions difficult to understand and respond to. (See Appendix.) It is apparent that the questionnaire had not been tried out among members of the general public (although we are told that elsewhere in Cornwall members of a PPG, who are of course people with an active interest in health matters, had seen it and commented on it).
For the results of a survey to be of any use, the questionnaire must be designed with an eye to how the responses will be analysed. This one clearly has not been so designed. Several questions (such as ‘What are the three most important things to you when you experience health and social care services and support in Cornwall?’; ‘What would you like to achieve in terms of your own health and wellbeing?’; and the double (or treble) question ‘Do you have any suggestions that would help to improve your overall wellbeing and better meet your health and care needs at less cost?’) are open to being interpreted by different respondents in different ways, so it will be a formidable task to categorize and analyse the responses, especially for someone who was not involved in designing the survey. As of 3 April 2016 (a week after the closing date for responding to the survey) no arrangements had been made for a person with relevant experience to do this.
Comparison of this survey with the Council’s Residents’ Survey carried out in 2014 by the market research company Marketing Means is instructive: it demonstrates conclusively that this one has been formulated and the distribution of questionnaires organized by people with minimal understanding and experience of social research and survey methods. As a consequence there is no way that the findings can be sensibly analysed to guide priorities for the future of health and social care in Cornwall. At best this survey can serve no practical purpose other than as a ‘fishing expedition’ for ideas: a ‘list of mentions’ based on an entirely unsystematic sample is the most we can expect to emerge from it.
The questions that were asked
Q1. Your closest large town
The online and paper versions ask ‘Where is your closest large town?’ We can infer that they are not asking for latitude and longitude, since the paper version adds ‘For example: Truro, St Austell, Camborne, Bude or Penzance’ and the online version offers a drop-down menu with a list of 20 towns on it. Notably, Camborne/Pool/Redruth appears in this list as a single entity, and St Just does not appear at all, although its population is larger than those of Fowey, Lostwithiel and Padstow, which are on the list. So respondents have to scratch their heads and make a judgment as to the largeness of nearby settlements: they can’t simply give the first part of their home address’s postcode, which would of course be a perfectly straightforward thing to do.
The easy-read version asks ‘Which is your closest large town?’ and again gives the examples of ‘Truro, St Austell, Camborne, Bude or Penzance’. Although there is no ambiguity in the question (asking ‘which?’ instead of ‘where?’), this again calls for a judgment as to what counts as ‘large’.
Q2. Your age
All three versions ask ‘What is your age?’, and offer seven age ranges: Under 11 / 11-18 / 19-35 / 36‑50 / 51-65 / 66-80 / Over 80. Evidently the designers of the questionnaire wanted to cover the entire possible age range, but we may reasonably ask them: ‘What response did you expect to get from children under 11?’ While this is clearly a laughing matter, it raises a serious point which is not a laughing matter at all: Who speaks for the children? The questionnaire is addressed solely to individuals who receive services or might do so in the future: it entirely ignores people who care for them. Parents and other carers are treated as though they don’t exist. They will have experiences in looking after children, people with disabilities, people with chronic illnesses, dementia, etc., but no questions whatever are directed to them.
Q3. Ethnicity and ethnic origin
On this topic we find a great muddle. The paper and easy-read questionnaires ask: ‘What is your ethnicity?’ The paper version offers no alternatives or examples from which to choose. The easy-read version does offer some examples: ‘White British, White Cornish, Black British, White Asians, etc.’ (no ‘mixed’ category is offered as an example). But the very concept of ‘ethnicity’ is a challenging one (the author of the easy-read version seems to have been defeated by it!), and indeed the term is not in use in questions asked by the Office of National Statistics for census purposes. Only the online questionnaire asks the question as it should be asked – ‘How do you describe your ethnic origin?’ – and it offers a range on a drop-down menu: ‘White (for example, British, Scottish); Mixed (for example, White and Asian); Asian or Asian British; Black or Black British; Cornish; Other.’ Even here, however, the category ‘Cornish’ seems to have been added as an afterthought: it is scarcely an alternative to ‘White’.
We may wonder why there is an ethnicity/ethnic origin question at all. What is its relevance? (Did the compilers of the survey feel they were expected to include such a question?)
Q4. Long standing health condition
The online and paper versions of the questionnaire ask: ‘Do you have a long standing health condition? i.e. a physical or mental health condition or illness that is lasting, or expected to last, for 12 months or more.’ Both versions ask for a ‘Yes’, ‘No’ or Don’t know’. The easy-read version uses slightly simpler language – ‘Do you have a long term health condition? … that is lasting, or could last for 12 months or more.’ – and one of the choices is ‘I don’t know’. Again, in all three versions this question is directed only towards people who receive services or, one presumes, who might do so in the future: it is not a question for carers.
Q5. Written care plan
The online and paper versions of the questionnaire both ask: ‘Do you have a written care plan? i.e. an agreement between you and your mental health professional or social services to help you manage your day to day health.’ And both offer three alternative answers: Yes / No / Don’t know. The easy-read version shows that an attempt has been made to use plainer language: ‘Do you have a written care plan? For example: Something between you and your mental health professional or social services to help you cope with your day to day health.’
Q6. Three most important things
Again, the online and paper versions of the questionnaire ask the same question: ‘What are the three most important things to you when you experience health and social care services and support in Cornwall?’ Contrast this with the easy-read version: ‘What are the three most important things that you need when you have health and social care services and support in Cornwall?’ In asking about ‘needs’ the easy-read version is taking a less abstract, more down-to-earth, approach.
Significantly, no suggestions are made for what these ‘important things’ might be. Given the emphasis in the publicity on ‘hear[ing] about your priorities’, one would expect the questionnaire to have been designed to elicit these and place them in order, but this is clearly not the case. Moreover, if respondents had been offered some alternatives, such as suggestions emerging from a trial of the questionnaire, they would have gained some sense of ‘what sort of things’ were wanted. A good survey goes at least some way towards creating a dialogue between surveyors and respondents: questions like these do not.
Q7. Health and wellbeing
Under this heading there are some striking differences in language between the online and paper versions and the easy-read version, as we see in the table below. Contrast ‘What would you like to achieve in terms of your own health and wellbeing?’ with ‘What would you like to do that could make your health and wellbeing better?’ The very language ‘to achieve in terms of’ is abstract, ‘in-group speak’, doubtless commonplace in county halls and commissioning bodies but not out on the street and in people’s homes. And the very term ‘wellbeing’ is used nowadays to describe a wide variety of states: notably economic, social and psychological. How were respondents to know which was meant?
And it is a basic error in questionnaire design to shoehorn two (or more) separate questions together, as in asking for ideas to improve wellbeing but at less cost. By all means ask for ideas about improving wellbeing, and ask for ideas for saving money, but these are separate issues and conflating them will confuse rather than assist respondents.
What was the real purpose of this survey?
Surveys are designed and run for different purposes. It is extraordinarily difficult to pinpoint the purpose of this one.
The publicity coming from Cornwall Council stressed that Cornwall Council and NHS Kernow want to ‘know your health and social care priorities’, but the word ‘priorities’ does not appear anywhere in any of the questionnaires. And as we have seen, no attempt was made to get respondents to place the ‘three most important things’ in order of their importance.
Likewise Cornwall Council’s newsroom news release on February 29th said ‘Public views [are] sought on 5 year health and social care plan for Cornwall …’. but a draft plan had not been published and did not accompany the questionnaires.
Again, although the questionnaire is entitled (on the paper version) ‘Health and Social Care integration questionnaire’, it did not actually contain any question about people’s experiences of health and social care services operating alongside each other, and how well-integrated those services were found to be.
One possible purpose might have been to analyse patterns of need, for example to examine whether there is a correlation between need and particular age/sex groups or geographical location, but the questionnaires included no question about the sex of the respondents, and the question about geographical location (the ‘closest large town’!) was so imprecise as to be useless. Importantly, for such a survey to have value it would have to be comprehensive. If it were felt to be prohibitively expensive to send a paper copy to every Cornwall resident (aged from 0 to 80+), some form of systematic sampling should have been employed, as was done with the Council’s 2014 Residents’ Survey. Or target groups could have been identified, perhaps from the lists of patients held by general practices or lists of clients held by social services. None of these was done.
A lack of forethought must also be responsible for the survey’s failure to incorporate questions for parents/carers. This is simply inexcusable. It means that the needs of those who are cared for but cannot express their needs – such as young children, people with certain disabilities, those who are housebound and don’t have access to the internet, those who have dementia – are likely to be overlooked.
Lacking a clear purpose, the only function we can find that this survey performs is that of trawling for ideas: a ‘fishing expedition’. This is perhaps the kindest interpretation that can be put upon this exercise. But fishing expeditions can provide no basis for making policy in the complex fields of health and social care.
A questionnaire incompetently administered
Whatever the purpose of a survey, it needs to be set up and run competently. As the above detailed critique shows, this one has not been.
(1) There should have been no need for a separate ‘easy read’ version of the questionnaire, using plainer language: it was necessitated only by the existence of ‘difficult to read’ versions in the first place. And the person tasked with making it easy to read clearly threw in the towel when he or she came to the question ‘What is your ethnicity?’
(2) A closing date for receipt of responses was published, March 25th. The closing date was then brought forward with no announcement to March 14th, which actually fell before the end of the series of public meetings. At the beginning of March the original closing date was restored, again with no public announcement. But at the Penzance drop-in session on March 8th paper copies of the easy-read version were still showing March 14th as the closing date.
(3) There can be no excuse for not ‘piloting’ a questionnaire among the general public before disseminating it, rather than simply showing it to a small group of health activists.
(4) It is unfortunate that a team from the University of Exeter who have been asked to take part in analysing and interpreting the results were not involved in designing the questionnaire: questionnaires should always be composed with ‘What will the answers tell us?’ in mind.
(5) Finally, it is not clear what advantage has been gained by employing a firm whose headquarters are in Sweden. Who had that idea? It may be that had a firm closer to home been employed some of the errors listed here could have been avoided.
Learning from the public events
Following a series of drop-in displays and question-and-answer sessions around Cornwall, a report presented to Cornwall Council’s Health and Adult Social Care Scrutiny Committee on 5 April 2016 listed a number of ‘topics … building on the emerging themes from the survey’:
- What’s good about health and care in Cornwall now?
- What’s not good about health and care in Cornwall now?
- How can we join up services to better meet your needs?
- How can we best use the resources and local assets we have got?
- How can we help you start well, live well and age well – away from hospitalised care?
- How can we help you access the right care in the right place at the right time? i.e. GPs/doctor’s surgeries, minor injuries and urgent care, operations/surgery, social care, mental health, community hospitals
These questions are manifestly much easier to understand than many of those in the actual questionnaires. Talking to people is often a very good way of clarifying one’s own thoughts and expressing them in plain language.
Conclusions: Lessons for future surveys
(1) It is important to be clear about the purpose of the survey. What do you want from it? As we have seen here, in the present case the aims were variously described as ‘knowing your priorities’, seeking views on a ‘health and social care plan’, and finding out about ‘integration’ of services, but there were no questions about priorities, there was no draft plan to comment on, and there were no questions about people’s experiences of service integration or lack of it.
(2) It is crucial to decide how to sample your population. Do you want a sample covering the whole population that you can generalize from with confidence? Are there distinct groups you want to cover, e.g. carers, parents, women/men, people with experience of hospitalization, people with long-term health conditions, people living in care homes, people who are housebound?
(3) The people who are going to analyse the results should be involved in designing the questionnaire. Among other things, they will always be asking: What will we do with the answers? What will they tell us?
(4) Draft questionnaire(s) should be tried out – piloted – on the general public, not on people who are already well-informed about the subject.
In conclusion, a point needs to be made about the language used by officers and other professionals. Their current preference for ‘engagement’ over ‘public participation’ and ‘public involvement’ not only writes the public out of the term, literally: it places the initiative with the body concerned. The Council and the Clinical Commissioning Group are implicitly seen as active, with officers doing the engaging: the public is passive, on the receiving end, being engaged with. The effect is to reinforce an official mindset that already finds it difficult to cope with initiatives and criticism that come from watchdog groups and others in the public realm. Can we get back to ‘public participation’, please?
I do not doubt the goodwill and sincerity of those who organized the present survey. But the official mindset has not served them well. And they have to realize that undertaking a social survey is not a job for the inexperienced, however well-meaning and enthusiastic they may be.
Peter Levin
© Peter Levin 2016. All rights reserved.
Appendix: Notes from a general practice waiting room in West Cornwall
The following notes have been supplied by members of a Patient Participation Group in West Cornwall, and are reproduced here without any editing or alteration.
HEALTH AND SOCIAL CARE INTEGRATION QUESTIONNAIRE
Several members of our Patient Participation Group took it in turns to spend time in the waiting room of our practice, encouraging patients to complete the Health and Social Care survey.
Whilst we enjoyed spending the time talking to and listening to what patients had to say to us, we found that the survey was neither clear nor user-friendly in language, and patients found it difficult to answer. In particular, we noted the following:
Q1 Where is your closest large town?
It would have helped to have a list to choose from, rather than just five examples. Were they meant to choose from those?
Q3 What is your ethnicity?
Patients were unsure what to write here – a list of options would have made this easier to answer
Q5 Do you have a written care plan?
Patients felt that the survey was not relevant to them if they did not fit in this category
Q6 What are the three most important things to you when you experience health and social care services and support in Cornwall?
As we were in a GP practice, patients instinctively commented mostly about their experiences of general practice, and couldn’t think much further than that
Q7a What would you like to achieve in terms of your own health and wellbeing?
Patients found that a strange question – they just wanted to be fit and well! They also used these questions to comment on their own experiences, though how that will be used in analyzing the survey remains to be seen
Q7b What are you already doing towards that?
Patients found that difficult to answer, and weren’t clear what they were supposed to say
Q7c What additional help might you need?
Patients found that difficult to answer too. They weren’t sure whether they were meant to say “more money”, or “more disability aids”, or something else they didn’t know
Q8 Do you have any suggestions that would help to improve your overall wellbeing and better meet your health and care needs at less cost?
Patients found this a baffling question as it asked three things in one sentence. One lady responded: “What the hell is that supposed to mean?”
Other comments
a) Through whose eyes?
Some people wanted to reflect on health and social care from a carer perspective, and there was little opportunity to do that unless they pretended to be the person they cared for. The same applied to parents of young children.
b) What is ‘well being’?
Generally, the term ‘well being’ is not one used by the average lay person; neither does s/he find it easy to identify their ‘health and care needs’, and certainly not ‘at less cost’, because they don’t know what everything costs anyway
c) A lost opportunity?
I think we as members of the PPG found this an interesting exercise but also perhaps a lost opportunity. A few people managed to express ideas for the way forward whether or not this answered a particular question. How the analysts will use this also remains to be seen. Had the survey been better written and presented, you might have gained really helpful insights into how the community feels about health and social care. As it is, we think you will find it very difficult to use this data effectively or gain meaningful findings from analyzing the data you receive. So it is rather disappointing.
FC, CG and MB 21.03.16