Questioning a Questionnaire
In October 2016 NHS Kernow published a ‘Have your say’ questionnaire on the subject of NHS 111 and out of hours service integration. Was it fit for purpose? This post summarises my assessment on its fitness, or lack of it. The full report, which again comprises the original questionnaire together with annotations, can be downloaded in pdf format here. In this post, the annotations are shown in blue.
NHS Kernow’s goal is to commission an excellent health care system for Cornwall and the Isles of Scilly, to improve people’s health and reduce health inequalities.
This ‘motherhood and apple pie’ mission statement – with which no-one could possibly disagree, of course – has nothing to do with the particular problem posed by the existence of two parallel services, 111 and out-of-hours (OOH).
Like the wider NHS, we face a serious and challenging financial position, largely due to the growing demand for healthcare to meet the needs of an ageing population and more chronic conditions. We have a fragmented health and care system, which leads to increasing pressures, particularly in the emergency department.
This paragraph gives prominence, and hence demonstrates priority, not to the problem faced by someone needing health treatment or advice but to the CCG’s financial position and the pressure on ‘the emergency department’ (presumably the A&E departments of acute hospitals).
We have an unauthorised overdraft that grows by around £1million a month. With increasing costs and needs, our projected year end deficit is £53 million. We need to be realistic about our future and just improving what the NHS does is not enough; we need to do things differently across the whole system to close gaps in health and wellbeing, finance and care and quality. Our vision is to create a health and care system that empowers people to improve their health and wellbeing and reduce their dependency on services.
Again, the emphasis is on the ‘whole-system’ finance issue, not on whether there is scope for rationalizing the 111/OOH services. The ‘vision’ of reducing people’s dependency on services is evidently also prompted by financial considerations and has no apparent bearing on the issue of how to integrate 111/OOH services. Empowering people ‘to improve their health and well being’ with no reference to how this might be achieved can be read as suggesting that people adopt a DIY approach to their health care and ‘don’t bother the doctor’.
We need to ensure there are a range of safe and effective services that meet people’s needs, when they need them, including GPs – both in and out of hours; minor injury units and urgent care centres; and NHS 111. Having these in place will support the emergency departments to focus on treating people with the most serious and life-threatening conditions.
This continuing preamble still fails to put the issue from the patient’s point of view. So by not mentioning the whole range of services that someone in need might call or visit – ambulance services are conspicuous by their absence – dilemmas such as ‘Do I call 111 or 999?’ are not addressed. In other words, the emphasis is on institutional rationalizing rather than on assisting patients to make informed choices.
NHS Kernow is now looking to commission a fully-integrated urgent care access, treatment and clinical access service incorporating NHS 111 and primary care out of hours’ services in place for 1 December, 2017. The new service will be compliant with national guidance specified in the Commissioning Standards Integrated Urgent Care document (ww.england.nhs.uk/wp-content/uploads/2015/10/integrtd-urgnt-care-comms-standrds-oct15.pdf).
The new service will provide rapid access to anyone who needs medical advice and treatment when they cannot get to see their GP. We want to improve people’s outcomes and experience of care; make better use of the limited system-wide clinical capacity; reduce unplanned demand for 999 calls and transfers and Emergency Department attendances. It is vital the new service delivers the A&E Delivery Board’s priority that NHS 111 should increase the number of calls transferred for clinical advice in order to decrease call transfers to ambulance services and reduce Emergency Department attendances.
Mainland Cornwall, NHS Kernow’s territory, has, besides a single A&E department, one urgent care centre and 11 minor injury units (MIUs). Given the stated need (see above) for ‘a range of safe and effective services’, and the reiterated importance of reducing demand for ambulance and Emergency Department services, it is surprising that the questionnaire focuses exclusively on 111 and OOH services and ignores the others. Even though the telephone numbers of the MIUs are published, the services they provide are not covered in this questionnaire.
We would expect the successful provider(s) to collaborate with other providers in the system to deliver the standards for an integrated urgent care system. In particular:
· To make better use of limited GP time across all parts of the urgent care system eg minor injury units, emergency departments and urgent care centres.
· To increase clinical triage and intervention in NHS 111 and out of hours to improve public confidence and reduce unnecessary use of the Emergency Department.
· To reduce duplication and costs to create reliable and easy-to-access alternatives to the Emergency Department.
Given that stress is explicitly placed here on integrating ‘all parts of the urgent care system’, it is again surprising that the use of the emergency services, urgent care centre and MIUs is not covered by the questionnaire. No allowance is made for the difficulty that members of the public can experience in deciding whether a situation is an emergency or merely ‘urgent’.
We expect to enter into a contract that provides sufficient sustainability for the potential provider(s) to enable them to invest in our community. The contract will be one that rewards both positive outcomes and experience for patients and incentivises efficiency and value for money.
NHS Kernow is considering a number of potential service models:
1. An integrated NHS 111 and GP out of hours’ with NHS 111 provision available 24 hours a day, seven days a week.
2. An integrated NHS 111, GP out of hours’ GP service and Urgent Care Centre model.
3. An integrated NHS 111, GP out of hours’ service and Urgent Care Centre model that is integrated with in-hours primary care.
4. An integrated 111 and GP out of hours’ service with reduced in-hours NHS 111 provision.
The language here is doubtless familiar to those who drew up this questionnaire, but it is not apparent to the lay person what is meant by ‘integrated’ and what the implications could be for patients and the public of ‘services’ and ‘models’ being integrated. No explanation is given of what each of the four alternatives ‘would mean for you’.
We are committed to involving people in Cornwall and the Isles of Scilly to ensure the future service meets their needs. We can only do this with your help and would like your views on our proposals. What you tell us will be considered in the planning of the service. Please tell us your thoughts.
To sum up at this point: It is worth looking back at the foregoing script and (a) noting the language and especially the technical terms used, and asking what a lay person would make of them; (b) considering what this tells us about the gulf of understanding between NHS managers and the public; (c) checking the script against the questions themselves, which will reveal the extent to which the questions are designed to shed light on the issues faced and the relative merits of the four potential alternatives; and (d) asking ourselves whether the questionnaire actually meets some basic standards of intelligibility and consistency.
NHS 111 and out of hours service integration
1. Do you know the difference between NHS 111 and the Cornwall GP out of hours’ service?
☐ Yes ☐ No ☐ Not sure
This question should have started by asking: ‘Are you aware of (a) the NHS111 and (b) the Cornwall GP out of hours services?’ It could then have gone on to ask: ‘What would you say is the difference between them?’ But ‘the difference’ is an abstract concept. It would have been better to ask: ‘Which one would you call if the need arose?’
The redundant apostrophe in out of hours’ does not inspire confidence in the proofreading of this questionnaire.
2. Have you used the NHS 111 service?
☐ I use the service ☐ My friend/family member uses the service
☐ I do not have direct experience of the service
Note the confusion between tenses here. ‘Have you used …?’ relates to the past: ‘I use …’ relates to the present. It is not clear whether the questionnaire is seeking information about past behaviour or present, on-going behaviour. A good questionnaire does not confuse the respondent.
Note too that, looking over the whole questionnaire, we can see that Questions 2-5 and 8-9 are all to do with NHS 111 while Questions 6 & 7 are to do with the relationship between NHS 111 and GP practices. (Questions 10-16 are all on the Cornwall GP out of hours service.) This questionnaire could well have been ordered in a more logical and systematic way, and perhaps divided into sections, which would have made it easier for respondents to comprehend it and get to grips with it.
3. How often have you used the NHS 111 Service in the past 12 months?
☐ Never ☐ 1 to 5 times ☐ 6 to 10 times
More than 10 times, please state number: [ ]
It may seem like a quibble to point out that, given the multiple-choice answers offered, this question should have read ‘How many times …?’, but a well-designed question will not interrupt a respondent’s mental process by inserting a task requiring mental agility, as this one does. Incongruously, while the respondent is not asked to state a precise number between 1 and 5 or between 6 and 10, for more than 10 – and correspondingly more difficult to recollect – a precise number is requested. Moreover, it is difficult to see how a scatter of responses above 10 could be taken into account in any analysis.
4. What was your reason for contacting NHS 111? Yes No
I have used this service before ☐ ☐
I was told to call NHS 111 by another service ☐ ☐
I wanted to know where the service was ☐ ☐
I wanted information on dental services ☐ ☐
I wanted advice on contacting a pharmacy ☐ ☐
I wanted advice on replacing a repeat prescription ☐ ☐
I wanted advice on an illness/injury ☐ ☐
I could not get a GP appointment ☐ ☐
I wanted to speak to an out of hours’ clinician ☐ ☐
I wanted to know the opening time/location of a ☐ ☐
service
Other, please specify:
[ ]
The concept of ‘reason’ is a problematic one to use here. We may have a reason that prompts us to call for advice or treatment in the first place and a further reason for choosing one service provider rather than another. Strangely, the list of ‘reasons’ does not include physical prompts such as: ‘I had this sudden pain in my stomach.’ No distinction is made between categories such as needs for treatment, needs for advice, and needs for information. Moreover the question as phrased makes sense only in relation to a single contact: it doesn’t allow for the respondent having had different ‘reasons’ on different occasions.
5. How do you feel the NHS 111 service dealt with your call? On a scale of one to ten, with one being very poor and ten as excellent, please could you state how you feel the NHS 111 service dealt with your call? Please circle your answer.
1 2 3 4 5 6 7 8 9 10
_____________________________________
Very poor Excellent
What is meant by ‘dealt with your call’? Conceivably it could have taken a very long time to speak to someone, or the person who first answered asked a lot of irrelevant questions, reading from a script, but then a clinician took over and was extremely helpful. Where on the scale should that response be marked?
6. Who do you think people should call for healthcare advice when their GP surgery is open? Yes No
NHS 111 ☐ ☐
Registered GP practice ☐ ☐
Very little can be done with answers to this question because it fails to go on to ask respondents why they have given that answer. And it may be that on some occasions they feel they need face-to-face contact with a clinician whereas on others they don’t.
7. In the future would you like to be able to make an appointment with your GP by calling NHS 111 rather than your GP practice?
☐ Yes ☐ No
Again, this question calls for a follow-up question: ‘Why?’
8. Do you feel you/your friend/family member was advised appropriately by NHS 111?
☐ Yes, all the time ☐ Yes, most of the time
☐ Yes, sometimes ☐ I have not used the NHS 111 service
☐ No, please state why: [ ]
The references to ‘the time’ in the optional answers to this question are at odds with the fact that the service is there to be consulted on discrete occasions, not in an on-going way, over time.
9. If you have used the NHS 111 service, do you think it could help with all your health-related requirements?
☐ Yes
☐ No, please state why: [ ]
This question is confusing in a number of ways. It does not specify what is meant by ‘health-related requirements’: these could include not only requirements for advice or treatment but also requirements for commonplace pharmaceutical products. Nor does it allow the respondent to say that the service could help with some (rather than all) such requirements. And crucially, it does not specify a point or period in time to which it applies. A good question would direct the respondent’s attention to their past or present experiences, not leave it open as to whether they are invited to speculate about the future.
10. Have you used the Cornwall GP out of hours’ service?
☐ I use the service ☐ My friend/family uses the service
☐ I do not have direct experience of the service
Given that the following questions 11-13 make no reference to the experiences of friend or family, it is hard to see the purpose of including the second of these choices in the list of possible answers to this question.
11. How often have you used the Cornwall GP out of hours’ service in the last 12 months?
☐ Never ☐ 1 to 5 times ☐ 6 to 10 times
☐ More than 10 times, please state number: [ ]
As with Question 3: It may seem like a quibble to point out that, given the multiple-choice answers offered, this question should have read ‘How many times …?’, but a well-designed question will not interrupt a respondent’s mental process by inserting a task requiring mental agility, as this one does. Incongruously, while the respondent is not asked to state a precise number between 1 and 5 or between 6 and 10, for more than 10 – and correspondingly more difficult to recollect – a precise number is requested. Moreover, it is difficult to see how a scatter of responses above 10 could be taken into account in any analysis.
12. What was the reason for using the Cornwall GP out of hours’ service?
I have used this service before Yes ☐ No ☐
I was unable to get a GP appointment ☐ ☐
I wanted to request a repeat prescription ☐ ☐
I wanted to speak to a clinician for advice ☐ ☐
I needed to be seen by a clinician at one of their ☐ ☐
treatment centres
Did you have transport to get to a treatment centre ☐ ☐
I needed to be seen by a clinician at home ☐ ☐
Other, please specify:
[ ]
As with Question 4: The concept of ‘reason’ is a problematic one to use here. We may have a reason that prompts us to call for advice or treatment in the first place and a further reason for choosing one service provider rather than another. Strangely, the list of ‘reasons’ does not include physical prompts such as: ‘I had this sudden pain in my stomach.’ No distinction is made between categories such as needs for treatment, needs for advice, and needs for information. Moreover the question as phrased makes sense only in relation to a single contact: it doesn’t allow for the respondent having had different ‘reasons’ on different occasions.
13. What was your experience of the Cornwall GP out of hours’ service? On a scale of one to ten, with one being very poor and ten as excellent, please could you state how you feel the GP out of hours’ service dealt with your call? Please circle your answer.
1 2 3 4 5 6 7 8 9 10
_____________________________________
Very poor Excellent
This question illustrates the difficulty that is created when two different questions are packaged into one. ‘What was your experience?’ invites a narrative response, a factual description of what happened when the respondent contacted the service. No space is provided on the questionnaire form for such a description. Instead, provision is made only for grading how the respondent felt, and on one occasion only.
Moreover, as with Question 5: It is not clear what is meant by ‘dealt with your call’. Conceivably it could have taken a very long time to speak to someone, or the person who first answered asked a lot of irrelevant questions, reading from a script, but then a clinician took over and was extremely helpful. Where on the scale should that response be marked?
14. Do you think the Cornwall GP out of hours’ service treatment centres should be based in the same building with other health services, such as minor injury units and urgent care centres?
☐ Yes ☐ No
This is the first reference in the questionnaire (including the preamble) to ‘GP out of hours service treatment centres’. There is no description of what they are and where they are to be found. Consequently it is difficult to see how a respondent can give an informed answer. And there is no provision for answering ‘Don’t know’ or ‘No preference’ to this question.
15. Do you feel you/your friend/family member was advised appropriately by the Cornwall GP out of hours’ service?
☐ Yes, all the time ☐ Yes, most of the time
☐ Yes, sometimes ☐ I have not used the NHS 111 service
☐ No, please state why: [ ]
As with Question 8: The references to ‘the time’ in the optional answers to this question are at odds with the fact that the service is there to be consulted on discrete occasions, not in an on-going way, over time.
And note that the box here that says ‘I have not used the NHS 111 service’ clearly has no relevance to a question about the GP out of hours service: it appears to have been erroneously copied and pasted from the draft of Question 8. This is a good illustration of the need to have a questionnaire carefully proofread before it is published.
16. Do you think the Cornwall GP out of hours’ service meets all your health-related needs?
☐ Yes
☐ No, please state why: [ ]
As with Question 9: This question is confusing in a number of ways. It does not specify what is meant by ‘health-related needs’: these could include not only needs for advice or treatment but also needs for commonplace pharmaceutical products. Nor does it allow the respondent to say that the service could help with some (rather than all) such requirements. And crucially, it does not specify a point or period in time to which it applies. A good question would direct the respondent’s attention to their past or present experiences, not leave it open as to whether they are invited to speculate about the future.
Note that this question refers to ‘needs’, whereas the similar Question 9 refers to ‘requirements’. A good questionnaire is consistent in its terminology, and so doesn’t distract the respondent’s mind by posing the puzzle of whether different terms mean the same thing.
17. Do you think that either the NHS 111 or Cornwall GP out of hours’ service should be changed?
☐ Yes ☐ No
If yes, what changes would you like to see?
This question, as worded, appears to rule out the possibility of the response that both services should be changed. It also requires the respondent to imagine possible changes, as none are put forward, even as prompts to thinking. If the question had been worded along the lines of ‘Can you suggest ways in which either or both services could be improved?’ it would be easier to respond to.
18. NHS Kernow is considering one of four options to provide NHS 111 and GP out of hours’ services in the future. Please tell us which one would
meet your needs: Yes No
A joined-up (one number) NHS 111 and GP out of ☐ ☐
hours’ service that is available 24-hours a day,
seven days a week)
An integrated NHS 111, GP out of hours and urgent ☐ ☐
care centre model
An integrated 111, GP out of hours’ service and ☐ ☐
urgent care centre model that is also integrated
with in-hours primary care (GPs)
An integrated 111 and GP out of hours service with ☐ ☐
reduced in-hours 111 provision
I would not recommend these but my choice is:
[ ]
Note that the reference in the first of these options to ‘A joined-up (one number) NHS 111 and GP out of hours service’ is the first reference in the whole questionnaire to a telephone number. The other three options all refer to an ‘integrated’ service without spelling out what is meant by ‘integration’ in this context.
West Cornwall HealthWatch asked NHS Kernow for further information about the four options. The following reply was received:
‘The options are deliberately only specified as high level concepts. One of the reasons for seeking feedback is to determine if any of the options should be worked up in more detail at which point we would undertake further public engagement.
‘Firstly none of the options describe the current model. The reason being that we currently have separate 111 and Out of Hours services provided by two separate organisations, namely SWASFT (the NHS 111 element) and Cornwall Health Out of Hours (The GP Out of Hours element).
‘Option 1 “An Integrated 111 and Out of Hours Primary Care Service” is closest to what we have now but assumes integration under a lead provider to reduce duplication of management overheads and processes.
‘Option 2: An Integrated 111, Primary Care Out of Hours & In and Out of Hours Urgent Care Centre Service This option would see clinicians who work in the 111 & Out of Hours Service co-located with Urgent Care Centres in the out of hours service. Option 1 does not assume any co-location.
‘Option 3: An Integrated 111, Primary Care (In and Out of Hours) Urgent Care Centre This option would build on the above, but also see patients being able to book appointments with their GP practice by calling 111.
‘Option 4: An Integrated Out of Hours, 111 and Primary Care Service (No In-Hours 111 Provision) This option would replicate Option 1 apart from a key difference that on Monday to Friday in the day-time, the 111 number would go to a pre-recorded message advising patients to call their own GP and/or include options to have the call passed directly to services such as Dental Helpline.’
While this reply clearly helps us to comprehend the implications of each of the options – importantly, it spells out that ‘integration’ can take the form of a single organization, a single location, and/or a single telephone number – the fact that ‘The options are deliberately only specified as high level concepts’ in a questionnaire ostensibly directed to ‘involving people in Cornwall and the Isles of Scilly is, we may justifiably say, remarkable.
19. Please feel free to add in any further comments in the box below:
[ ]
SUMMARY AND CONCLUSIONS
As the content and language of the preamble show, this questionnaire was written with issues for management rather than for patients at the forefront of the authors’ minds. The authors were evidently confused about its purpose: the objective of ‘We … would like your views on our proposals’ is not consistent with the options being ‘deliberately only specified as high level concepts’.
Although, creditably, an attempt has been made to allow respondents to reply on the basis of their own experience, many of the questions are complex. Some use terms that need to be ‘unpacked’, broken down, e.g. ‘reasons’, ‘dealt with your call’, ‘health-related requirements’. Many are expressed in language that requires respondents to perform mental gymnastics, e.g. asking ‘how often?’ when ‘how many times?’ is meant.
Evidently the questionnaire was not piloted, or even proofread, as that would have revealed these problems.
This questionnaire is not fit for purpose. It is not a professional piece of work.