How we analyse evaluation and wellbeing survey data

Re-engage never uses a person’s responses to these questions to include or exclude them from what we do. We look at totals and averages rather than individuals, so that we can consider the extent to which our services are helping overall, provide reliable and up-to-date evidence, and make improvements.

Translating responses into scores

Although we call these numbers ‘scores’, they are not used to rank older people. The numbers simply make it easy to calculate averages and changes.

For each of the wellbeing questions, people are asked to give an answer on a scale of 0 to 10, where 0 is ‘not at all’ and 10 is ‘completely’. Answers are categorised as:

Responses to the first three loneliness questions were designed to be analysed by turning them into a number: ‘Often’ = 3, ‘Some of the time’ = 2, and ‘Hardly ever or never’ = 1.

Responses to questions in the first part of the survey, the direct question on loneliness, and the questions on relationships and trust, are not turned into scores but analysed as the categories that are listed for respondents.

Analysis

We look at:

• Averages – for example, the average rating of happiness given by all older people when they join us.
• Categories – for example, how many older people often feel lonely at a particular point in time.
• Changes – for example, how the number of older people who feel they have people to call on for support changes over a particular period.
• Comparisons – either internal (for example, comparing how those joining call companions feel to how those joining tea parties feel) or external (for example, trends in ONS UK population data vs. trends in our data).

The first part of the survey helps us to see if some services are more successful in one regard than other, and to identify if there are groups of people who do not seem to experience the same benefits as others when they join Re-engage.

As the second part of the survey is used by ONS and lots of other organisations, we can compare our findings to the general population and to other interventions (benchmarking). This gives us an accurate picture of our impact and enables us to clearly attribute improvements to our services, as opposed to interventions from elsewhere.

Why it matters

As you can see, these surveys are crucial to our work, as they are to the work of various organisations trying to make a difference to people’s lives. By asking these questions on a regular basis, we can see how our older people feel overall and if/how that changes, which enables us to evaluate what we’re doing and make adjustments where we need to.

That’s also why it’s important that we record people’s responses accurately. What someone says during conversation or the tone of their voice cannot be used to interpret their mood or guess what their answers will be: this would be inaccurate, and unfair - people prefer to be given the opportunity to answer for themselves.

Evidence suggests that when survey questions are worded differently, people can understand them differently and give different answers. That’s why we’re careful to use standardised questions placed in the recommended order. Sometimes the person you’re speaking to may need a further prompt to help them, and that’s fine. It’s just important to avoid phrasing that might encourage a particular answer.