In Defence of Star Ratings

Last Autumn (too long ago, sorry Chris) – I had a conversation with Chris Goode on Twitter about the Albany’s feedback forms, prompted by his tweet:

‘Bit sad… to see brilliant @TheAlbanySE8 soliciting star ratings from audiences on its otherwise interestingly nuance feedback forms…’

His comment related to a specific question on Albany audience feedback forms which asked:

‘if you were reviewing this production for tomorrow’s papers, how many stars would you give it?’

This question sat alongside a number of ‘differentials’, asking audiences to rate their night based on five dimensions of the audience experience: 1) engagement and concentration 2) learning and challenge 3) energy and tension 4) shared experience and atmosphere 5) personal resonance and emotional connection.

He asked if the data gathered from the star ratings informed decisions about which artists get supported/asked back, and pointed out that the question about how star ratings are used in feedback forms is an adjunct to a live conversation about how star ratings are used more generally to critique work, particularly in media reviews.

Andy Field, artistic director of Forest Fringe, had recently written:

“Star ratings are the very worst – nuance’s kryptonite – tiny nuggets of sadness harvested in the darkest heart of consumer capitalism and sent to cling grimly to the surface of art like old shopping bags floating down a river.”

Which seemed to summarise the general mood in the alternative arts world about star ratings fairly neatly. He wrote:

“In (using star ratings) we’re helping to perpetuate a thinness to the way in which people can engage with our work, and the means they have of navigating their way through it. To an extent, the very stars that we use to attract people are actually perhaps limiting the number of people who will engage with our work and the ways that they have of engaging with it. It’s a vicious circle, but the point is, we are part of that circle, not simply it’s victims.”

I remained somewhat more ambivalent on the matter. While I could recognise the frustrations some artists felt about the reductive nature of critiquing work via a star system, I thought in the context of our feedback forms – where we ask our audiences for their perspectives in a range of ways, not simply via a star rating – they might be ok – and even useful, to both venue and artists alike. But I hadn’t written our feedback forms (they predated my time at the Albany), so being the thorough professional I am I decided to undertake a deeper exploration of perceptions of feedback forms, and review our current feedback forms based on this.

Here’s what I did. I undertook a survey of Albany staff as well as artists and producers we work with regularly, garnering their views on our current feedback forms, and how they use them in their decision making processes.

I reviewed the literature around approaches to gathering feedback from audiences. There’s not much around. Matt Truman wrote an interesting blog about feedback processes here which called on venues to reject feedback forms, seeking opportunities for more nuanced, direct conversations with audiences, particularly in the context of works still in development, where audience feedback is likely to have an impact on the direction of the work.

The most significant, and comprehensive, review of how feedback is gathered remains that published by ITC (Independent Theatre Council) way back in 2005. The publication outlines research undertaken by the New Economics Foundation on ‘capturing the audience experience’. It offers an in-depth analysis of how feedback can meaningfully point to the impact of an artistic experience on the general wellbeing of an audience member. It is from this report that the questions on our own feedback form are drawn, including that thorny star rating. In the report, the star rating is not designed to, in and of itself, offer a measure of the value of a particular experience – but to be used in reference to a number of other evaluation questions, based on different areas of audience perception, to establish what aspects of an artistic experience lend it value from the point of view of an audience.

I also co-hosted an event with Jake Orr and Maddy Costa exploring the relationship between critics, marketers and audiences, where a discussion took place regarding the use of star ratings. Maddy summarised the event here. While a range of views were expressed regarding star ratings, I found myself in the naughty corner with a bunch of arts professionals who thought that, after all, star ratings might be a simple metric that’s comprehensible to everyone.

On the strength of this period of investigation, I reviewed our feedback forms, and reprinted them with some slight changes. Perhaps the most significant was moving the star rating to the bottom of the form (below the audience experience differentials outlined above), with the intention of encouraging audiences to think through the different aspects of their experience before offering a star rating. (A link to the new feedback form appears at the bottom of this article). We have also reviewed the way data sources are drawn on in the decision making process at the Albany, and how this information is shared with artists.

Nothing that emerged through this investigation convinces me that the negative impact of star ratings is such that it outweighs the many ways in which they are useful.

We’re used to rating our consumer experiences with stars in modern society – on Ebay and Tripadvisor – and perhaps this is what Andy means when he writes they are ‘harvested in the darkest heart of consumer capitalism’. But a more optimistic view might be that the invitation to ‘rate’ an experience makes us more proactive and engaged as consumers. Star ratings are by their nature an oversimplification; but the benefit of simple evaluation tools is that they are accessible. Star ratings offer a means of ‘taking the temperature’ of our activity evenly across the season, with the broadest possible cross section of audiences – not only those that have the confidence to articulate complex perspectives.

Clearly this needs to sit alongside a more nuanced programme of engagement, and the Albany’s commitment to drawing audiences into the creation of work from the very beginning is a clear example of how this happens here (recent examples include Ampersand Media’s Storylines, a monthly performance created with a newspaper reading group; and Simon Mole’s No More Worries, on which he’s run workshops with community groups during its creation), as well as, for example, the chalk board we place outside the theatre for audience to share their immediate response to performances, or regular post show discussions.

But the consistent information we’re able to get from feedback forms has an important role to play. It can be a departure point for a conversation – why do certain shows appeal to our audiences more than others? Do certain shows resonate with particular demographic groups, or work better at one venue than another – and if so, why? As with all evaluation tools, we use them as a way to reflect on the decisions we’ve taken and the impact they’ve had, rather than as a means of assessing ‘success’ or a ‘failure’.

Audiences, incidentally, are generally far more generous in their response than critics. That disparity is interesting – in my view it reflects the fact that the majority of professional theatre criticism emerges from a pretty narrow cross section of world views – and can be a means of making an argument for the value of a piece of work that doesn’t necessarily sate the appetite of the mainstream media.

The best evaluation synthesises information from a number of different sources, and interprets it through the lens of human intelligence. Quantitative data – for example star ratings – are one source of information. Qualitative data – for example personal comments and reflections gathered from audiences via feedback forms, social media, and good old-fashioned face-to-face conversations, are another. But in none of the organisations in which I have worked has either of these sources of information outweighed the importance of the personal understanding we have of the artist’s creative aims, and how these marry with what we consider to be urgent questions and approaches that are relevant to the contexts in which we operate. Nor are strong leaders of arts organisations shy of programming or producing work that may be more challenging for audiences – in fact doing so can often be the most powerful decision they can make.

You can view our new feedback for here.

Amber Massie-Blomfield,Head of Communications, The Albany