PR Evaluation Survey 2 - 45% say there's a "lack of information"

This is the second of two posts on small-scale research into current PR measurement and evaluation practices. See # below for the methodology.

The first posting identified that while 80% claimed to formally or informally evaluate PR activity, 43% continued to use AVE as a prominent measurement metric.

This post reports on who is doing evaluation, the percentage of PR activity evaluated, some attitudes, knowledge of the Barcelona Principles and students’ views on the whether evaluation was being undertaken ethically.

Who did the evaluation of PR activities?

Account team/PR staff – 45.6%

In-house research and outside media analysis company – 26.5%

In-house research section – 16.3%

·         It looks as if self-certification of PR activity is common

What percentage of PR activity was evaluated?

100% - 35.7%

90% - 19.0%

80% - 11.9%

·         Good news – Two-thirds of the 80% who evaluate their PR coverage (53.3%) measure and monitor 80% to 100% of their PR activity

Questions about attitudes to PR measurement and evaluation

1)    PR budget is difficult to obtain: 37.7% agree or strongly agree; 28.2% disagree or strong disagree

2)    There is a lack of information on PR evaluation: 45.3% agree or strongly agree; 34.0% disagree or strong disagree

3)    There is a lack of time for PR measurement: 55.6% agree or strong agree; 24.1% disagree or strongly disagree

4)    PR is difficult to measure: 35.8% agree or strongly agree; 35.8% disagree or strongly disagree

5)    Practitioners fear evaluation: 25.0% agree or strongly agree; 42.3% disagree or strongly disagree

6)    Without measurement, PR’s future is threatened: 64.1% agree or strong agree; 20.8% disagree or strongly disagree

·         Mixed messages – 7% more say budget is difficult to obtain; there is a gap of 11% between those who agree that there is a lack of information on PR measurement (a large 45.3%) and those who don’t; a clear majority don’t have time to evaluate ('too busy doing PR'); there is a balance between those who find PR activity difficult to measure and those who don’t, which is an improvement; Many disagree that practitioners fear evaluation, but nearly-two thirds (64.1%) agree that PR’s future is threatened without the consistent use of measurement and evaluation.

·         The most concerning attitudinal outcome is that 45.3% of organisations say there is a lack of information on PR measurement and evaluation methods. This is a negative comment on the professionalism of many practitioners who can’t be bothered to look at abundant resources in terms of online materials (often free), books and training courses. Measurement and evaluation has been a major education and training topic since the mid-1990s and appears to have been ignored by them.

Students were asked about the percentage of PR budgets that were applied to PR measurement and evaluation. Most, not surprisingly because of their junior positions, ‘Didn’t know’ (53.8%) but the next largest valid percentage was for 1-3% of total budget (17.3%), which aligns with other research in the UK and Australia.

Barcelona Principles

Students were questioned whether the Barcelona Declaration of Measurement Principles (AMEC 2010) was referred to or mentioned at their main placement. Their answers were wholly negative with 55/55 ticking “NO”. Bearing in mind the support that AMEC, CIPR, PRCA, PRSA, IPR, Global Alliance, etc have given to the Barcelona Principles in the past three years, this is a very disappointing result but is similar to US research (Ragan and others) that found low awareness.

Was PR evaluation undertaken ethically

YES – 74.0%; NO – 26.0%

No comment!

# Methodology: PR students at Bournemouth University were surveyed recently about their experiences of evaluation practices during their 2012/13 sandwich year placement. 55 students (85%) took part, voluntarily, in the self-completion survey. As all but one (98.2%) had been on placement for nine months or more in a single organisation, they can be considered valid observers of practices taking place around them or in which they participated. The data were analysed using SPSS which provided descriptive statistics, mainly frequencies. The data used in these posts is based on ‘Valid Percent’ which omits missing answers unless they are a large part of the sample.

PR Evaluation Survey 1 – 80% do measurement; 43% use AVE

PR students at Bournemouth University were surveyed recently about their experiences of evaluation practices during their 2012/13 sandwich year placement. 55 students (85%) took part in the self-completion survey. As all but one of them (98.2%) had been on placement for nine months or more in a single organisation, they can be considered valid observers of practices taking place around them or in which they participated.

Headline news, using valid percentages, was that they reported

• 80% of placement organisations undertook formal or information evaluation of PR activity
• 79.2% of activity was measured in those organisations that undertook evaluation
• The main measurement was of media coverage, by 10 times (at least) ahead of measurement of KPIs, social media or organisational objectives (in that ranking)
• AVE was used in 43.2% of placement organisations; On the other hand, it was not used in 56.8% of placement organisations

Calculation of AVE

Students who reported that AVE was used were asked “How was AVE calculated at your main placement organisation?”

Their verbatim replies indicate that (1) AVE continues to be widely used and (2) some AMEC members are actively offering products and services to calculate the metric.

Got it from Precise; PR Value = AVE x 3 divided by 100

Based on rates from Gorkana

By media evaluation agency as percentage of editorial value

[Media coverage multiplied) X 3

Via Metrica and Cision's algorithms

We rang companies for the figures or used Precise or Gorkana

We used Mymarket Monitor to indicate AVE which they used page space to decide value

A piece of coverage that took up 1/4 of the page, for example, was divided by 4 and x (multiplied) by 3 or 5 depending on whether it was online or print coverage.

PR value x 3

PR Value = Advertising by 3

For press coverage a value was calculated at the end of the month

Depending on the size of coverage times x 3

Through Precise Media and AVE reports for each campaign

We would send coverage to an outside agency (Kantor) they would reply with AVE

By measuring coverage, working out the costs of this space by advertising rates and times value by 3

Calculated the published article against (an) advertising rate card. Also against viewers

Not sure: PR Value x 3 (divided by) 100%. Same strategy applied for all.

From Precise media cuttings package

Despite the Barcelona Principles, which were announced with very visible support from CIPR and PRCA in 2010, AVE is as widely used as it ever was. And AMEC members, who wrote and adopted the Barcelona Principles which barred use of AVE, are leading the way in its continued usage.

More research outcomes follow soon in PR Evaluation Survey 2.

Memo to Rob Flaherty

It’s been announced that Rob Flaherty, CEO of the Ketchum PR consultancy group, is be keynote speaker at AMEC’s European Summit on Measurement in June. According to Ketchum’s PR Newswire message, he will be calling for “unified effort to establish measurement as key tool”.

I don’t know Rob but have met his predecessors David Drobis and Ray Kotcher. I have a lot of time for the consultancy’s immensely likeable European CEO, David Gallagher who, like me, is a former PRCA chairman. There are lots of very able people at Ketchum, so this “memo to Rob” is offered collegially.

Having researched the history and practice of PR measurement and evaluation since 1992, this issue comes around again and again. Nothing new is said and PR people continue not to evaluate.

I have delved recently into the International Public Relations Association (IPRA) archive which starts in 1953 and am planning to write a paper on the practice topics that appear decade after decade. For industry leaders who think they have a new insight to offer: forget it, it has probably been said ten times before. There is little institutional memory in the business.

For Rob’s address to the Madrid jamboree, here is a timeline about PR measurement and evaluation (see also Watson, 2012).

1905/6: The Publicity Bureau of Boston, which Scott Cutlip says was the first PR agency in the US, developed the ‘Barometer’. It was a researched guide to the attitudes and interests of newspaper editors to help with accurate placement of editorial material.

1920s onwards: AVEs and multipliers start to be used in press agentry and publicity work in the US. They continue to this day.

1928 to mid-1940s: Arthur W. Page uses extensive opinion research to shape AT&T’s communications, public relations and customer-facing behaviours.

1950s onwards: Cutlip and Center’s PII (Preparation, Implementation, Impact) model of PR planning and measurement appears in the still-published Effective Public Relations. Generations of PR practitioners have been taught PII.

1977: James Grunig, in association with AT&T, starts continuing academic research into measurement and evaluation. This leads to a flowering of research and publication that continues to now.

1990: Glenn Broom and David Dozier publish the still-excellent Using Research in Public Relations, which has been used extensively around the world.

1993: Walter Lindenmann, who worked for H&K and Ketchum, introduces his Three-Step Yardstick of Outputs, Out-take and Outcome. It has become the standard terminology of public relations measurement.

Late 1990s: The three-year-long 'Proof' campaign to promote best practice in PR planning, research and evaluation is launched in PR Week (UK) in collaboration with PRCA and the-then IPR (now CIPR).

2010: The Barcelona Principles, a benchmark statement of existing practices, is launched by AMEC and supported by PR organisations widely.

My case is that practitioners have been offered well-developed methods of PR measurement and evaluation from at least the late 1970s onwards. In 2008, a paper by Anne Gregory and me reviewed the range of methodology and called for practitioners to use it. No further basic research was needed, we said. There were no knowledge barriers; it was time to borrow Nike’s theme and for PR people to “just do it”.

There have been innumerable books written and industry initiatives conducted, but there is still very low take-up by practitioners. Only recently, a Ragan survey found that around two-thirds of US practitioners had not heard of the Barcelona Principles. So it’s not methods that are needed, it is for practitioners to open their minds and change their behaviours.

So Rob, when you stand on the Madrid platform with your “roadmap on the future of PR”, please propose that practitioners take their own futures in their hands and apply the PR measurement and evaluation methods that have been around for decades. They are well-proven.

Best wishes, TOM


Here’s some reading to help you prepare the paper

Broom, G.M. & Dozier, D.M. (1990). Using research in public relations. Englewood Cliff, NJ: Prentice Hall.

Gregory, A., & Watson, T., (2008). Defining the gap between research and practice in public relations programme evaluation – towards a new research agenda. Journal of Marketing Communications, 14(5), 337-350.

Lindenmann, W.K. (2006). Public relations research for planning and evaluation. Gainesville, FL: Institute for Public Relations. Available from: http://www.instituteforpr.org/topics/pr-research-for-planning-and-evaluation/ 

Watson, T. (2012). The evolution of public relations measurement and evaluation. Public Relations Review, 38(3), 390-398. DOI:10.1016/j.pubrev.2011.12.1018.

ROE may be more accurate than ROI

I was interested in Andrew Smith’s post on CIPR's Conversations blog (March 22) about Oliver Blanchard’s book Social Media ROI. However, both blogger and book shared same old misunderstanding of Return on Investment (ROI)'s application to PR.

Through 2011 and 2012, there was a three-nation debate about whether ROI was appropriate for public relations or not. It was carried out by PR academics and practitioners in Germany, UK and US.

[It is available on the IPR’s website (http://www.instituteforpr.org/) and academic research is online at PRism journal, (http://www.prismjournal.org/vol8_1.html).

At its heart was the contention by Prof Ansgar Zerfass (Leipzig University, Germany) and me that practitioners mis-apply ROI to their disadvantage. In 2011 we undertook a Europe-wide study of practitioners’ understanding and usage of the ROI concept.

The sample of more than 2000 responses showed wide national variations but generally, practitioners had very loose, inaccurate views of ROI’s application. These very different to classic definitions of the term as a ratio of monetary value created, divided by the costs incurred and multiplied by 100.

Where PR folks get it wrong, is that unlike a business ROI which related to capital expenditure over time, their ROI fail to account for all costs, don’t have a hard-edged financial objective and rely on estimates and not accountancy. So what their figures are not recognisable as a ‘return on the capital employed’. They are guesses which are not related to the specific financial management term.

In most cases, intangible results of public relations activity did not lead to financial outcomes. This is especially relevant in governmental and non-profit organisational communication even though there are attempts to employ "ROI" in those fields.

Our conclusion was that ROI was an inappropriate term for public relations measurement and evaluation. This conclusion was supported by the eminent PR theorist Prof James E. Grunig who wrote that he would “cease and desist” from using ROI in the public relations context.

In the US, Prof Don Stacks of Miami University contends that, rather than creating a PR-ROI, the practice should consider a Return on Expectations (ROE) measurement which could cover both financial and non-financial situations. He has published on this recently. His case is worth considering.