How much faith can you have in salary surveys?

Aug 1, 2021 | Salary Surveys

 

Can we be certain the figures are right? 

 

 

In Reward, we get asked this often. Salary surveys are used all the time in Compensation and Reward, and pros rely on them to inform the pay recommendations they give to hiring managers and company leaders throughout the yearly Reward calendar. But when Reward pros get into conversation with senior managers, the course of the discussion often requires the Reward person to steer the manager towards adopting a realistic interpretation of what salary surveys can and cannot do.

 

 

Many times, after gathering market pricing data on a clutch of jobs, I’ve been in conversations with managers, or with HRDs and HRBPs who ask me – “are you absolutely certain about these figures? Can we be 100% sure these are accurate?” What they’re asking at these moments is for me to give them a guarantee that the market data they’re reading is providing a true reflection of the competitive market for whatever the jobs in question are – specialised IT roles, jobs in Legal team, positions in Marketing or Business Development, for Engineers or whatever else. It’s expressed almost as a “would you risk your reputation on it?” type of question. Would you stake your job on it? Well…frankly no. But surveys are better than relying on anecdotal evidence.

 

 

 

 

 

Reward professionals who spend a lot of time analysing salary surveys basically report back on what the survey shows. Sometimes the survey has insufficient data on a particular job – for instance, I recall recently looking at a major survey to get salary data for the job of Cybersecurity Specialist, and was surprised to find that the survey provider had not gathered enough data to enable a precise recommendation on what is now a critical role in modern workplaces. In such a situation, we might look at another survey (if our organisation has paid the money to buy and take part in it), or we can base the recommendation on a job sub-family or even the whole job family and level (which is definitely not ideal). This is not high powered maths that we’re talking about. Basically, the Reward person reports back on what the survey data shows. It’s then for them and their client manager to interpret and decide how they’ll use these figures to inform the salary recommendation.

 

 

A partial map of reality – the data isn’t perfect

 

 

We only ever get part of the picture – despite what the layman imagines, Reward is NOT a perfect information game. Salary surveys are not the complete reality of what’s out there.

 

 

If we’re going to be relying on survey data to inform salary recommendations, it’s important to understand what the limitations of survey data are. Data sets used in salary surveys are based only on data from participating companies, and are not a survey of the entire market place – of all organisations and all jobs. This is why it’s important to look at the participants list for each survey you’re considering spending money on. The most important participants will often be your direct competitors – organisations in the same industry, probably of comparable size, and who are competing for the same talent. So let’s think here for a moment about how your client managers want to use the data –

 

  • The problem with job titles I mentioned the problem of making judgements based on a job title alone. In some organisations, managers place great importance on job titles, and its not uncommon to find managers who think “give the guy a big title and keep him happy” is perfectly reasonable as a way to motivate and retain people. Of course, this makes your job matching inaccurate, and in some cases this can be to the point of rendering our findings pretty meaningless.

 

 

  • Survey questionnaire completion can be an unwelcome task Think for a moment about completing the survey data questionnaire for your organisation. Think about how unpleasant many Reward teams find this activity, something that’s often left till the deadline. The survey vendors are working on the assumption that participating organisations always provide accurate data. But there are often input errors, incorrect job matches (it’s hard to make judgements based only on a job title), and errors on grading a levels. So there’s always an amount of crap that finds its way into these data sets. Then multiply that error rate by a few hundred or a few thousand participants….well, you get the idea. Maybe this is why recently there are more people calling “bullshit” on salary surveys. Maybe someone will come up with a better tool. In the meantime, the best approach is to use surveys that are as relevant and data-rich as possible.

 

 

  • They might be mainly concerned about what specific competitors are paying. So if you’re in a big chemicals company, and you’re looking for Ph.D and Postdoc level Chemists, you might be very interested in what companies like BASF, Dow Chemical, Du Pont, Akzo Nobel and Bayer are paying for people with these qualifications and experience. But maybe these guys are not in the survey data because they don’t participate. Well, maybe instead our company participates in an industry compensation group that shares data and best practice – but the truth is that group participants mostly regard salary data as being too commercially sensitive to share. Managers might ask that you pay for a custom survey, but this is obviously time consuming and often prohibitively expensive to do. Our survey, which is probably a large generic survey (such as Hay, Willis Towers Watson, Radford, Culpepper, Mercer, et.al)  might have some chemical industry participants but not the specific ones your client managers are interested in.

 

 

  • Alternatively your managers might be interested specifically in similar industries, or companies of a particular gross annual turnover, or companies with similar employee numbers. However the survey might not provide the exact data slices you want. Inevitably this means that the Reward pro has to make the most of the available data to make any meaningful comparative analysis. That means choosing carefully, and the couching the findings tentatively so as not to make firm conclusions from limited data.

 

 

 

 

 

 

  • The limitations of the data Salary surveys don’t capture data on what every organisation is paying, and are only ever a representative sampling.  It’s like a street map where a lot of the streets aren’t even shown. Managers sometimes make the unwarranted assumption hat survey data covers most or a very high proportion of the job incumbents. This would only be true if survey participation was mandated by law – and even then, there would be companies that failed to comply. Survey data offers no more than a snapshot in time. It’s worth reflecting that in Reward it’s your job to make the maps, not just follow them.

 

 

  • What was versus what is now We want accurate data of the market as it is today. However it’s important to say that even the largest and most recent salary surveys are reporting on data what they captured about 6 months ago. Yes, we might “age the data” to take account of that, applying the same percentage figure uniformly across all jobs, but this is often inaccurate. There are fluctuations, albeit modest in some cases, but in markets with lots of competition for talent, six months can make a difference that means the survey data is no longer useful. Lets not forget that the pandemic, financial crises and major economic trends can all make a difference – the picture at the time of collecting the data might look a lot different now

 

 

The bottom line is that there are several reasons that your salary survey data has real limitations, and that you should be very cautious in drawing conclusions from it. This is made all the worse for organisations that rely too heavily on only one data source or one salary survey to inform all their salary ranges. Still, if that is your situation, then you have to find ways of working with survey data, making sense of what you find there, and making the most of it. You’re being paid for your advice, your judgements and your recommendations, and of course, managers might feel free not to take that advice. Salary surveys provide indications, offer directions, but are not the definitive be-all-and-end-all of salary decisions.

 

 

Coach and educate your managers

 

 

At the end of the day, was we do in Reward is mainly advisory. What we’re doing is laying out the data, presenting it to guide the conversation, reminding them that this is the best data we have available to us, and then advising the client manager that this supports them in making their salary decisions. Base pay is never the only factor, and they might want to weigh up how the survey data sits within the broader picture of organisational changes, structure and hierarchy, bonus and performance payments, etc. The survey figures are not gospel, and should not be taken as such. It’s sometimes like the Emperor-Has-No-Clothes epiphany – managers are surprised that survey data is not the last word. We have to educate managers to know that the surveys aren’t some magical elixir, an oracle with all the answers. But they provide useful landmarks for better pay decisions.

 

 

 

 

 

So, we remind the client manager of the survey data limitations, mention the advice with any caveats, and arrive at a judgement of what works, and what can be done within the limits of the company’s Reward philosophy and principles. When we get a manager who wants it to be a recommendation signed in blood, our best way forward is to educate them. Take a moment to preface the advice with a brief description about how surveys are built, and about their limitations, and about how best to use them.

 

 

The big survey providers are good at marketing, and the non-HR person often encounters survey data with an uncritical level of confidence and belief in the numbers. Yet even the supposed absolute logic we find in mathematics is itself based on assumptions. So, we must go and earn the money they’re paying us – putting some thinking into it, using our professional expertise, applying our judgement, and offering a more rounded view on the figures. There are no certainties in salary survey figures – so it’s best to be careful of accepting what they tell you without some healthy skepticism. That way, we can provide more balanced advice to our client managers.

 

 

Recent posts

Is this the “Moneyball” moment for HR & Reward?

Is this the “Moneyball” moment for HR & Reward?

    The movie Moneyball tells the real-life story of how a new manager at baseball club Oakland Athletics, Billy Beane used analytics to make massive improvements in the performance and results. Guys with spreadsheets and maths degrees showed their approach...

Who drafts the Job Description?

Who drafts the Job Description?

Writing job descriptions   Question is - how do you get consistently well-written, high-quality and compelling job descriptions for all the roles in your organisation? Back when I was starting out in HR, I had to conduct job analysis interviews, and capture what...