Pay Data and its limits: precisely inaccurate and maybe inadequate?
Pay data can be confusing at the best of times. It is amazing how often even experienced survey analysts and pay consultants inject spurious precision into what is emphatically NOT a precise science. ("You are paying these job-holders 9.45% below the market rate of 47,527".)
Matching errors, sampling errors, and more matching errors
Taking the most common type of pay survey - the job-matched survey - how can errors creep in?
These surveys start with the pay data providers producing a Job Profile, to which reward managers match their companies’ jobs. It is here that the first major source of error appears: job matching error. No real job matches the survey company's Job Profile precisely. Inevitably, reward mangers have to choose the Job Profile which best fits the role they want to benchmark – not an exact science! So the data that is input to survey databases is never entirely appropriate to the Jobs as described in the Profiles.
Then, of course, we need to remember the obvious: not all companies participate in all pay surveys. Pay data will always only be based on a small subset of companies that have any particular role. This leads to the second major source of error: sampling error. This is less of a problem, however, as it is better understood and can be estimated.
The third major source of error creeps in with the application of pay data. Here, more matching error is likely to occur: companies are applying best-fit survey Job Profiles to the jobs they actually have and inaccuracies are sure to abound!
These errors are amplified when it comes to IT. The nature of many IT jobs requires specialist knowledge on the part of reward managers. This is a major issue in specialist areas. Surprisingly it can also be acute in common-place jobs: even standard terms such as ‘Business Analyst’ and ‘Project Manager’ can convey staggeringly different realities in different companies. Here, much will depend on the clarity of the Job Profiles.
What does this mean?
Taken as a whole, these factors can very seriously undermine confidence in specific pay surveys, or in pay surveys in general. There is no doubt that they underlie many of the ‘discrepancies’ between data from different pay surveys, or even from the same pay survey, sampled at different times by different people.
But do these errors mean that pay data is totally meaningless? No, but they do mean that the utmost care is needed before applying pay data. There are very many potential pitfalls.
The biggest hazard lies in applying pay data in a superficial way, with no awareness of the possible inaccuracies. Sadly, too many compensation and benefits analysts are only too happy to be told a precise and - they believe! - highly accurate pay figure for a given job. After all, it makes their job easier in the short term. And many data suppliers are happy to go along with the idea that their data is both precise AND accurate. This creates problems that emerge only in the longer term.
What can you do about it?
Like explosives, fast cars, and small babies - pay surveys can be deadly in the wrong hands! So how do you ensure your pay data is being interpreted and used safely? There are two simple actions you can take to text your pay data provider:
1. Ask your pay data provider about errors or sources of error in their data - How aware are they of this as an issue?
2. Get them to estimate the level of error (even very approximately) in any comparisons they supply, in relation to jobs you actually have - Can they talk confidently and sensibly about this?
If they pretend the problem is a minor one, or if they cannot discuss the inaccuracies of pay data, be warned. You may not be getting the quality of advice you need to make good decisions.
The bit that might be missing: context!
These observations may leave the reader wondering whether there is any point whatsoever in using pay surveys. Experienced reward professionals will say there is, but only as part of a wider picture. In fact, their very inaccuracy might cause you to participate in more than one survey! Then, if two sources of market data are consistent in terms of what they are telling you about your IT pay, you can proceed with a much greater level of assurance.
In general it is wise to regard pay surveys as contributing just one part of the picture. Other sources of data that will help you to decide where your pay practices fit in the market include exit interview data, retention statistics and issues, and recruitment successes or failures. If these are mostly telling you the same thing as the market pay comparisons, you can proceed with confidence to respond to the underlying pay issues. But pay comparisons on their own should never be allowed to drive your pay policies and practices.