pay data precision

Last updated: 28 March 2009

Pay Data precision and accuracy: not always the same thing!

Pay data can be confusing at the best of times. It is amazing how often even experienced survey analysts and pay consultants inject spurious precision into what is emphatically NOT a very precise science. ("You are paying these job-holders 9.45% below the market rate of £32,527".)

Taking the most common type of pay survey - the job-matched survey - how can errors creep in?

Such surveys start with the survey company producing a Job Profile, to which data providers match their own jobs. Inevitably, the data providers have to choose the best fit Job Profile against which to submit their pay data. And this introduces the first major source of error: data provider matching error. No real job matches the survey company's Job Profile very precisely so the data that is input to survey databases is never entirely appropriate to the Jobs as described in the Profiles. Then, of course, we need to remembers that the survey company will only have a sample of all the employers of this type of job in their database. This leads to the second major source of error: sampling error. This source of error is better understood and can be estimated. Now comes the third major source of error: the point when users of surveys use the data. Here, more matching error creeps in because they are applying best-fit survey Job Profiles to the jobs they actually have.

Matching error is a particular problem in IT pay surveys because of the knowledge required on the part of those using surveys. This can be a major issue in specialist areas of IT, but it can also be acute in common-place jobs, where even standard terms such as "Business Analyst", "Project Manager" etc can convey staggeringly different realities to different people. Here, much will depend on the clarity of the Job Profiles.

Taken as a whole, these factors can very seriously undermine confidence in specific pay surveys, or in pay surveys in general. There is no doubt that they underlie many of the "discrepancies" between data from different pay surveys, or even from the same pay survey, sampled at different times by different people.

But do they mean that pay data is totally meaningless? Obviously not, but they do mean that the utmost care is needed before applying pay data. There are very many potential pitfalls. The biggest pitfall lies in applying pay data in a superficial way, with no awareness of the possible inaccuracies. Sadly, too many compensation and benefits analysts are only too happy to be told a precise and - they believe! - highly accurate pay figure for a given job. It makes their job easier in the short term. And many data suppliers are too happy to go along with the mindset that sees data as both precise AND accurate. This can create problems that emerge only in the longer term.

Like explosives, fast cars, and small babies - pay surveys can be deadly in the wrong hands!

So how do you ensure your pay data is being interpreted and used safely? Here is one way of checking out your supplier of data or your pay consultant:

1. Ask about the errors or sources of error - see how aware they are of this as an issue.

2. Ask them to estimate the level of error - or inappropriate comparison - even very approximately (in percentage or £ terms) of any comparisons they supply, in relation to jobs you actually have.

If they pretend the problem does not exist, or if they cannot begin to discuss these subjects, be warned. You may not be getting the quality of advice you need to make good decisions.