How have technical, social, economic and legal forces influenced the business of people assessment? New technologies used in assessment include smartphone and mobile sensing, ambulatory assessment and ecological momentary sampling, text mining, sensors and wearables, as well as virtual and augmented reality. Gone are the old days of application forms, interviews and references. Exploiting and scraping the web are in.
Those interested in assessment often seem transfixed by the how questions (how we measure people) which do change compared to the what questions (what aspects, features) which do not. There is also the question of whether new technology improves the breadth or depth but more importantly the accuracy of assessment
But does a new technology adds more, new, relevant information that we need, rather than simply new ways of collecting and refining data? Also, those who use new technology (AI algorithms) might expect a number of lawsuits and would do well to start preparing their defense based on all the relevant criteria as well as predictive validity.
There has always been the call for faster, cheaper, more accurate and more fake-resistant ways of assessing people. And, as one might expect there are always people happy to supposedly “supply that need”. Indeed, there is a lot of money to be made in this area. “Start-up watchers” beware.
Essentially there many fundamental questions
- What we are trying to assess? The answer appears to be no: selectors are still interested in an individual’s ability, personality and motivation as well as their integrity and health. Whilst new concepts appear every so often (e.g agility, resilience) there has not been much change in the fundamentally features of what people are trying to assess. The predictors of success have not changed.
- How can we assess individuals? This is about the development of new measurement techniques (mostly web-based, behavioural and physiological) which may be superior to those used in the past. But shiny new toys need to be proved to be better.
- The cost of those assessments? A central question is organizational budgets and it seems some, realizing the cost of selection errors, are willing to spend greater amounts in the hope of better assessment and selection.
- What we are allowed to assess? For many, the new world is one of increasing legislation where there are a number of questions and details it is unadvisable and illegal to ask as they may be related to anti-discrimination laws. This issue is getting much hotter: watch this space.
- Who does the assessment? This is about whether companies should outsource assessment to experts or do it in house. More and more it is B2B cutting out the expensive middle men: test publishers and consultants.
- How the assessment data is used? Is the data fed into a complex and sophisticated algorithm or used more impressionistically by an individual or small team? Is it stored and used to help validate instruments and decisions?
- To what extent is the assessment data fed-back to the individual and or used by HR to develop a training program to exploit this data?
- Where the data is stored: i.e. in the cloud and all that that implies?
- Is their “joined-up” data collection and analysis in the different parts of the organization? Or do they jealously guard their own patch?
- Are there any special problems associated with on-line assessment, like being clear about who is actually taking the assessment?
There are also unintended consequences and effects of these developments. The use of the internet does expand the applicant pool but also increases the number of under-qualified and out-of-country applicants. It is easy to be flooded with inappropriate applicants and there is also the loss of personal touch that both assessor and assessee value and respect. There are further concerns about cheating if timed ability tests are used and adverse impact on those who not have access to the technology to take the tests.
Talent identification in the HR world is shifting, indeed has shifted from the traditional methods of assessment, including job interviews, assessment centres, cognitive ability tests, personality inventories, to techniques like digital interviewing and voice profiling, social media analytics, web scraping and text analytics, internal big data and talent analytics and gamification.
Marketing in the Brave new world
Those who favour and sell many news assessment products argue that many employers are overwhelmed with large applications They want to reduce the time and costs and have a clear and fair method to differentiate candidates. They want methods that are focused on diversity and inclusion and provide a good experience for their candidates.
They use the following catch phrases to sell their ideas and product:
“ Next generation” technology; 21st century generation”;“ Digs deeper” “Reveals more”; “Powered by Neuro-Science”; “State-of-the-Art”;“ Has less adverse effect….leads to more diverse choices”;“Authentic” and “real world”; “Disruptive”, “exciting new and different”
Some argue that their techniques have better psychometrics: particularly predictive validity: they are more accurate than the “old” well-used and tried methods. They suggest that their new methods in fact lead to a reduction/avoidance of “older method” issues/artifacts (e.g. impression management). That is, the more traditional methods have well known problems associated with them and these new methods largely overcome them.
They also suggest that many new methods provide a better candidate experience: that is that candidates are much more positive about the whole experience. This leads hopefully to better PR for the tester and company doing the assessment and selection. These new and improved techniques are it seems more up-to-date, fairer, and more engaging which reflects very well on the selectors.
Beware the Jingle-jangle Effect
The Jingle-Jangle fallacy refers to the idea that two different things are the same because they bear the same/very similar names (jingle fallacy) or that two identical or very similar concepts are different because they have different labels (jangle fallacy). The question is what jingles vs jangles and why? Old concepts simple re-packaged for the modern ear; or new, different concepts hiding under familiar umbrella terminology?
Fashions change; ideas and measures need revitalisation. So it is not difficult to take an old test and idea and repackage it, which is, of course, what many do. Manufacturers who prefer the jingle fallacy. Notice how the cheaper store’s product has a name and package almost identical to the much more expensive, exclusive brand. They want you to think that a thing with a near identical name, colour, label is essentially the same at half the price.
New technology
Many attempt to exploit the opportunities that new technology offers to assess people more accurately, easily, and cheaply. Some are early adopters, indeed even pioneers, in the field. Others find that it is client demand that causes them to investigate, and then use, new tools and techniques that show that they are at the cutting edge of psychometrics. The question for many must be the investment of time and money in techniques that in the end fail to deliver what they promise and may indeed cause many additional problems.
There are changes in the law, and all the issues surrounding discrimination. There are changes in how tests are administered and scored. There are changes in how tests “get to market”
There are plenty of speculators and futurologists in this area, both academic and non-academic, the latter often being science journalists, practitioners and consultants. An example is McHenry (2017), himself both an academic and a test publisher. He made five assertions, nearly five years ago, about the future of psychometric tests:
1. Smartphones will replace computers for employee assessment.
- High-quality psychometric testing services will be sold directly to consumers.
- Advances in the neuroscience of personality will reveal which are the most valid individual differences to measure and how best to measure them.
- The digital badging movement, coupled with the use of big data and new forms of digital CV, will render many of the current applications for high-stakes testing redundant.
- The basis for employee development will in the near future be derived from the data yielded by wearable devices and not from psychometric tests.” (p. 268).
One of the most comprehensive and up-to-date reviews called “Personnel selection in the Digital age Wood et al. (2019) reviewed all recent research 2010-2020. Their focus was on Digital Selection Procedures (DSP) and the main applications and emergent evidence.
They observed: “Digital technology is flexible and easily updated and adapted and so information from users, clients and others can be used to continually and rapidly improve the way that, for example, software or online systems function.”…….. The rapid configurable nature of digital assessments means a fundamental shift in the way we approach validation,from an “endpoint” of instrument creation to an ongoing accumulation of insight into a technique or methodology”(p71)
They detailed many studies which compare old and new methods (electric vs paper-and pencil; proctored vs non-proctored) and different tests (personality vs intelligence). Most showed no differences. However, they do note the problem of impersonation and fraudulent completion of tests and that candidates often preferred internet testing over the traditional methods
New developments in Situational Judgement Tests include the use of videos with some evidence that they were more valid predictors pf work performance that traditional written methods. There is an interest in Digital Interviews where people record video or digital answers to predetermined questions which can be easily used for comparison. This data can be subject to all sorts of AI and other analysis.
Some studies suggest that candidates do not like these techniques being less fair and stilted and “Creepier and less personal” with the traditional methods though that may change over time.
In examing Gamified Assessments they note arguments in favour of reduced faking and social desirability while promoting “fun, transparency, challenge and interaction”. But they suggest that despite much buzz about the use of gamified assessments in practice, there remains scarce published literature on the construct validity of gamified assessments and applicant reactions to them.
With respect to using social media and network sites to gather digital footprints they note that it is possible to gather information that would seem to predict work success like breadth of professional and non-professional experience social capital, interest in updating their knowledge.
In this excellent review they cover various crucially important issues:
Validity
This is clearly the most important issue and they conclude like so many others: “Alongside issues of construct validity is arguably the most critical gap currently in the literature on the validity of DSPs; namely the absence of peer-reviewed published studies of criterion validity. In the papers we reviewed, only two reported criterion-related validity of digital forms of assessment in the context of selection” (p. 69)
Adverse Impact
It is argued that new technology can be used to reduce human bias in selectors but many maintain (through AI technology) biases that are found in society.
Privacy
Clearly some people are really put off the idea that selectors themselves or hire others to screen all their online content. Further it can be challenged in the law
Digital Familiarity
Access to, and familiarity with, technology, may discriminate older, poorer people in developing countries as there is a digital divide.
A Skeptics Response
Sherman (2019) notes in a blog in PSYCHOLOGY TODAY warned about various trends:
Trend 1. Neuroscience
Some companies measure how fast you react to flashing objects on a computer screen and say that their assessments are based on neuroscience. Neuroscience is the study of the structure and function of the nervous system. Even though such a broad definition leaves room for debate, the reality is that neuroscience concerns the function of individual neurons and the brain (i.e., a large mass of neurons)……
Trend 2. Big Data and Deep Learning
Some companies brag about their stacks of big data and their use of machine learning or artificial intelligence to produce talent insights. However, if you dig deep, you find that most of the data these companies collect are useless; they aren’t even using it. For example, millions of mouse-movements, keystrokes, and response times can be measured in a 10-minute assessment. But are they consequential? Do they predict anything?
Trend 3. Gamification
…. The idea is that if job applicants have more fun taking the assessment, they will be less likely to drop out of the application process. Although the data show that candidates do enjoy game-based assessments, the data also indicate that gamification doesn’t improve performance predictions. Research indicates that applicants who drop out during the assessment process are unlikely to be your strongest candidates anyway…..
Trend 4. Profile Matching
First, they assess your high-performers. Next, they see what differentiates your high-performers from a larger population of people who have taken the assessments. The differences between the two create a high-performer profile. Although this profile matching approach used by many companies seems intuitive, only a proper validation study that differentiates high and low performers will give you an accurate profile. Don’t fall for assessments that are only validated on high-performers.
Trend 5. Emphasizing Irrelevant Information
New and old assessment companies often emphasize the total number of applicants, time to hire, and the diversity of the hiring class as selling points……When it comes to performance, the only thing that matters is validity: how well does the assessment predict performance? The reality is that some assessments predict job performance better than others. Assessment companies that don’t show or emphasize validity probably don’t have any. With no validity, they have no choice but to emphasize irrelevant features.
Conclusion
The cozy and profitable world of assessment has changed. Not long ago the situation went like this: Authors and Academics with a test/model went to publishers who sold the printed tests to consultants who sold them in some form to clients. Probably the authors made the least and consultants most money in this chain.
Now Techie-startups seek out authors/academics to help them devise state-of-the-art, delivery platforms they sell to anyone. The “middle-men” get cut out. There are now a number of new products in the assessment business and the buyer is spoilt for choice. They look wonderful; and promise the earth: assessment is faster, deeper, cheaper and more fun. Indeed: but is this at sacrifice of validity. The problem is that it takes time and money to get the data to establish test validity: and some entrepreneurs are not willing to wait.
Venture capitalists have noticed these new assessment companies, and many are happy to invest. Hence the growth in assessment companies and a complex, crowded and confusing marketplace. Some of the old hands, like those above, caution against all the hype and a new south sea bubble. Of course there is a difference between skepticism and cynicism, and being Luddite in these new and exciting times. So as always caveat emptor.
About the Author
Adrian Furnham is Professor of Psychology at BI: Norwegian Business School. He is also an author and devisor of psychological tests. He has been used by a number of organisations to investigate the psychometric properties and claims on new tests
References
- Chamorro-Premuzic, T., Winsborough, W., Sherman, R., & Hogan, R. (2016). New Talent Signals: Shiny New Objects or a Brave New World?. Industrial and organizational psychology: Perspectives on science and practice, 9(3), 621–640.
- Furnham, A. (2021). Twenty Ways to Assess Personnel Different Techniques and Their Respective Advantages. Cambridge: CUP
- Ihsan, Z., & Furnham, A. (2018). The new technologies in personality assessment: A review. Consulting Psychology Journal: Practice and Research, 70(2), 147–166.
- McHenry, R. (2017). The future of psychometric testing, In B, Cripps (Ed). Psychometric testing: Critical perspectives. London; Wiley pp. 269-281.
- Sherman, R. (2019). Beware These Marketing Trends in Psychological Assessment And why you shouldn’t fall for them. Psychology Today: October 17th
- Woods, S. Ahmed, S., Nikolaou, I., Costa, A., & Anderson, N. (2020). Personnel selection in the digital age. European Journal of Work and Organisational Psychology, 29 (1), 54-77.
