Hard News by Russell Brown

Read Post

Hard News: The shaky ground of psychometric testing

154 Responses

First ←Older Page 1 2 3 4 5 6 7 Newer→ Last

  • Rich of Observationz, in reply to SteveH,

    I think the questions were put there in response to various moral panics over the years.

    Back in Wellington • Since Nov 2006 • 5550 posts Report Reply

  • Ian Dalziel,

    Alan Key...
    Mandatory Turing Testing for management?

    Christchurch • Since Dec 2006 • 7881 posts Report Reply

  • Sacha, in reply to BenWilson,

    because you'll probably lie

    screws the aspies

    Ak • Since May 2008 • 19667 posts Report Reply

  • Sacha, in reply to Martin Brown,

    whereas I only got to supervise preparing breakfast for my department at a retreat.

    Ak • Since May 2008 • 19667 posts Report Reply

  • andin,

    Worked for a large multinational ad agency which used to do that “survey” across the entire group, each year.

    Oh you lucky lucky bastard!
    The fancy jobs get "psychometric testing" those lower down just get drug testing.

    raglan • Since Mar 2007 • 1881 posts Report Reply

  • izogi, in reply to BenWilson,

    I found the Myer Briggs analysis useful when done on myself, by myself.

    Is that possible? I remember reading many years ago that one of the things about the MBTI is that it can only be administered by a qualified person, in part, I think, because of the ease with which a result can be misinterpreted and used in the wrong way. I’m wondering if you mean something like the Keirsey Temperament Sorter, which was a very similar set of questions once published in a book about personalities.

    Back in the day when I was really fascinated by all of this stuff to the extent of joining mailing lists full of “like-minded people” on the internet, I ended up finding an evening course (at Wellington High) on personalities using the MBTI, and where the trainer administered it to participants properly. I went into that test knowing exactly which personality I wanted to be, and I totally nailed it to the extent that the course instructor was holding me up as an example of one of the more clearly defined personalities. An interesting effect, in hindsight, of my getting hung up in the whole idea in a way that was probably not very healthy or useful at the time.

    As others have said, I think the greatest thing about this type of stuff is to gain a better understanding of how there are actually different types of people out there, and that judging someone because they see or approach something in a different way from yourself is not always helpful.

    It’s also often more about personality than ability. Making critical presumptions about a person’s abilities, based on their personality (if it’s safe to even assume a test is accurate), seems like a very dodgy business.

    But hey, many psychometric tests probably aren’t all personality tests, plus I’m not a psychologist, an HR person, or a manager.

    Wellington • Since Jan 2007 • 1139 posts Report Reply

  • 81stcolumn,

    Okay – This is a long post but I had to say something……………

    Some points to consider when using psychometrics of any kind in the workplace.

    Problem 1.

    i) Take one group of people tell them that you will film them.
    ii) Tell them that you are measuring a desirable behaviour for example, letting someone finish a sentence (‘cos we all know that’s a good thing don’t we?).
    iii) Do above measurements.
    iv) Ask the same group to recall how many times they let people finish a sentence in a conversation.
    v) Compare scores.

    We found that even though people knew that they could not lie about it, they still over-estimated the number of times that they demonstrated a desired behaviour. Their recall reflected their beliefs about themselves: “I am a sociable person - so of course I let people finish a sentence”. This is not duplicity but the normal functioning of a social brain. I am not arguing that there is no deliberate image management going on when completing psychometrics. I am arguing that even if there wasn’t bias would remain. The way people view themselves is a function of the way they wish to be, this process is to some extent unconscious or more specifically due to memory priming effects.

    My point: People tell the truth as they experience it and may not be lying if their behaviour differs from what they say in a questionnaire.

    Problem 2.

    Even the best of Psychometric theory is grounded in the concept of latent variables. Latent variables reflect the balance between things we cannot truly share with others because of the uniqueness of experience/meaning and those things we can measure using common language which reflects experience/meaning that we can share. For example most people can agree on whether the animal in a picture looks like a cat, but when asked to describe what defines a cat those definitions will vary. Factor analysis and structural equation modelling seek out the common dimensions of meaning and those questions which best reflect them. Latent meaning remains in both the respondent and the person interpreting the results of psychometric measurements. Consequently a large degree of variability in meaning remains when one person fills out a questionnaire and another person tries to derive meaning from those responses. This is why psychometrics can be both valid and reliable but only meaningful to the respondent. Add another layer of meaning to their interpretation and value diminishes. Psychometrics are at their best when measuring changes in personal experience not choosing between different individual experiences of the same socio-cultural phenomenon (the rules change for large numbers of people but still sub serve to explained variance).

    My point: Anyone who chooses between individuals using psychometrics does not understand their purpose. At best they permit decisions about what other measurements to take from people. The origins of psychometrics are in norms and diagnostics (still problematic), not deliberate selection between individuals.

    Problem 3.

    Raymond Cattell, the father of all things psychometric proposed a model of personality that included questionnaires, observations and behavioural test data. The main point here is that good practice (as has been advocated for more than 50 years) demands data from two or more valid, reliable sources. At the end of this process you might argue to know something about a person. Even then, you still have the problem of matching knowledge, selection and workplace performance. That is to say, do you have any clear evidence or robust theory about who would make the best contribution to workplace performance? I stand to be corrected here, but to my knowledge no robust theory exists that can link psychometrics to workplace performance. A problem exemplified by the nature of decisions involved, should the person required; i) Duplicate or replicate existing skills in workplace (affirmation approach). ii) Complement or contrast with existing skills in the workplace (additive/diverse or dialectic). iii) Like or value the skill involved with the job or a particular organisation regardless of the above (personality matching). Data driven theories that actually use meaningful performance measures in this respect are sparse to the point of non-existent.

    My point: Theory linking psychometrics (even when performed under best practice guidelines) still has to be matched to organisational needs; this problem has never been adequately addressed.

    Footnote:

    I have yet to read a convincing argument for the derivation of binary constructs (eg. Introversion-Extroversion) from factor analysis neither the method nor the statistics are designed to produce this outcome. Structural equation modelling cannot do this either; though you can use binomial regression to test derived models which would argue for the same thing.

    I have been surprised by the largely atheoretical approach in management, business and HR when it comes to using SEM (causal correlative modelling). SEM like factor analysis is very prone to GIGO albeit in a more complex and modern way.
    No evidence for learning styles or functionally equivalent processes in the brain (sorry folks!).

    Problem 1 is based on data I have published but using slightly different variables.

    Final point; There is no advantage to using psychometrics for workplace selection over managerial intuition other than that it has the cloak of perceived (but erroneous) scientific legitimacy.

    Minor corrections 21:52.

    Nawthshaw • Since Nov 2006 • 790 posts Report Reply

  • Rich Lock, in reply to B Jones,

    I think it selects for people who are ok with putting their employer’s interests ahead of the normal rules of interpersonal interaction.

    I had something of an epiphany a year or two ago when I realised that the function of HR was not to act in any way whatsoever in the interests of the employee, but rather to act as an enforcement arm for management to prevent uppity employees from any sort of boat-rocking.

    'They're just questions, Mr Lock. Perhaps you'd like to tell me about you mother'

    back in the mother countr… • Since Feb 2007 • 2728 posts Report Reply

  • BlairMacca,

    We use these tests to death in my line of work (social sector agency). But I have seen them used well, particularly going into management (god forbid) to understand people’s communication styles so I can tailor my approach to different people. So its a small tool, amongst actually getting to know staff, for me as a manager, but not to find out who I should or shouldn't fire.

    Wellington • Since Apr 2007 • 208 posts Report Reply

  • Susannah Shepherd, in reply to Morgan Davie,

    That said, I have come across (second hand) a recruitment consultant who uses psychometrics in a way that I can cautiously endorse – principally by making psychometrics subordinate to other evaluations and processes, rather than using them as an all-powerful oracle or as floating evidence that can rationalise any desired position.

    I suspect this is a fairly common way to use psychometrics in workplaces that make the hiring choice directly. It would be the only effective way to use pysch tests in restructures that involve changing how the business operates rather than just shedding jobs (past performance isn't much of a predictor if the new roles need different skills).

    I do quite a bit of recruitment, supported by HR, and we always use the "personality" tests as a way of tailoring some of the things we'd probe into in an interview (i.e. inquiring about how a rules-driven person copes with stuff coming out of left field). And if someone comes out high on the anxiety spectrum, we'll cut some slack for interview nerves if we see signs of them.

    We don't screen people out of the process on the basis of anything in the personality profile, although I can see how this would be a temptation if you're dealing with 400 applicants instead of the 10-50 I normally get. I work in an area where we struggle to find as many good candidates as we'd like, so we actually want to see the potential in people, not find excuses to kick them out of the running.

    Wellington • Since Jan 2008 • 58 posts Report Reply

  • Ben Austin,

    While I am rather surprised that anyone would think it a good idea to use personality/psych testing as part of a selection process, I would note that in the vast majority of the redundancy process I've consulted on, the employer/management team are absolutely stuck as to how to appropriately and fairly access staff.It can take quite some mucking about before something is found and I guess on that basis, the temptation to use psych testing must be rather strong as it appears like a sound methodology.

    It is rather hard for most employers because the kind of information or records one keeps on file is not necessarily going to be useful in making that kind of selection. The only people who find this easy would be those who are well organised, who make a deliberate effort to conduct focused appraisals, who keep detailed employment files. So, usually those with HR teams.

    London • Since Nov 2006 • 1019 posts Report Reply

  • Bart Janssen,

    So this is my fundamental problem with all these tests ...
    There are approximately 20 billion neurons in your head, yes billion!
    Each neuron has upwards of 2000 connections with other neurons.
    Each connection is variable, meaning it is much more than a simple on/off switch.

    And these tests resolve that complexity to how many categories? Absurd!

    Auckland • Since Nov 2006 • 4450 posts Report Reply

  • Rich Lock, in reply to Bart Janssen,

    back in the mother countr… • Since Feb 2007 • 2728 posts Report Reply

  • Ben Austin, in reply to SteveH,

    t

    I thought it was in part (the moral panic argument sounds right too) to later provide an excuse to withdraw permission, residency or citizenship to entrants/applicants who had lied on the form. I'm thinking here about the various European migrants to the US after the war who later turn out to have been war criminals.

    When I've gone through immigration forms in the UK, they always make it quite clear on this kind of question that lying could later result in such an outcome.

    London • Since Nov 2006 • 1019 posts Report Reply

  • Lucy Stewart, in reply to Ben Austin,

    I thought it was in part (the moral panic argument sounds right too) to later provide an excuse to withdraw permission, residency or citizenship to entrants/applicants who had lied on the form. I’m thinking here about the various European migrants to the US after the war who later turn out to have been war criminals.

    When I’ve gone through immigration forms in the UK, they always make it quite clear on this kind of question that lying could later result in such an outcome.

    Bingo. They can't toss you out of the country for crimes committed in another jurisdiction (especially when they may or may not have been illegal in that jurisdiction) but they CAN throw you out for having lied on the application. Well, they can also toss you out for the vague "displaying poor character" (also known as "the reason I won't be trying marijuana even if I visit Colorado") but I believe that usually requires being charged with something in the US.

    Wellington • Since Nov 2006 • 2105 posts Report Reply

  • Ben Austin, in reply to Lucy Stewart,

    In which case it is a pretty good way of dealing with that particular problem, plus it gives the rest of us a good laugh.

    London • Since Nov 2006 • 1019 posts Report Reply

  • Lucy Stewart, in reply to Ben Austin,

    In which case it is a pretty good way of dealing with that particular problem, plus it gives the rest of us a good laugh.

    Dunno, the Communist Party question is still insulting - membership is not illegal in the US and never has been, and your average Russian spy probably isn't a member of the Communist Party these days. Plus they gender-link some of the terrorism-related questions, which is frankly weird.

    Getting more on-topic, psychometric testing for hiring purposes (rather than redundancy or management decisions) is actually not legal in the US. That doesn't mean it isn't used, of course, it just means that companies that do open themselves up to lawsuits. OTOH, in the current employment climate they can pretty much ask applicants to dance the macarena naked and not get a quibble, so the odds of being sued for it are low.

    Wellington • Since Nov 2006 • 2105 posts Report Reply

  • Ben Austin,

    I don't think psychometric testing is unlawful in the UK unless it is used in such a way that could support a discriminatory outcome, which certainly is possible. Or perhaps if the testing incorporated some sort of pre contract health questionnaire (which are illegal). There are a few cases that go either way really.

    London • Since Nov 2006 • 1019 posts Report Reply

  • BenWilson, in reply to izogi,

    I’m wondering if you mean something like the Keirsey Temperament Sorter

    No, I meant MBTI. It was part of management training. We did the test, then got trained on how to interpret it, over a couple of days (amongst other training). But they didn't interpret it for us, the aim was to get us to understand what it might mean for ourselves, and also to get the limitations. It made sense, because pitching it like it was actually some kind of test that had to be passed, would have lost pretty much all the value that I could see in it. It wasn't for screening purposes, but training purposes.

    I researched it quite a bit before and afterward as well just to know what I was getting in for. It's an interesting idea. But, as Bart says above, minds are pretty damned complex things, and trying to cluster them in 4 dimensions leads to massive simplifications. Add another dimension, and you can find people who are in the same one of the 16 MB categories might be poles apart. Why choose those 4? The division between the ideas was not really that clear, nor do they comprehensively cover the ways we learn and interact. Also, they can be highly situational - a person can go from introvert to extrovert in the right context, so it's really rather arbitrary to just sum them up as an extrovert for answering that they're comfortable in enough situation associated with extroversion. MB's idea of what extroversion even meant was rather idiosyncratic - it's "draws energy from other people, rather than from themselves". What about people who draw energy from something else altogether, or just seem to have an unlimited supply, or no supply? Or they draw it only from specific people, whom they are always around? And what is this energy anyway?

    Auckland • Since Nov 2006 • 10629 posts Report Reply

  • Lucy Stewart, in reply to BenWilson,

    What about people who draw energy from something else altogether, or just seem to have an unlimited supply, or no supply? Or they draw it only from specific people, whom they are always around? And what is this energy anyway?

    The "spoons" kind of energy, is my understanding - after a hard day, do you recharge by hanging out with friends or by having alone time? Is social interaction something that you reach a limit on relatively quickly? Like you say, it's more a gradient than a binary, but I find it a broadly useful idea about how people work.

    Wellington • Since Nov 2006 • 2105 posts Report Reply

  • BenWilson, in reply to Lucy Stewart,

    do you recharge by hanging out with friends or by having alone time? Is social interaction something that you reach a limit on relatively quickly?

    Are these questions really about the same thing?

    Like you say, it’s more a gradient than a binary, but I find it a broadly useful idea about how people work.

    To be fair, the testing gives a score along a scale with a discrete even number of points, not necessarily 2. I think we were doing 8 on the test I had, so the level of each attribute was something that you also were told to consider. But it's still only 4 dimensions.

    I found it somewhat useful as a thinking tool, too, but mainly because it drew attention to something, gave a little bit more structure to the analysis. But you have to be careful not to overgeneralize with it. OTOH, management is often struggling in the dark, with soft squishy overgeneralized ideas at all times, so you tend to latch onto what you can.

    Auckland • Since Nov 2006 • 10629 posts Report Reply

  • Ross Mason, in reply to Rich Lock,

    I had something of an epiphany a year or two ago when I realised that the function of HR was not to act in any way whatsoever in the interests of the employee, but rather to act as an enforcement arm for management to prevent uppity employees from any sort of boat-rocking.

    BINGO!!!!!!!!!! Bingo Bingo Bingo ! ! !

    Upper Hutt • Since Jun 2007 • 1588 posts Report Reply

  • Don Christie,

    This testing may tell you something about your tempriment, or not. The evidence is to the latter. However, it doesn't at alll work as a predictor of your suitability for a particular job.

    "Hey, we've never had an opera lover in this role."
    "That must mean opera lovers aren't suitable for this job."

    "Hey, the only person in our team that loves opera is Emily."
    "Doesn't sound very compatible with the team".

    Emily gets the pink slip.

    ================

    The lesson from all this? If you're an opera lover...lie.

    Wellington • Since Nov 2006 • 1645 posts Report Reply

  • Rich of Observationz,

    your average Russian spy probably isn't a member of the Communist Party these days

    I'd imagine the "are you a member of the Communist Party" is asked of potential FSB employees, and with a negative answer required.

    Back in Wellington • Since Nov 2006 • 5550 posts Report Reply

  • Bart Janssen, in reply to Rich Lock,

    HR was not to act in any way whatsoever in the interests of the employee

    Um yeah. They are employed by management to manage a problematic resource with least disruption to business. You couldn't find colder fish in the antarctic ocean.

    Auckland • Since Nov 2006 • 4450 posts Report Reply

First ←Older Page 1 2 3 4 5 6 7 Newer→ Last

Post your response…

Please sign in using your Public Address credentials…

Login

You may also create an account or retrieve your password.