Posts by linger
Last ←Newer Page 1 2 3 4 5 Older→ First
-
Speaker: ‘Kiwimeter’ is a methodological…, in reply to Tze Ming Mok,
One might politely enquire as to how the percentage of Maori respondents currently compares to the population proportion.
-
Weird… I was in the middle of replying to Tinakori myself. What we do know is that the question was intentionally worded that way to mirror the phrasing of talkback discussion – which probably is intended to provoke reaction. So, yes, kind of, but not in that direct a fashion.
-
Speaker: ‘Kiwimeter’ is a methodological…, in reply to Geoff Lealand,
a neutral mid-point is essential, for it can include a range of possible motivations
That's exactly why it's dispreferred: by design, it provides a response that is not easily given one single interpretation. (That said, a neutral option can be offered when those other competing responses are less likely: e.g. on uncontroversial topics, in short questionnaires that are easy for their respondents.)
-
Speaker: ‘Kiwimeter’ is a methodological…, in reply to Sacha,
Reverse scored items have their purpose. It is still recommended that at least one opposite-polarity item be included in each set of items attempting to measure the same thing – as an internal check that respondents are reading the questions rather than blindly choosing the same option, and as a way of minimising the inherent bias arising from forced choices (see previous comment). But wording such items takes some care, as they shouldn’t be made harder to understand than the corresponding positive statement.
-
Speaker: ‘Kiwimeter’ is a methodological…, in reply to Andrew Robertson,
>#ad-targetting
I guess that is possible, but I really really doubt it.
It's hard to see any other meaningful purpose to the exercise, though.
-
There are arguments on both sides as to whether semantic-differential or Likert scales should have an odd number or an even number of options.
The former allows a neutral midpoint, which may well be a valid response, but can also be a popular choice if respondents are tired, or don’t want to think about the question (which is not the same as being neutral), or don’t understand the question.
The latter forces respondents to read and think about the question and choose an opinion in one direction, even if only by a slight margin; but it removes the ability to signal true neutrality on a single item. However, neutrality may still be allowed to emerge by including two opposite-polarity items asking for the same information. -
Is Kiwimeter “research” at all?
Results of an online self-selected survey (which may be expected to be biased towards middle-class Pakeha) show, at best, groupings of beliefs that co-occur in subsets of that dominant set of respondents.
Use of factor analysis based on that dataset will further consolidate dimensions based on Pakeha opinion, while combinations of views representing minorities will be lost in the noise. In case it’s not obvious: this step of the analysis assumes that the population has one consistent factor structure for its beliefs, and that groups of respondents merely differ quantitatively on those factors. If minority groups have qualitatively different belief structures, that is not allowed to emerge: instead, what is constructed is a factor structure for the dominant group. (This remains true even if responses for underrepresented minorities are weighted to more accurately reflect their share of the overall population.)
Clusters are constructed from scores on the overall dimensions, so retain and reify the bias towards the dominant factor structure.
Subsequent thinning of items for the final survey further reduces robustness of results for minorities.
Hence extrapolation of findings to “New Zealand” as a whole is dubious in the extreme.
Media organisations will want a survey with results reflecting their target disposable-income demographic, and biases in that direction are fine with them.
University researchers, however, should know better. -
Speaker: ‘Kiwimeter’ is a methodological…, in reply to Moz,
The whole purpose of a pilot is to improve the survey design so as to remove item ambiguity, maximise completion rates, and maximise interpretability of the results. (“Making the test shorter” is not an aim in itself: it is only useful if it leads to higher completion rates, without harming interpretability.) So those are things you definitely need to measure for the pilot: non-completion has to be recorded.
Serious ambiguity should first be eliminated by testing items on small focus groups, who are also interviewed to check their understanding of and reactions to each item and response option. Then a second stage pilots the entire draft survey by asking respondents to complete a comment log as they fill out the survey (possibly also with follow-up interviews). This can be used to address problems in item sequence and timing, as well as remaining ambiguities and difficulties in the individual items. All of this is supposed to happen before doing any testing on the general public.
But, for “Kiwimeter”, the results of the public “pilot” survey were instead used only to narrow the range of items to those most strongly characterising the groups of respondents (“clusters”) with similar response patterns in the pilot results. Removing item redundancy in this way reduces survey length, but it also reduces item interpretability, as you lose the ability to crosscheck item responses to confirm the meaning inferred by the respondent. Which is important because, with an increased diversity of respondents, there is a risk that some items retained will fail for some groups underrepresented in the pilot stages – and so the overall results will be less robust.
-
Hard News: Media Take 2016: the new…, in reply to Daniel Carr,
Surely you jest, given TV3 management's track record in recognising quality content. More to the point, why isn't TV1 running it instead of seven blunt?
-
…apologises for an entire blog about survey methodology
No apology necessary. Survey geeks represent!
On the cognitive testing/ factor analysis confusion:
Maybe the “special treatment” item does correlate reliably with many other items, mutually measuring some underlying dimension of belief (accepting/ rejecting the dogwhistle probably correlates quite highly with social conservatism/ liberalism, voting National/Left, etc), so a statistician looking at the results (possibly without even reading the wording of the items) will see the item as psychologically valid: it “works” in the sense that it helps measure something, for most of the respondents in the dataset.
I gather this is where van der Linden is coming from.However, (i) it is a bad idea to include an item that is ambiguous or stressful to answer, even if that item scores as statistically “valid”, as the reaction to that item may affect responses to, or completion of, subsequent items too; and (ii) it is a really bad idea to include an item that is systematically more ambiguous or more stressful for one particular subset of respondents. If that subset is a minority, you won’t be able to notice any problem by looking at overall factor loadings in a pilot survey; you can only identify the problem by cognitive testing (piloting with respondents self-reporting their interpretations and emotional responses to items).
N.B. for this purpose the pilot group doesn't have to be representative, but it does have to be diverse; a stratified sample (i.e. with specified quotas for groups whose responses will be compared) would be better than a random sample. A self-selected sample is the worst of all possible starting points for a pilot of a survey intended for the general population.
After the full survey is performed, if you then check the factor loadings of each item for each subset group separately, you might notice differences in completion rates, and/or in factor structure. But by then, it’s too late: your results are already suspect, and you’ve damaged your credibility among the survey takers.