Cleaning up a Questionnaire
A: The content to be modelled A: The Learning and Development Inventory (LDI) questionnaire
B: How we modelled 'The Learning and Development Inventory'
C: A summary of criteria used to suggest improvements to the LDI
Researchers at The Yale University Child Study Center produced The Learning and Development Inventory (LDI), a questionnaire containing 125 items to be administered to thousands of high school students in the USA. The aim of the questionnaire was to study the relationship between academic success and personal and social development. See: http://impactanalysis.org/products_ldi.htm
We were invited to review the questionnaire and offer suggestions for improvement from a clean/NLP perspective. Below is an extract from the LDI. It shows the introduction telling the student how to complete the survey and eight statements we have selected which cover a range of criteria we used to suggest improvements.
To improve the quality of the results obtained by the questionnaire we modelled what students would likely have to do internally to make sense of each statement and how that might influence their answer. We then:
- Proposed a ‘cleaner’ wording for each statement. One which gave the student more freedom to answer from their own experience rather than to be constrained or primed by the statement. (While aiming to preserve what appears to be the purpose of each statement.)
- Gave a brief explanation of our reasoning for each of our suggestions.
Below is an extract from the High School LDI with 8 sample questions
- Summarised the criteria we had used to evaluate clean-ness and to suggest improvements.
The purpose of this survey is to understand how students like you experience their high school years. This is a survey, not a test. There are no right or wrong answers. Please be as honest as you can be. We promise not to share your individual answers with anyone. Please tell us how strongly you agree or disagree with each statement by filling in one of the five responses:
Scale: Strongly Disagree Disagree Not Sure Agree Strongly Agree
1. When I see or hear or feel something that confuses me, I think about it until I can make sense out of it.
2. If another student makes me angry, I am able to put my anger aside after a short time.
3. I write in my day planner all the small steps that I have to take over time in order to accomplish a far-away goal.
4. When another young person is being made fun of in a humiliating way, I join in the laughter.
5. In just about every class, my teachers are making sure that I want to really think about what is being taught
6. Whenever I am working on a project or assignment, I review my work frequently to see if I can find a way to do it better.
7. When I am having personal problems because of something going on at school, I find an adult—either in school or outside school—and I ask for help.
8. I don’t like writing papers/essays, because I have a hard time coming up with ideas to write about.
B: Our Modelling produced suggested improvements1.
When I see or hear or feel something that confuses me, I think about it until I can make sense out of it.
When something confuses me I think about it until I can make sense of it.
- Simplified by removing the specific means of being confused “I see or hear or feel”.
- Simplified by removing the metaphor “out”.
If another student makes me angry, I am able to put my anger aside after a short time.
If I get angry at another student I let go of it after a short time.
- The suggested formulation does not presuppose that another student can "make" them angry.
- Is the intention to find out whether they think they “am able to” or whether they actually do something to reduce their anger? If the student thinks “I am able – I just don’t” how should they answer?
- ‘Let go’ of anger is an alternative and possibly a more common metaphor than “put aside”.
I write in my day planner all the small steps that I have to take over time in order to accomplish a far-away goal.
I write in my day planner the steps I need to take to accomplish a future goal.
- “all” and “small” limits the scope and requires an extra evaluation.
- “have to” is often associated with obligation, whereas “need to” is more a requirement.
- "Over time" is presupposed by “small steps” and “far-away” and is therefore unnecessary.
- “far-away” is a clear spatial metaphor while ‘future’ or perhaps ‘long-term’ is more general. These would make sense both to people who have “far-away” goals and those who don’t.
- Is the question about achieving the outcome, or the process of achieving the outcome?
When another young person is being made fun of in a humiliating way, I join in the laughter.
When another person is being made fun of in a humiliating way I join in the laughter.
- The modifier "young" limits the context. Does it matter whether the person is “young” or not?
In just about every class, my teachers are making sure that I want to really think about what is being taught.
In most classes my teachers encourage me to really think about what is being taught.
- “most” is simpler to compute than “just about every”
- Our formulation removes the presupposition that teachers are "making sure" that students "want" to think about what is being taught. We think it is sufficient for teachers to ‘encourage’ students to do the act of thinking about what is being taught. This leaves the responsibility for what the student “wants” with them, and the behaviour of ‘encouraging’ with the teacher.
Whenever I am working on a project or assignment, I review my work frequently to see if I can find a way to do it better.
I review my work to see if I can improve it.
- "Whenever” means it always happens. If they only review it most of the time should they “agree” or “disagree”?
- How often is “frequently”? If they only need to review their work once or twice, do they “agree” or “disagree”?
- “I am working on a project or assignment" specifies two particular contexts. If the answer is limited to these, what about other types of work? We think the statement is simpler and more direct without this introduction.
- Replaces “find a way to do it better” since it is presupposed in “improve it”.
When I am having personal problems because of something going on at school, I find an adult — either in school or outside school—and I ask for help.
If I have problems because of something going on at school, I ask an adult — either in school or outside school — for help.
- “When” presupposes they have had (or will have) “personal problems” whereas “if’ leaves it open.
- “I am having” more likely associates the reader into the ongoing experience of personal problems compared to “I have”.
- “Personal” limits the kind of “problems” they can consider. What one person considers personal could be completely different to someone else. Whereas "problems" on its own leaves the reader free to choose the kind of problem that they want to answer about. It depends on the purpose for the statement: Is it more interested in the personal aspect or the ‘asking an adult for help’ aspect? Our suggestion assumes the latter.
- The statement is ambiguous, how should a student to answer the following?:
I have personal problems at school and I ask an adult for help.
I have personal problems at school and I don’t ask an adult for help.
I have problems at school but they are not personal and I ask an adult for help.
I have problems at school but they are not personal and I don’t ask an adult for help.
I don’t have personal problems because of something going on at school.
- Simpler to just say ‘ask an adult for help’ since this presupposes they “find” and adult.
I don’t like writing papers/essays, because I have a hard time coming up with ideas to write about
I don't like writing papers/essays.
It is difficult to come up with ideas to write about in papers/essays.
- Is the question trying to find out if students like writing papers/essays or whether they have a hard time coming up with ideas? By having both in the sentence you will not know which their answer applies to.
- It may also bias the answer to “disagree” as you have given them two chances to disagree. Of course you’ve also given them two chances to “agree” but we suspect that the ‘away-from’ reaction might override the ‘towards’ response.
- "Because" specifies this particular reason for not writing papers and essays, and there might be plenty of other reasons.
- If the student agrees with one half and not they other they may well experience a form of ‘cognitive dissonance’ which may affect their state which in turn may influence their answers to subsequent questions.
C. A summary of criteria used to suggest improvements to the LDI
Aim to preserve the meaning/purpose of the original question/statement.
Get clear about the information the statement aims to discover.
What part do you want the reader to most attend to?
Minimise students’ cognitive load:
Use simple vocabulary.
Use simple sentence construction.
Arrange the sentence in the sequence that the events happen.
Minimise number of things the reader has to agree/disagree with (ideally one).
Maximise comprehension for the greatest range of people.
Keep it simple:
Use as few words as possible (remove unnecessary words).
Only ask one question - do not ask multi-part questions.
Do not duplicate information that is already presupposed.
Do not restrict by specifying the means of something happening.
Do not restrict with unnecessary modifiers.
Do not specify the context unless this is precisely what you want.
Minimise use of explanations, e.g. “because”.
Maximise applications – unless want answer to be in a specific context.
Use generic metaphors rather than specific metaphors.
Don’t use metaphors that presuppose events/people cause internal reactions
Do not refer to internal motivations. Use behaviours instead.
Ensure responsibility is with the right agent.
Avoid pre-defining experience
Do not presuppose the reader has had a particular experience.
Avoid associating reader into an un-resourceful state.
Minimise implied judgements.
Do not use universals of:
scale “just about every”