Lorem ipsum dolor sit amet consectetur.

How do you determine if you need an external research firm to assist with your survey? Surveys are great tools to help you assess an organization’s culture, measure customer satisfaction, and gain critical information to help you develop effective strategies to address many organizational challenges. It’s no wonder survey platforms such as SurveyMonkey, Zoho Survey, SurveyGizmo, and SoGoSurvey have become so popular—they make it quick and easy to design, execute and report the results of your survey. But do DIY survey tools provide you with the kind of valid, reliable and actionable data you seek? Possibly; but ensuring confidence in your survey results is less about the technology employed and more about the skillset and experience of those who are designing and administering the survey instrument, then analyzing, interpreting and reporting the results. If you have the internal resources and expertise, by all means you can probably go it alone. If you’re not sure, here are six questions to ask yourself to determine if you need an external research firm to assist with your survey. 1. Are you able to ensure confidentiality? Many surveys address very sensitive topics and, often, respondents are concerned about how their confidentiality will be ensured and how their responses might be shared. For example, if employees who work in a safety conscious environment are asked to participate in a survey, they will want to be sure that if they provide honest and open evaluations of their workplace they can do so without fear of retaliation. They need to have strong assurances of anonymity and awareness that they can provide critical opinions without being personally identified. While you may be able to provide such assurances, this condition can best be guaranteed when surveys are conducted and results are analyzed by an independent, external party. 2. Will your stakeholders have enough confidence in the neutrality of the survey and its results to participate? Similarly, to ensure sufficient survey participation, it is often helpful to involve an impartial third party that can work with and reconcile the needs of different groups of stakeholders. For example, to gain wide support and endorsement of the effort, it is best practice to have a neutral party engage representatives of various stakeholder groups in the design process to ensure the validity and reliability of the line of questions. In addition, knowing that an impartial third party is responsible for data collection and analysis can provide respondents with the needed assurance that survey results will be accurate, neutral, confidential and credible. 3. Are you confident in the validity and reliability of the survey design? To provide objective and actionable assessment, it is crucial to ensure that the survey instrument is both valid and reliable. This means that the instrument must actually measure the dimensions it was designed to measure and do so accurately. The questionnaire design should follow rigorous, tried and true methodological procedures for design, review, and pre-testing. Ideally, one would, for instance, conduct a literature review of existing or similar survey results, review and map survey questions, then conduct pre-testing and revision of the survey instrument, all before it is fully deployed. Only a trained and experienced methodologist can provide such assurances. 4. Do you have the needed expertise to conduct, administer, analyze, and report survey findings? External research firms have specialists who are expert not only in design methodology but in analysis and reporting as well. Because they have extensive experience administering surveys and with the facilities and equipment required to do so, they almost assuredly will be able to provide you with better quality results—faster and at a lower cost—than you could yourself. 5. Are you prepared to conduct a survey awareness campaign to maximize stakeholder willingness to participate? We find that a well-prepared survey awareness campaign provides a significant boost in response rates and mitigates any potential stakeholder concerns regarding the legitimacy of the survey effort and confidentiality in its results. Such a campaign requires a sequence of multiple messages as well as materials that inform stakeholders about the survey effort and address how privacy, confidentiality, and aggregated reporting will be ensured. You should be prepared to conduct such a campaign or rely on others with experience doing so. 6. Would you benefit from a new perspective? When a survey is conducted by an independent and external organization, it provides the added benefits of objectivity and a new perspective. For example, within a large, diverse organization, such objectivity can provide a means of capturing a unified picture of the work environment that goes beyond the perspective of any particular individual or group, while at the same time ensuring that diversity across the organization in not overlooked. Then, assuming a consistent set of measures is applied across the organization, meaningful comparisons can be made and benchmarks can be standardized across the organization. Article adapted from Conducting Safety Conscious Work Environment Survey — The Case for Engaging an External Survey Organization.

Behavioral economics is a broad discipline that includes both neo-behaviorism as well as cognitive decision theories that describe the economic behavior of what are essentially, irrational humans. In their seminal book, Nudge: Improving Decisions about Health, Wealth, and Happiness, as the title suggests, Thaler and Sunstein brought the behavioral economic framework to a broader understanding of behavior change. It is through this far narrower prism that the nudge came to represent behavioral economics in its application to public policy and health promotion. There is plenty of evidence that the “nudge” has been in vogue recently, both in the EU and US, as the platform for social and behavioral change.[1] [2] What is Nudge? One of the seductions of the nudge is that it proclaims that substantive changes in human behavior can occur with only slight, but clever manipulations of the decision space in order to favor promoted behaviors without prohibiting any alternatives or changing the incentive structure. Thus, the goal of a nudge is to make engagement in the desired behaviors less effortful than the alternative, more salient and, in fact, almost unconscious or “automatic.” Nudge in our daily lives In fact, we experience these nudges in our daily lives in relatively mundane situations. For example, in the workplace signing up for a retirement plan is the default option to increase saving rates (requiring an employee to take extra steps to opt-out); “smart lunchroom” approaches make healthy sides the default meal option (therefore forcing one to ask for greasy French fries); and at the register, healthy snacks are displayed more prominently than unhealthy ones. Moreover, to increase physical activity certain building designs feature staircases that are more attractive than elevators, and elevators display reminders to take the stairs. Famously, pictures of flies were put in urinals as the reminder nudge to reduce “spillage.” Nudge or nannyism? This decision space engineering approach is often contrasted with population-based regulatory approaches which nudge proponents termed “Nannyism.”[3] This “Libertarian paternalism” as endorsed by Thaler and Sunstein, which does not ban or restrict any behavior in principle, is offered as an alternative. The nudge is also contrasted with social marketing.[4] Nudging is advocated as a cheaper and more effective alternative to more intensive efforts such as media campaigns that rely on changing attitudes as a prerequisite to behavior change. Some proponents of behavioral economics, in fact, claim that behavior change can occur without changing minds.[5] Does Nudge always work? However, before we pen a requiem for social marketing as we know it, we should take a closer look at the behaviors given as examples of successful nudging, such as savings, recycling, healthy eating, or increasing physical activity. One thing becomes apparent: most of us endorse all of these are behaviors, at least in principle. So, what happens when audiences do not endorse the behavior? In one case, low-income taxpayers did not default to saving when their tax returns were defaulted to US saving bonds.[6]The reason offered by the authors is that the low-income taxpayers already allocated their returns to specific expenditures; saving the tax return was not something on their agenda. Similarly, opt-out approaches for organ donations are not universally effective, with some countries that utilize opt-out schemes (i.e., presumed consent unless one specifically opts out) having the same rates as those with opt-in options (i.e., active consent).[7] Cultural differences in the acceptance of organ donations are likely to be a factor regardless of the opt-in or opt-out defaults. The possibility of nudges failing when trying to facilitate behaviors that may be inconsistent with our sentiments or goals would, in fact, be expected under Kahneman’s dual system decision model that underlies most of behavioral economics. System 1 is fast, habitual, heuristic, and mostly unconscious. System 2 is conscious, slow, and deliberate. Thus, if System 1 leads us down the path of least resistance, it is System 2 that keeps us on the straight and narrow. Encountering a nudge for a behavior that is not consistent with our goals, likes, or aspirations is likely to be noticed by System 2, leading us to invest more effort into the decision regarding behavior. This is exactly what the low-income taxpayers did, or what a person opposed to donating one’s organs on religious or cultural grounds would do. Whereas, some inconsistent nudges may involve the wrath of System 2 by trying to sneak past the monitoring function in one’s subconscious, others may in fact structure the decision space to make information more salient and hence more accessible to System 2. For instance, if we assume that most food decisions are habitual or influenced by subconscious triggers, manipulations of the decision space such as posting calorie information on restaurant menu boards may encourage healthier choices byinterruptingthe mindless ordering of the usual menu items.[8] In other words, seeing a calorie count for a double bacon cheeseburger may evoke the, “OMG, do I really want to spend most of my daily calorie allocation on THAT?!” response. Yet, making the information more salient (and hence invoking the System 2 processing) may not necessarily lead to the desired results. The inconsistent findings regarding providing calorie information in fast food establishments[9] may be explained by the possibility that some customers are not particularly concerned with the calories they consume. Just because the information is there, it does not mean that it will be used in decision making. The same can be said ofreminders, another type of nudge in the behavior change arsenal. For example, “forgetting to get a flu shot” is the most frequently reported reason for not getting a flu shot among individuals with already favorable predispositions toward seasonal flu vaccination.[10] It follows that reminders from physicians or point of sale signs at retail establishments advertising availability of the flu shots are likely to fall on deaf ears among those not favorably predisposed to get one. Nudge contributes to persuasion By no means should this analysis be construed as the outright negation of behavioral economics or the nudge itself. The decision theories underlying the nudge have a lot to contribute to persuasion as well, especially in terms of how information is framed and processed and how these cognitive processes may lead to attitudinal as well as behavior change. In public health, a behavioral economic framework can equally contribute to the understanding of behavior by emphasizing incentives and disincentives for given behaviors beyond the emphasis on health as the individual goal.[11] Elsewhere, it is seen that the decision theory underlying behavioral economics, in fact, offers a better cognitive conceptualization of attitude and behavior change than the value expectancy models that often underlay social marketing approaches.[12] Yet, the claimed effectiveness of the nudge as the clever population-based intervention and as an alternative to more comprehensive marketing efforts is overrated. Surely, nudges work, but only when they are consistent with the attitudes and goals of those being nudged. To paraphrase Kenny Rogers, “we gotta know when to nudge’ em.” Otherwise, we may have to change minds. We invite you to explore the rest of our web site and reach out to explore what might be the best fit for you. Let’s get the conversation started. Bibliography [1] http://swampland.time.com/2013/08/09/nudge-back-in-fashion-at-white-house/ [2]http://www.nytimes.com/2013/12/08/business/international/britains-ministry-of-nudges.html [3] Bovens, L. (2012). Read nudge. European journal of risk regulation, 2012(1), p 43-46. [4] Young, S., & Caisey, V. (2014). Behavioural Economics and Social Marketing: Points of contact? In D.Stewart (Ed.), Handbook of Persuasion and Social Marketing. London: Praeger. [5] Dolan, P., M. Hallsworth, D. Halpern, D. King, R. Metcalfe, and I. Vlaev. “Influencing Behaviour: The Mindspace Way.” Journal of Economic Psychology 33.1 (2012): p 264-77. [6] Bronchetti, E.T., Dee, T.S., Huffman, D.B., Magenheim, E. (2011). When a nudge isn’t enough: Defaults and savings among low-income tax filers. National Bureau of Economic Research, Working Paper 16887,www.nber.org/papers/w16887. [7] http://www.organdonation.nhs.uk/newsroom/statements_and_stances/statements/opt_in_or_out.asp [8] Downs, J. S., Loewenstein, G., & Wisdom, J. (2009). Strategies for Promoting Healthier Food Choices. The American Economic Review, p 159-164. [9] http://www.rwjf.org/content/dam/farm/reports/reports/2013/rwjf406357 [10] Carter, W. B., L, R, Beach, Inui, S., Kirscht, J.P. & Prodzinski, J.C.(1986). Developing and testing a decision model for predicting influenza vaccination compliance. Health Services Research, 20, p 897-932. [11] Cawley, J (2004). An economic framework for understanding physical activity and eating behavior. American Journal of Preventive Medicine, 27, p 117-125. [12] Ulasevich, A. (2015). Behavioral Economics and Social Marketing: Toward Integrated Approaches to Behavior Change. In DW Evans (ed.) Social Marketing: Global Perspectives, Strategies, and Effects on Consumer Behavior, Nova Science Publishers. Updated fromNudging the Choirby Alec Ulasevich, PhD, at EurkeFacts Smart Research Solutions, © 2016.

Abstract This study evaluates the effectiveness of the think-aloud method used during cognitive interviewing for survey-item testing with children. This study examines the application of a think-aloud technique during testing of a computer-based assessment, across two different grades and age-groups: grade 4 (age 9-10) and grade 8 (age 14-15). We compared the effectiveness of the think-aloud method between the two grades by evaluating the verbal reports length, time of response, and the quality and the completeness of the student responses. We found that the think-aloud method does not vary in the quality of the verbal output obtained from children in the two age groups. Although research suggests that young children typically find the think-aloud method to be difficult (Someren et al, 1994), this research shows that there is no difference in the effectiveness of the think-aloud method between students in Grade 4 and in Grade 8. Introduction High-quality survey data often requires laboratory pretesting of the survey questionnaires or educational assessments. During the last two decades, cognitive interviewing became one of the most widely used pretesting methods. Cognitive interviewing is intended to uncover problems with the questions that may compromise the quality of responses before the survey or assessment fielding. One of the procedures used during the cognitive interviews is the think-aloud that requires respondents to verbalize their thoughts as they answer survey or test questions. There is little empirical research on the effectiveness of the think-aloud method utilization during the pretesting of surveys or assessments with children. Previous research has shown mixed results. Some reported that young children and teenagers lacked the ability or motivation to articulate their thoughts, while others showed that the think-aloud works well with teenagers but requires age-relevant instructions and more follow-up probing with younger children (Leeuw, et al. 2004; Someren et al, 1994). Objectives To evaluate effectiveness of the think-aloud procedure and quality of the verbal protocols produced by children in Grade 4 (ages 9-10) and Grade 8 (ages 14-15). To compare the effectiveness of the think-aloud procedure and quality of the verbal protocols between children in Grade 4 (ages 9-10) and Grade 8 (ages 14-15). To compare the ability to recognize and categorize the information from verbal reports into four processes (comprehension, retrieval, judgment, and response) identified the Question Response Model (Tourangeau, 2000) across grades. To evaluate the ability to evaluate question processing at each of four response stages (comprehension, retrieval, judgment, and response) based on the information provided in the student think-aloud. Methodology Ten 4th grade students and ten 8th grade students participated in cognitive interviews. Each student group completed 4 grade-specific science questions. Concurrent think-aloud procedure utilized during the cognitive interview. Students were provided with detailed think-aloud instructions, a demonstration, and a think-aloud practicing session before attempting to answer each question while thinking-aloud. Consistent think-aloud procedure and instructions across students ensured with a detailed script. The following four (4) factors were controlled for between 4th and 8th grade in terms of question type assignment: oSubject: Science NCES Questions administered in 2009 oContent Classifications: Physical Science, Earth and Space Science, Life Science oQuestion Difficulty: Easy, Medium, Hard oQuestion Order: Presentation Demographics A total of 20 students completed the cognitive interview. The sample consisted of a mix of SES, race/ethnicity (Table 1), and gender (Table 2). ontrolled for experience and frequency of use of technology (tablet, phone, and computer) across grade 4 and grade 8 students. Results There was no significant difference in the verbal report’s length (number of words) provided during the student think-aloud across questions between grade 4 and grade 8 students. We found a significant difference in the amount of time the student spent doing a think-aloud across student grades. Overall, grade 8 students took more time on their think-aloud than did grade 4 students. Relevance of the Verbal Reports There was no significant difference in the relevance of the student think-aloud to the topic of the question between the two grades. Both 4th-grade and 8th-grade student’s think-aloud remained equally focused on the topic of the question. Response Process Evaluation We examined whether there are differences in the ability to evaluate question processing at each of four response stages of the question response process (3-point Likert Scale: 1–Unable, 2–Somewhat, 3–Able). Overall, we found no significant differences between grades in the ability to evaluate information at each of the response stages. Results showed differences in the ability to identify and evaluate the judgment process between the two grades, where it was more difficult to assess judgment in 4th graders than in 8th grades. Verbal Reports Completeness We examined whether there are differences in the completeness of verbal reports regarding the inclusion of information illustrating four stages of the question response process (3-point Likert Scale: 1–No, 2–Somewhat, 3–Yes). Results showed no differences between grade 4 and grade 8 students in their completeness of the verbal reports in terms of information relevant to comprehension, retrieval, judgment, and response processes. Analysis Student verbal reports were coded by three coders. The following measures were included to evaluate verbal reports quality and completeness: Verbal report length: number of words Duration of the think-aloud (seconds) Verbal report quality score based on the following criteria: Relevance of verbal reports content to the question tested on a 3-point Likert Scale (1–No, 2–Somewhat, 3–Yes). Completeness of the verbal report content in terms of including information illustrating four stages of the question response on a 3-point Likert Scale (1–No, 2–Somewhat, 3–Yes). Ability to evaluate question processing at each of four response stages: (1–Unable, 2–Somewhat, 3–Able). Findings Results show that the think-aloud method does not vary in the quality of the verbal output obtained from children in the two age groups. The think-aloud method is equally effective in producing verbal reports of sufficient quality and content to evaluate question processing and performance in both Grade 4 and Grade 8 students. Although one might expect to find that 4th graders are more likely to verbalize information that is not related to the topic of the question being tested, results show there is no observable difference in the relevance of the topic of the student think-aloud between the two grades. The verbal reports from both groups did not differ in terms of inclusion of information that pertains to stages of questions response processing: comprehension, retrieval, judgment, and response. However, it may be more difficult to identify and evaluate information related to the Judgement stage in verbal reports from 4th-grade students compared to 8th-grade students, although this stage is, in general, more difficult to evaluate compared to other processing stages Bibliography Conrad, F. G. and Blair, J. (2001, August 5-9). Interpreting verbal reports in cognitive interviews: Probes matter. Proceedings from the Annual Meeting of the American Statistical de Leeuw, E., Borgers, N., and Smits, A. (2004). Pretesting questionnaires for children and adolescents. In Presser, S., Rothgeb, J. M., Couper, M. P., Lessler, J. T., Martin, E. Martin, J., and Singer, E. (Eds) Methods for testing and evaluating survey questionnaires (pp. 409 – 429). Hoboken, NJ: John Wiley and Sons, Inc. Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press. van Someren, M. W., Barnard, Y. F., and Sandberg, J. A. C. (1994). The think aloud method: A practical guide to modelling cognitive processes. London, England: Academic Press. More Information Please send questions or feedback to Director of Cognitive Research Mila Sugovic: sugovicm@eurekafacts.com About EurekaFacts EurekaFacts is a full-service market research and analysis firm that provides information and insight that helps identify and maximize the opportunity for a better impact. Our mission is to equip the people who lead efforts toward building a better, safer, healthier world with the information needed to shape and improve the market strategy and tactics of their organizations and initiatives. In order to deliver insightful, accurate, and on-target information at a reasonable price, our teams of talented professionals use data collection and knowledge-discovery methods and technologies that best meet the demands of each situation and organization. We measure our progress in terms of our role in the increased effectiveness of the efforts, companies, and organizations we serve.