Article 1:
Survey Effectiveness Index Construct

Article 3:
Taking Surveys to the Next Level

Article 4:
Customer Loyalty

Article 5:
Employee Benefits

by Robert A. Page, Jr., Ph.D. and Edward W. Tamson, Ph.D.

Contact PDi about these publications

(copyright 2002 by PDi. All rights reserved. Please do not modify, copy or distribute this article -- see website terms and conditions of use.)

Introduction

Surveys have amazing potential. Carefully crafted survey instruments can accurately measure and report on the attitudes and behaviors of large groups of employees and customers more cost effectively than virtually any other method of data collection. The challenge lies in survey construction, for designing effective survey instruments is very tricky. In fact, it is so challenging that despite the best efforts of experienced organizational internal and external change agents, surveys often do not perform as expected. When surveys are not all they are supposed to be, at best some of the data is useless, representing a waste of time, effort and money. At worst, some of the data is inaccurate and potentially misleading. This article shows you how you can trust your survey findings.

Here are the steps involved in a complete survey process. Some steps can be abbreviated or excluded altogether, depending on current circumstances and needs, and what you plan to use the survey data for. Others are vital, and can compromise the validity of the survey and the accuracy of the research findings if they are not handled properly.

Needs Analysis
Sampling Strategy
Survey Design
Survey Pre-testing and Validation
Survey Administration
Data Entry
Survey Analysis
Hypothesis Testing
Survey Feedback and Implications
Survey Post-testing and Validation
Survey Synchronization
Survey Implications

Let's examine each stage in turn, both to understand the challenges it poses and to assess how PDi can help organizations meet those challenges, and take survey efforts to the highest levels of performance.
Think of this survey process as an "organizational photography session". The ultimate goal is to end up with a set of "photographs" which captures the organization (i.e. Employees, Customers etc.) accurately, and in detail, at a particular moment in time. Using this photography metaphor:

Needs Analysis/Goals & Objectives - What is worth taking a picture of?

Sampling Strategy - Who will and will not be included in the pictures?

Survey Design - What type of equipment will be used to take the pictures?

Survey Pre-testing and Validation - Do you need a trial run before the real shoot?

Survey Administration - The photography shoot.

Data Entry - How carefully is the film handled during development?

Survey Analysis - Numbers that describe the most important features of each picture.

Survey Reporting - How can the pictures be presented to maximize their impact?

Hypothesis Testing - Can a composite picture be accurately substituted for a whole set of single pictures? Do the behaviors seen in one picture affect those seen in another?

Survey Feedback and Action Planning - The photograph album, with snappy captions that highlight strengths and identify areas for improvement, that concludes with recommendations and an action planning roadmap for change and transformation.

Survey Post-testing and Validation - Moving up the experience curve by improving the process and upgrading the equipment so the pictures come out better next time.

The challenge is to avoid photographs which are inaccurate or misleading, because they miss important details, or are so fuzzy or distorted they are confusing. Given that organizations are difficult subjects to photograph - complex, mutli-faceted, and often out of focus - taking quality photographs can be problematic. Fortunately with the right training, equipment and assistance, professionals can learn how to take good pictures even when the light is poor and the subject is moving. The same is true with organizational surveys.

Needs Analysis / Goals and Objectives

An initial needs analysis and identification of goals and objectives is a reality check; the more comprehensive the initial analysis, the greater the success of the survey and change efforts in addressing areas that really matter.

Why Bother? The alternative is to arbitrarily select the needs someone thinks are important and assume that everything will work out for the best in the end: "Ready, fire, aim!" Unfortunately, without a comprehensive needs analysis, the survey and change efforts may be missing needs that are critical, may prioritize needs many regard as unimportant, and may focus on interventions that are comfortable and familiar rather than necessary and relevant. In short, without a reality check, survey efforts may end up merely telling us what we wanted to hear, and miss what we needed to hear. Change efforts may end up playing to and focusing on our strengths, rather than helping us improve areas of weakness, and that can become a real problem. In other words, if I am good with a hammer, everything starts looking like a nail. Hopefully, not too much gets smashed in the process.

Technique: Fortunately, qualitative techniques adapted from ethnographic research methods in the field of cultural anthropology offer our clients a variety of proven tools to insure that needs analyses and identification of goals and objectives are not rushed or superficial. Typically they involve in-depth individual interviews or focus groups. For clients who lack the time to conduct a thorough needs analysis, or who desire interviewers trained in qualitative research method to provide a fresh and more objective perspective, PDi can be a valuable resource.

Sampling Strategy

Sampling strategies determine who does and does not get to speak their mind in your research findings. If you want your survey report to be worth anything, every important internal and/or external constituency whose opinions are relevant should have a chance to contribute. If not, they have every right to attack and dismiss your findings as one-sided.

Why Bother? Typically we see organizational surveys that involve samples of convenience - whatever groups of employees happen to be easily accessible at the time. However, convenient samples may not be representative samples - some important, relatively accessible groups are likely over-represented, and other important, relatively inaccessible groups are likely under-represented. This is unfortunate, since different groups in different functions (marketing, operations, engineering, support, etc.), different locations (regions, nations, continents, etc.) and different business units (product lines, brands, etc.) often see things quite differently. This means that survey findings may not be believed by the groups who were not adequately represented, and who do not feel they were listened to. Further, the findings themselves may be misleading, since the sample of convenience favored the perspectives of some groups over others.

Technique: Fortunately, there are rigorous statistical sampling strategies that will insure representative data sets. Stratified random sampling and cluster sampling are the most common, generally accepted sampling strategies. For clients who lack the expertise and time to design and conduct a representative sample themselves, PDi will provide that service.

Survey Design

Survey designs figure out how to take a look at the topics that matter for organizations. Since important topics are usually complex, survey designs decide where and how closely to look at each of the key areas.

Why Bother? This stage is one of the most important and complex stages of the survey process, because survey findings are only as good as the questions they are based on. The quality of the survey design determines the quality of the data, and the accuracy of the research findings. As the adage warns: "Garbage in, garbage out," often applies to poorly designed surveys. Organizations often ask PDi to audit and analyze existing surveys/data collection methods. Over the years we have found that approximately 20% of the surveys we review are fatally flawed, to the point of being a waste of the organization's time resources, 40% have major weak spots with serious validity problems, 30% are just passing basic validity criteria, and the remaining 10% are working well. In short, the average survey effort is only about 50% effective, and PDi can improve that percentage.

Technique: High quality survey design involves matching the topic of interest with the type of survey and rating scales which will measure it most effectively. The survey construct adds, deletes and consolidates categories and sub-categories until the demands of content validity have been satisfied. Individual questions are refined until they are face valid. Transitional and framing paragraphs are added to insure that respondents do not become confused. This design process requires a mix of extensive practical business experience and advanced statistical understanding. Consider the following issues:

A. Survey Strategy. The first task it to decide which type of survey best matches the content being assessed. Do we design 360 degree profile feedback instrument? Should we conduct an attitudinal assessment? Administer a behavioral assessment? Implement a multi-stakeholder assessment? Should we use objective or subjective performance indicators?

Another set of decisions involves the number and sequence of rating scales, and the introduction to, and transition between, different sections of the survey, particularly when rating scales will be changed. The challenge is to insure respondents have an appropriate perceptual set, or frame of reference, when answering each section of the survey.

B. Content Validity. Does the survey construct do justice to the concepts being investigated? Given what is known in organizational behavior and development about these concepts, are all important aspects being assessed, so the research effort will be comprehensive? Are categories or questions being included that are not properly associated with the concepts of interest? When the survey is including irrelevant topics, or excluding relevant ones, the data set becomes either superficial or overly selective, and the research findings potentially misleading. Careful survey construct design avoids many of these content validity problems.

C. Face Validity. One aspect of face validity involves the language used in the development of survey questions. Is the phrasing clear, unambiguous, unbiased, and easily understood, or has it fallen into common wording pitfalls, such as double meaning, complex or verbose terminology, objectionable or inflammatory language, overestimation of understanding, or demand bias? Inappropriate phrasing introduces measurement error, which means that you can not be sure what the ratings are telling you - does it represent honest opinions about the issue, does it represent subjective reactions to the poor phrasing, or some combination of both? You usually need another survey to find out.

Another aspect of face validity involves the match between the questions and the rating scales. What type of rating scales should be used with which questions: ordinal, nominal, or interval (continuous) scales? How long should the scale be and should interval scales include a neutral midpoint and/or a non-response option? Are the value labels on the interval scales symmetrical? Is the conceptual distance between rating points on an interval scale consistent? Does the wording of each question match the scale that is supposed to assess it? Are transitions between different rating scales appropriate, or are respondents likely to become confused? Once again, when rating scales are inconsistent, confusing or inappropriate, accurately interpreting their ratings becomes problematic.

D. Expert Reviewers. Establishing survey face and content validity is a function of insight, education and expertise. In general, the more people involved, the better. PDi provides well educated, experienced reviewers to assist in analyzing survey content and phrasing. PDi also offers the ultimate in rigorous review - the expert panel, where academics, researchers, industry experts and practitioners all review and refine the survey. Lastly, PDi has developed an extensive Library of Survey Questions that has already been validated in a number of our organizational surveys.

Survey Pretesting and Validation

Sometimes, particularly when the stakes are high, you want to make sure your survey is a lean, mean assessment machine before using it in your organization. Think of the pre-test/pilot as a test drive, where you make sure that everything is performing as expected, and make appropriate adjustments, before you have to do it for real.

Why Bother? Given the additional time of pre-testing/ pilot, many organizations choose to skip over this step and go directly to the general administration. If the survey has been carefully designed, more than half of the questions and categories will work fairly well. The others will have unexpected problems, and the question becomes "Can you live without them?" This may be a viable option when the results are being used for developmental efforts only, or when the survey can be refined in future administrations. However, when the survey findings will be used for the following purposes, you skip pre-testing/pilot at your own peril:

To establish benchmarks for comparison with future administrations
To provide information for compensation or promotion purposes
To help plan or implement major change initiatives or new strategies

Further, pre-testing/pilot provides a statistical basis to shorten long survey instruments without sacrificing their explanatory power.

Technique: Pre-testing/pilot involves administering the survey to a small, but representative, sample of the organization, and statistically analyzing the results. The data are analyzed for:

Highly inflated or depressed mean scores (skewness statistics)
Restricted variance (kurtosis statistics)
Flat or bimodal distributions (kurtosis statistics)
Reliability of categories and sub-categories (Cronbach Alpha coefficients)
Discriminant validity of categories and sub-categories (Exploratory Factor Analysis)
Convergent validity of categories and sub-categories (Pearson Correlation Matrices)

These statistical analyses identify a number of potential problem areas. When ratings are so consistent that you can guess the answer to the question before it is asked, the question can be reworded and improved. When categories and sub-categories are giving similar ratings, they can be reworded or consolidated. When questions are loading on multiple categories, they can be reworded or eliminated. On the basis of these results, the survey usually can be both improved and shortened considerably.

Survey Administration and Data Entry

Survey administration and data entry concerns the web site posting or printing of the surveys, e-mailing/distributing the surveys to the respondents, monitoring the response rate, and processing the data that is returned.

Why Bother? If the survey is not web site posted, printed, distributed and administered in a professional manner, response rates are negatively affected. The lower response rate, the greater the chance that survey findings will be invalided by selection bias - some groups being over-represented, and others under-represented, in the data set. Further, if the survey is not processed professionally, the data will be entered incorrectly, and can not be trusted.

Technique: PDi offers surveys in a variety of media, ranging from the old standard of paper and pencil surveys to state-of-the-art web based surveys. Survey distribution and collection through PDi, an independent, external firm also lends credibility to any confidentiality agreements made with the survey respondents. Responses rates can be tracked, and a variety of interventions are available to boost inadequate survey participation to satisfactory levels. All data is carefully and repeatedly checked to insure high quality and accuracy.

Survey Analysis and Reporting

Survey data analysis includes all of the standard descriptive statistics the discriminating organizations would desire including reports for each category, sub-category, hidden category and question. When qualitative data (from interviews, written comments, etc.) are available, examples and stories can be integrated with the statistics for an in-depth picture of important survey findings. The PDi Executive Summary Report and on-site presentations are also provided to concisely communicate important survey results and improvement recommendations.

Why Bother? Survey results may not even be noticed unless they are effectively presented in a brief executive summary format, enticing managers and executives (who normally lack both time and interest) to investigate important findings further. In many reports, your findings, however interesting and intriguing, can often lose both their impact and understandability if they are not presented in the user-friendly, visually accessible format PDi provides - your audience will just tune out. The PDi Executive Summary Report also contains the qualitative data (Sample of written comments) that provides the stories and examples which make the dry numbers come alive, and appear more important and relevant to the employees or customers you are trying to reach.

Technique: A variety of survey software products (including CorporatePulse, SPSS, Microsoft Excel, and others) provide a professional, visual representation of mean scores in a variety of formats, including horizontal and vertical bar charts, pie charts, and line graphs. Content analysis of verbatim responses or interview is also available, when qualitative techniques such as open-ended questions are used to solicit even more in-depth information on important topics. Ideally, qualitative data provide the examples needed to illustrate each of the important findings from the survey data, building convergent validity.

Hypothesis Testing

Once you have the numbers, the next question is "So What?" Hypothesis testing helps you understand what the numbers mean, and even allows you to actually test your pet theories about how things work in your organization.

Why Bother? Descriptive statistics give you a lot of information, but leave you to make the inferences yourself. How big does a difference between scores have to be before you can say it is real/significant? Are your categories and subcategories related to each other in meaningful ways (when ratings in one category increase, do the ratings of other categories also tend to increase or decrease)? And what do you say when someone challenges your conclusions, saying that those differences could be the result of random error, coincidence, or chance? It is hard to hold people accountable for survey results when you are not sure what you can legitimately infer from those results. Obviously, advanced analysis that is both understandable and insightful is needed.

Technique: Higher level inferential statistical analyses can help, such as:

Analysis of Variance (ANOVA): Analyze the differences between the ratings of different groups, and how significant they are.
Pearson Correlation Coefficient Matrices and Multiple Linear Regression: Analyze which variables are significantly related to a key (dependent) variable, how they are related, how much that relationship explains, and how much explanatory power each variable adds. This will tell you which categories and sub-categories tend to have real impact on the issues you feel are most important.
Time Series Analyses: Test if improvements or declines in ratings over time are big enough to make inferences from, with confidence.

Survey Feedback and Action Planning

Clients who would like the survey expert on hand to assist in the executive debrief, facilitate planning change initiatives, and answer any questions have one on demand from PDi.

Why Bother? One of the most challenging aspects of survey research is helping clients understand the significance of the findings, and make use of them in an appropriate and positive way. PDi can assist in drawing implications from survey data and from other information sources, and in planning and implementing change efforts. In short, with the help of PDi, when someone asks, "Where's the beef?" you will be able to give them all the meat they can handle.

Technique: The first prerequisite for a successful survey results presentation is to have the PDi consultant to both present results and facilitate positive communications in the survey debriefing meeting. Feedback of any kind is always somewhat threatening, and survey results tend to uncover all sorts of sensitive issues for discussion, when the surveys are done well. The PDi consultant takes responsibility for establishing and maintaining an atmosphere of open communication, understanding and learning, rather than blaming.

The second prerequisite for success is to have the PDi consultant there who understands the survey process and methodology, and can answer any questions or challenges that may come up. Keep in mind that the easiest way to ignore uncomfortable results is to attack the credibility of the survey process, so those results can be safely dismissed without further consideration. Having a PDi survey expert handy helps keep the dialogue focused where it belongs -- on survey results and recommendations for improvement.

Finally, having an action planning methodology to help structure the results and the planning process often proves to be invaluable. PDi has the action planning materials developed for individuals and groups which can be customized to meet the specific needs of most organizations.

In short, PDi consultants play whatever role the organization needs. Some organizations with gifted human resource specialists with these skills need only basic coaching and feedback. Others have internal change agents who lack the time and resources to comfortably do this themselves, and want PDi consultants to take a more active role. Some wish to farm out the task entirely, and allow PDi to prepare and deliver debriefing sessions, and develop action planning materials customized to both the specific needs of the client and the nature of the survey findings.

Survey Post-testing and Validation

Sometimes, particularly when the stakes are high, you want to make sure your survey is "a lean, mean assessment machine" before using it again. Post-testing takes your survey to the next level, correcting problem areas and filling in any unexpected holes.

Why Bother? Given the additional time and expense post testing requires, many organizations choose to skip over this step. This may be a viable option when the results are being used for developmental efforts only, or when the survey will not be administered in the future. However, when the findings will be used for the following purposes, you skip post-testing at your own peril:

To establish benchmarks for comparison with future administrations
To provide information for compensation or promotion purposes
To help plan or implement major change initiatives or new strategies

Further, post-testing provides a statistical basis to shorten long survey instruments without sacrificing their explanatory power.

Technique: Post-testing involves statistically analyzing the results of the general administration to identify a number of potential problem questions or categories, which can be reworded, consolidated, or eliminated. Problems include questions, subcategories or categories which feature: (a) ratings so predictably and consistently high or low that you can guess the answer before the question is even asked; (b) ratings so similar to those of other questions or categories, they are virtually identical, making them redundant and superfluous; (c) ratings with identity problems, which load on multiple categories in factor analysis. On the basis of these results, the survey usually can be both improved and shortened considerably.

When survey findings are used to make important managerial decisions, which can range from major strategic initiatives to individual compensation or promotion opportunities, further tests are often necessary to establish predictive (criterion) validity. This involves comparing the survey data to a set of independent performance indicators, to insure that there is a significant relationship between the category scores and the actual behaviors the company wants to pursue or to reward. Once such a relationship is proven, decision makers can rest assured that survey findings represent important and relevant information on the issues being measured - information they can have confidence in. Further, with such analysis there is no legal basis for challenging the survey results.

Survey Synchronization

Given the number of improvement efforts going on in the typical organization, why not coordinate them? Better collaboration would reduce duplication of effort, and increase the explanatory power of the data. For example, when employee and customer satisfaction survey efforts are linked, employee behaviors which drive customer satisfaction can be identified.

Why Bother? Many organizations seem to be swimming in surveys. Dedicated managers and internal change agents are diligently seeking out information to the pressing issues of the day, and employees are filling out surveys again, and again, and again. Or perhaps not at all - some organizations have become so sick of surveys they either avoid them like the plague or the employees simply refuse to respond to them (the surveys end up in the "circular file"). Even if respondents are willing to give all those surveys the good old college try, respondent fatigue is, in and of itself, a major cause of measurement error (the survey equivalent of cancer). There has to be a way to increase the information and explanatory power gleaned from all this data while reducing the effort and inaccuracy involved in collecting it, right? As a matter of fact, there is.

Technique: A variety of strategies can link customer and employee data together. These range from simple correlational analyses of independent data sets over time, to multi-method analysis which relates a variety of types of data on both employee and customer satisfaction together. The gold standard here is predictive validity - to know what employee behaviors drive customer satisfaction, and to have those drivers linked to objective indicators rather than subjective perceptions. If your MIS system is up to the challenge, PDi can get you there.

Survey Implications

The scariest thing about surveys is that there is no such thing as the "perfect survey." At PDi we constantly strive for the "perfect survey"; the best you can hope for is to minimize measurement error, not to eliminate it. Why? Because survey construction and interpretation is as much of a balancing act as it is a science. Consider the following dilemmas:

How do you interpret survey findings accurately, without undue optimism or pessimism?
Is your survey construct too comprehensive or too superficial?
When should you worry about survey findings being legitimately challenged?
If your survey is too short, you sacrifice vital areas of content, but if your survey is too long, people won't complete it, so what is the right length?
Will your survey questions stand up to criticism?
Are the findings actionable, so you know where to improve?

Enquiring minds want to know.

So how do you insure your survey is an effective, high quality instrument, and that you are interpreting survey findings correctly? You could involve a team of insightful colleagues, but chances are their experience in constructing, validating and interpreting surveys is limited. You could contract with your favorite consultant, but relatively few consulting firms have the resources to keep experienced survey experts on staff, so your survey is likely to be reviewed by someone whose sole survey expertise consists of occasionally reviewing surveys for the past couple of years. You could contract with your favorite university professors, if you could afford their consulting fees, and have your survey reviewed by a graduate student who needs the practice (under the supervision of the professor, of course. Don't you feel reassured?).

If you are not comfortable with the gaps and blind spots that are likely in such assessments, there is an alternative. You could contract with PDi, a firm devoted specifically to survey design and validation, run by business consultants who specialize in surveys, who have extensive experience designing a wide variety of surveys, who has the statistical expertise to validate surveys, and who have the academic background to incorporate relevant research and theory from the field. This is the mission of PDi, directed by the founders Robert A. Page Ph.D., and Edward W. Tamson Ph.D., business consultants who combine research insight and rigor with down home practical experience and business savvy.

In conclusion, clearly the process of survey research is time consuming, complex and difficult to do well. For those organizations who would like to improve the efficiency and effectiveness of their survey process, PDi offers a variety of cost effective services and resources which can help. Let PDi help you transform your survey and measurement processes into a "lean, mean assessment machine" that is guaranteed to take great pictures, suitable for framing.

Contact PDi about these publications

(copyright 2002 by PDi. All rights reserved. Please do not modify, copy or distribute this article -- see website terms and conditions of use.)