Problems with Telephone Surveys

• As the public continue to opt out or become unreachable, conventional techniques e.g. telephone and at the door interviewing become even less reliable.
• Even before addressing issues of not at homes and refusals, 15% of households nationally and 30% in Sydney/ Melbourne now have unlisted telephone numbers. The figures are even more dramatic in the USA where close to 10% of people now have only a cell phone contact number.
• The residential telephone directory is 12 months out of date by the time of its release i.e. around 10% of numbers will have changed. These are more likely to reflect the more transient social stratum.
• Response rates have also been found to vary dramatically between big cities, other capitals, medium sized cities and country towns. Even between postcodes within a city.
• Aggressive Telemarketing activities have also made people less willing to cooperate with survey companies.
• Privacy regulations are becoming ever more stringent for all forms of research due to aggressive telemarketing, spam and other techniques.
• One of the reasons for declining response rates has been failure to follow up effectively not at homes.¹ Wiseman and McDonald in their 1974 paper showed that when up to four call-backs were made, responses more than doubled. Data regularly skews away from those not often at home, shift workers and persons who are not present in the home between 6pm and 9pm in the evening.
• Bednall and Shaw noted in a paper reported on the MRSA website in February 2005:
  
  “In a typical ad hoc telephone survey, half of the calls go unanswered and refusals outnumber interviews by up to six to one. Poor responses lead to an increase in research costs, threaten sample representativeness and result in poor decision making.”

The researchers acknowledged that three call backs were built into the design of many projects but surprisingly in some cases there was none. According to MRSA website Jan 28, 2005, most Australian market research agencies are not doing anything like this. (From paper by Dr. David Bednall presented at the 2004 annual MRSA conference).

• Recently academic researchers in the US have argued that unless 7-10 call backs are made across seven days, a large proportion of a sample may not be reached.
• 1. According to MRSA website Jan 28, 2005, most Australian market research agencies are not doing anything like this. (From paper by Dr. David Bednall presented at the 2004 annual MRSA conference).

Problems with At the Door Surveys

• In addition to increasing levels of refusal, security issues in the modern city have also become a problem for at the door interviewers as well as respondents. At the door interviewers also face access problems in major cities with increasing numbers of security buildings (houses and apartments) and gated communities.
• Cost today for at the door surveys is almost prohibitive when an extensive call back plan is enacted. As a result, call backs are curtailed and interviewing is carried out according to cluster sampling techniques where fewer and fewer clusters of larger numbers of respondents are being sampled again due to cost and time constraints.
• Successful interviews achieved at contacted households may be as low as 10% particularly in areas of major cities where there are large numbers of security apartments or gated households and communities. Researchers who use cluster sampling with limited starting points may require up to 100 doorbell rings to achieve 10 interviews.
• This amounts to little more than quota sampling and the effects of this can be seen in the failure of even some of the most respected research firms to accurately predict election results both in Australia and in the USA.
• Cluster samples are also notoriously unreliable when geographic factors such as TV or radio reception or product distribution are influenced by location. For example, a store may have 100% of the business in one region, 50% in another and no business in a third region. Its average share is 50% but if in the cluster sampling, the 100% share region was omitted by chance, then the market share of the store would be shown as 17%.
  (Refer Bednall and Shaw as reported on MRSA website February 2005: “Poor response rates lead to an increase in research costs, threaten sample representativeness and result in poor decision making”)

Problems With Online Panels

• Opt-in databases are currently confined to Internet surveys which are often considered unreliable despite 70%+ of adults having access to the internet.
• This is because samples are usually drawn from email marketing lists. These are based on people who want to be on lists to get rewards or incentives. These people are not representative of the population as a whole. David Pring (Research World, Jan 2005) makes the point that actual numbers of respondents to most online surveys in the US represent probably no more than 2% of the population, citing examples of how the same respondents are doing most of the surveys (See results of COMSCORE survey reported in Research World, January 2005)
• The Professional Standards Committee appears to be coming to the view that a reliable online panel can only be recruited “offline”.

However, if a reliable online panel can be recruited offline then a reliable phone at the door and mail panel could also be recruited offline.