The Deceit of the Directed Survey

Surveys, which have lately become a very popular mechanism for collecting data and reflecting the public will, are dangerous vehicles when they are carelessly conceived or inexpertly constructed. Indeed, it is not only lack of the necessary competence that can turn a survey from a useful expression of the popular voice to a deliberately misleading reflection of special interests or private biases. Crafted with questions that lead to predetermined or desired outcomes – “directed” in effect – surveys can misrepresent their ‘universe’ and offer conclusions that are inaccurate.

The authors of the FocusBurlington (FB) 2026 Budget Survey are quite familiar with my feelings on this subject.  Surveys are highly scientific instruments that require considerable expertise and experience. They need to receive data from all demographics and all locales within a particular subject area and they need a large enough sampling of these constituencies to be statistically valid. None of these criteria are satisfied by the FB Budget Survey. Two hundred and twenty two people responded.  This is less than 2/1000ths of 1% of the city’s 2022 voting population (over 147,000). There is no breakdown by age, income, gender, occupation or educational level. The geographic distribution is unvalidated (e.g. I used another postal code, not my own) and the respondents are largely drawn from visitors to the FB website. In other words, as a collective, those who answered the survey probably identify with the priority areas identified by the FB creators; areas that arguably align most closely with conservative or right of centre policies and values. And the survey questions often beg a single, predetermined answer. For example, question #2 which is truly silly. Does anyone actually not want COB to “clearly communicate the tax increase”? Similarly, question #3 poses four scenarios with the obvious choice being the completely unspecified and uncosted  “find efficiencies to maintain services and hold the tax increase to the rate of inflation”. Question #4 asks which “programs” should be continued and lists a selection of nine choices with no rationale for isolating these specific items. What does a fireworks display have in common with Burlington Green funding or membership in the Federation of Canadian Municipalities? Similarly, question #9 identifies four potential “revenue generating opportunities”. Why these specifically and why not broaden the question to deal more generically with categories of possible revenue generation?

I could go on but if I haven’t made my point with this much verbiage then more will not help. The FocusBurlington (FB) 2026 Budget Survey is a flawed instrument and anecdotal at best. Ironically, it suffers many of the same shortcomings as the COB surveys that BRAG used to criticize as being predetermined, leading to desired findings and favourable results. As it stands, it is an interesting tidbit that reveals as much about the authors as it does about the respondents. Is it an authoritative source of data and an example of effective public consultation? Hardly.

Totally Out of FOCUS