8 signs the data from the Trudeau government’s online electoral reform survey is useless

Can anyone trust the data from the Liberal government’s new online electoral reform survey?

Prime Minister Justin Trudeau seems to think so – last week, Trudeau suggested the “fun little questionnaire” at MyDemocracy.ca will help “drive a solution for electoral reform.”

And Democratic Institutions Minister Maryam Monsef said the online survey will “make sure that the values that (Canadians) feel are most important in a voting system” are “represented in the final outcome,” adding this is something she’ll consider “before we introduce legislation.”

Plus Mark Holland, parliamentary secretary for democratic reform, argued the fun quiz is actually a “scientific study done by professional scientists,” designed with “the integrity of the data” top of mind.

The quiz might have value as a fun educational tool – but can Trudeau’s Liberals seriously use it to justify their decision on a new electoral system?

At the same time, Monsef has been busy trashing an all-party parliamentary committee and her own electoral reform town halls, both of which endorsed proportional representation. Other Liberal MPs trashed those conclusions on the grounds that their own town halls are “not a representative sample of the Canadian population.”

So, how much integrity can the data from this fun little quiz really have? Here are several things to consider:

1. Bad questions

When the online survey first went live this week, many Canadians felt there was something fishy about the survey’s questions: 

Every question in that manipulative survey is either beside the point, or misrepresents the issue, or is framed to elicit a specific answer.

— Andrew Coyne (@acoyne) December 5, 2016

One question framed proportional systems as a choice between wanting parliament to “represent the views of all Canadians” and opening the floodgates to “radical or extreme” parties:

Other questions follow a similar pattern, framing the electoral reform debate as a series of exaggerated and unrealistic choices.

Makes you wonder if this fun little quiz is really an attempt to sabotage proportional representation?

2. The explanation for their bad questions makes no sense

Defending the survey’s questions, Monsef claims these false dilemmas are crucial to understanding the underlying “values” Canadians have about democracy, that’s why they don’t directly ask which system they want.

MyDemocracy.ca is live. PR advocates won’t be happy. Survey doesn’t ask if party’s popular support should be reflected in HoC seats.

— Althia Raj (@althiaraj) December 5, 2016

There was simply no room for such an unimportant question in a 20-question survey that asks about mandatory voting twice. https://t.co/XzzQtOWET5

— Chris Selley (@cselley) December 5, 2016

Except that point is undercut by the fact the survey asks several direct questions about non-electoral system related issues like voting age, mandatory voting and the length of campaigns:

Apparently a simple yes or no is good enough for some issues, but not for others – namely issues where the Liberals have a different preference in mind. 

Not everyone strongly agrees with that approach, though:

Sending this back to @MaryamMonsef to clear up any confusion from the survey. You should too! #cdnpoli #rejectedERQs pic.twitter.com/4AXDHFF9sc

— Melissa Farrance (@MelissaFarrance) December 8, 2016

3. We can’t get a straight story about who came up with the bad questions

Who wrote the questions?

Rayes: Who created the mydemocracy.ca survey questions? Monsef: survey is based on values at heart of democratic system #qp

— CBC Politics (@CBCPolitics) December 8, 2016

Here’s a short list of the wide-range of answers we heard over this last week:

• Asked earlier this week who designed the survey’s questions, Vox Pop Labs CEO Clifton van der Linden (whose company built the survey) would only tell PressProgress that the survey was “reviewed and approved by the government” and “an independent academic advisory panel.”

• Monsef told the House of Commons the survey is “a Canadian made application, formed by Canadian academic experts to engage Canadians.” 

• However, Monsef repeatedly refused to disclose the names of the members of the mysterious expert panel. 

• Vox Pop Labs eventually released the names of the expert panel. The Ottawa Citizen reported one panelist “confirmed that he was involved in the review process, although he noted that he didn’t actually see the final draft of the survey until he took it himself” and added that “they had no role in actually coming up with the questions.”

• The expert panel later released a statement stating they had, in fact, “proposed survey questions” and said “questions written by Vox Pop Labs” too – a detail the company didn’t initially volunteer.

• In the meantime, the government’s own hotline was apparently telling Canadians that an all-party parliamentary committee approved the questions in the survey – a claim that is entirely false.

One thing’s for sure: no one seems eager to take credit for this survey.

4. Polling experts say the data is “unusable”

Here’s how one Canadian pollster put it: 

I’ve seen @Cosmopolitan quizzes that were better designed than #MyDemocracy ‘s attempt at a segmentation. (1/2) #erre #electoralreform

— Mario Canseco (@mario_canseco) December 6, 2016

Sample size can be 10 million for all I care. Bad questionnaire on #MyDemocracy = Unusable data. (2/2) #erre #electoralreform

— Mario Canseco (@mario_canseco) December 6, 2016

5. One expert defending the survey admits the data has no “significance”

In an interview with Maclean’s titled “In defence of the Liberals’ electoral-reform survey,” Carleton University professor Philippe Lagassé argued the survey has value as an “educational tool.”

But if you read past the headline, Lagassé doesn’t actually sound like he thinks anyone should “compile this data and present it as some kind of decision point”:

“If the purpose is at the end of this to produce a report and say, ‘This is what people thought,’ then I agree, you can’t do that … I think everybody would readily attack them if they claimed this was a survey of some kind of significance if you’re able to re-take it over and over again.”

And yet, that is exactly what Trudeau, Monsef and Holland have all suggested they want to do.

6. No one will explain how they stop people from filling-out the survey multiple times

Another problem: the survey doesn’t ask for your name or attempt to verify your identity in any way.

How do they prevent anonymous Internet users from filling-out the survey multiple times? If you check out the FAQ page at MyDemocracy.ca, you’ll see a vague blurb claiming the survey has built-in “controls” that prevent “repeat entries or malicious use” of the survey.

Thing is, there’s no transparency about how these “controls” actually work.

Asked to elaborate on their “controls,” Vox Pop Labs would only tell PressProgress they have a “series of proprietary techniques” that “screen for multiple entries from the same user” – unfortunately, they say they can’t explain what they are or how they work because “disclosure of these techniques would embolden attempts to circumvent them.”

That explanation doesn’t quite square up with other details on MyDemocracy.ca.

The same FAQ page also encourages rural and low-income Canadians who don’t have Internet access to go use the computer at “your MP’s office, a local library, friend or family member’s house” – in other words, MyDemocracy.ca explicitly encourages multiple people to fill-out the quiz using the same computer, something they supposedly screen against.

In an interview with Maclean’s this week, van der Linden said that “when we’ve identified observations that are assumed to be from the same user, we take the first set of responses.”

But asked if the survey is able to tell the difference between one person voting multiple times from home versus multiple persons voting from the same MP’s office, Vox Pop Labs said it wouldn’t be a problem because “in most cases it would be possible to tell the difference.”

The takeaway appears to be that you’re encouraged to fill-out the survey multiple times from the same computer, except they’ll block your responses if you do, and they’ll use techniques that must remain secret in order to discourage you from filling-out the survey multiple times from the same computer.

Sound like we’re going around in circles? Maybe greater transparency would avoid that problem.

7. The survey “excludes” over one million Canadian expat voters

Over one million Canadian voters live and work in other countries, and Monsef says she’s committed to meeting “the needs of highly mobile Canadian citizens who live in today’s increasingly interconnected world.”

However, it turns out those voters are actually blocked from submitting survey responses.

Vox Pop Labs told PressProgress they have “a series of proprietary techniques” that “screen for multiple entries from the same user – both at the point of collection and when screening the data – as well as to identify non-Canadian respondents.”

“It is likely that many Canadian expats will be excluded from our results,” van der Linden told PressProgress. “We do not have a control variable in the survey to identify them as such.”

8. Oh, and they may have excluded your results too

And a million expats might not be the only ones surprised to learn their survey results didn’t count.

Take, for example, anyone who follows the survey’s instructions.

When you get to the final page of the online survey, it asks you to fill out a few general demographic questions about where you live and how much you earn. A blurb on the page reads:

“This portion is completely optional. Even if you choose not to complete the questions in this section, you will still be presented with the full results.”

Well, it turns out that’s not true.

Tucked away in the privacy page of MyDemocracy.ca is a note that says the following:

“While answering the profile questions is optional, not answering these questions will result in your input not being included as part of the overall results of the study.”

For her part, Monsef says she “can assure all Canadians that providing personal information is completely optional” and “can assure all Canadians that we will incorporate all feedback into the final result,” assurances contradicted by what it literally says on her own website. 

What a fun little mess!

^{Photo: PMO.}