Hoping to track how Alaskans use and perceive marijuana, the state health department has begun taking annual surveys -- but with the first results published this month, some professionals are raising a red flag about the study's findings.
Alaska legalized the recreational use of marijuana in November 2014. With the shifting landscape, marijuana use and perceptions will likely change, the State of Alaska's section of epidemiology writes.
The survey, released Feb. 1, is the starting point for tracking those changes. But is the data reliable?
"There are just so many points that we could pick out and say 'this is a problem,' that I don't know where to start ... As it stands, I would put very little credence on the survey," said retired statistics professor Don Stevens, who worked at Oregon State University and the University of Alaska Anchorage, and who now lives in Wasilla.
The survey asked 750 Alaskans a series of questions about marijuana use, Alaska's marijuana laws and about their beliefs on the risks of using cannabis.
Among the findings:
• 22 percent of respondents identified as current marijuana users. Those users overwhelmingly chose to smoke the plant, and 60 percent had consumed within the last 30 days.
• Knowledge of Alaska's marijuana laws varied: More than 90 percent of respondents knew a person could be arrested for driving high; only 20 percent knew Alaskans could legally grow up to six plants for personal use.
• Seventy percent of respondents felt that daily use by teenagers was very harmful, while 37 percent felt the same way about daily use by adults. A majority of people, 56 percent, were "highly concerned" about young children eating edibles on accident; only 32 percent had the same worries about adults and adolescents.
However, most of the people surveyed were older, white Alaskans. About 65 percent were older than 50 and 83 percent were white. Seventy-six percent of those surveyed had an annual household income of $50,000 or more.
"That in itself would be enough to raise a huge red flag," Stevens said.
The state agrees that the demographics aren't ideal. "They don't seem to represent Alaska as a whole," said Yuri Springer, an epidemic intelligence service officer for the U.S. Centers for Disease Control and Prevention who wrote the bulletin, but did not help to create the survey.
Matt Berman, a professor of economics at the University of Anchorage Alaska's Institute of Social and Economic Research, agreed.
To have a representative sample, "According to census data … only 39 percent should have been over 50," he said. "Also, 83 percent white is quite a bit higher than the state population, which is about 64 percent non-Hispanic white."
Does the over-representation of older Caucasians skew the results?
"I don't know, you'd have to do another survey," Springer said.
Apples to Apples
Hays Research Group conducted the survey over the course of four days in early May 2015.
The study cost $30,000, pulled from the division's federal funding, according to Springer. Next year, the Alaska Mental Health Trust Authority will fund the study.
The resulting demographics aren't surprising, said research director Adam Hays.
"Sometimes you have to do one survey to find out what variables to account for in the next one," he said. "Ideally we'd have a little better scope for the demographics."
Still, "I think it is relevant," Hays said of the data.
One reason for the demographic leanings may be the method of calling -- 80 percent of phone calls were to landlines. Only 20 percent were made to cellphones, a ratio chosen by the state and based on another national phone survey, Springer wrote.
Hays said he would "advise (the state) to increase the number of cellphones," to get responses from a younger and more diverse group, although he noted cellphones are more expensive to call.
After the interview, Hays called back to say "for that survey we basically followed the directions to the tee ... (The state) specified what they wanted and we did it."
This survey will be used to compare behavior from year to year. So can the state make an accurate comparison with this starting point?
"It should be highly accurate if you survey the same population," Springer said.
Hays said that despite the demographics, the responses can be weighted to give an accurate picture of behavior going forward: "You can compare apples to apples," he said.
Weighting is a tool used in market research to make up for demographic discrepancies, which gives individual responses more or less "weight" to align with the actual known makeup of a population. "That's not uncommon at all," Hays said.
Yet weighting may not fix the problem, according to both Berman and Stevens, as it can bring more errors into play.
'Statistically reliable'?
If there is a low response rate -- meaning the number of people who took the survey were a small percentage of people they tried to contact -- the survey may not paint an accurate picture of Alaskans' beliefs around marijuana, Berman wrote.
Hays calculated the response rate at 34 percent. Both Stevens and Berman, using a different calculation, arrived at just over 5 percent.
There are different ways to factor that rate, according to Hays, but Stevens said he'd "never seen it done (Hays') way before."
"They have misrepresented their response rate," Stevens said.
Berman believes that going forward, it will be difficult to compare this year's survey to future studies.
"I do not think (the) results ... could be considered statistically reliable," Berman wrote.
UAA statistics professor, Rieken Venema, disagreed. "Numerous studies have found that the response rate in and of itself is not a good measure of survey quality," he wrote in an email. Most biases can be corrected with weighting, he wrote.
For the state's part, Springer said that, "I really don't see that that's a huge issue … in terms of comparability" to next year's study. He noted a 2012 study by Pew Research Center found 9 percent response rates are common and can still provide accurate data.
It's also difficult to generalize whether weighting and response rates will be problematic, Springer said, as "it really is dependent on the context."
For next year, Springer said the state will "apply lessons learned" to modify the survey. The state will add new questions to distinguish between medical and recreational marijuana, he wrote.
In terms of demographics, "We'll work to address those issues in subsequent surveys," Springer added.
Stevens said that "there is some potential for salvaging (the survey), but there are certainly things I think they need to look closely at."