Start with “how do I know it’s working?”, not “what can I count?”
I participated in an email thread there recently that began with this question:
“How would you measure engagement on public issues via interaction in online spaces?”
It led to a lively exchange, but it left me unsatisfied. “Measure” and “metrics” create a kind of tunnel vision, focusing attention on what’s easy to count on the web (hits, number of posts per day, number of posters, pageviews, unique visitors), and away from our understanding and experience of online forums.
As it happens, Steve Clift, the founder of e-democracy, recently reported the results of a grant to create four online forums centered on a number of towns in rural Minnesota.
The report discussed a number of ways that forum discussions had affected their communities:
- A discussion about new regulations regarding the handling of household wastewater led the county’s director of planning to reconsider regulatory language.
- Discussions in a second forum generated stories in the local newspaper.
- Participants used a third forum to get advice on how to fight the city’s withdrawal of their permit to raise chickens in their backyard.
- Participants in another e-democracy forum, not covered in this report, used it to organize their response, including meetings with city officials, to a mugging near a transit stop. The transit agency’s community outreach staffer joined the forum, and then the discussion, based on their actions. The president of the local neighbors group participated as well.
- The report also noted that local government websites had linked to some of the forums and, in one case, the local government had sponsored the start-up of the forum.
These stories suggest a variety of measures that could be applied to e-democracy forums:
- How many local government officials are forum members? What percentage of all local government officials are members?
- How many of these post, and how often do they post? How many of these posts reflect concrete changes in behavior (meetings scheduled, agenda items added or changed for official meetings, changes to legislation or regulation)?
- How many discussions have been used to organize meetings in the community or with government officials?
- How many discussions have received links from local or regional newspaper websites?
These measures all need development, and we could likely find booby traps in each. But consider the conclusion of two (hypothetical) reports on community impact of a (hypothetical) forum in Smallville:
Based on web metrics
The forum received 500 pageviews from 200 unique visitors per month.
It had a membership of 128 at the end of the year.
The average length of a visit was 3.5 minutes.
The average visit includes 4 pageviews.
Based on “grow your own” measures drawn from these stories
Two of the five members of the city council became members of the Issues Forum. One joined after a discussion of the city’s response to last winter’s monstrous snowfalls erupted in the forum and led to a delegation of forum members testifying before the council about ineffective snow plowing.
One councilperson posts at least once a week. Three times over the course of the past year, she has responded to questions in the forum or asked for further information. She also introduced an amendment to a local zoning ordinance based on concerns raised in the forum.
A dozen forum members from Smallville South used the forum to organize a meeting with the transportation department to discuss the pothole problem on local streets. They are using the forum to follow-up on the meeting, and an official from the transportation department posts updates on actions taken relating to potholes at least once a month.
The Smallville Gazette has used the forum to solicit feedback on its coverage of the city council.
On average, one out of four of its weekly online issues include a link to one or more Forum discussions.
Which would be more likely to persuade you that Smallville’s online forum had actively engaged the community?