Grow your own metrics for online engagement

Posted: May 3rd, 2010 | Author: | Filed under: Civic engagement, Metrics | 3 Comments »

Start with “how do I know it’s working?”, not “what can I count?” has 16 years of experience in creating and hosting online civic forums via email and the web.

I participated in an email thread there recently that began with this question:
“How would you measure engagement on public issues via interaction in online spaces?”

It led to a lively exchange, but it left me unsatisfied. “Measure” and “metrics” create a kind of tunnel vision, focusing attention on what’s easy to count on the web (hits, number of posts per day, number of posters, pageviews, unique visitors), and away from our understanding and experience of online forums.

As it happens, Steve Clift, the founder of e-democracy, recently reported the results of a grant to create four online forums centered on a number of towns in rural Minnesota.

The report discussed a number of ways that forum discussions had affected their communities:

  • A discussion about new regulations regarding the handling of household wastewater led the county’s director of planning to reconsider regulatory language.
  • Discussions in a second forum generated stories in the local newspaper.
  • Participants used a third forum to get advice on how to fight the city’s withdrawal of their permit to raise chickens in their backyard.
  • Participants in another e-democracy forum, not covered in this report, used it to organize their response, including meetings with city officials, to a mugging near a transit stop. The transit agency’s community outreach staffer joined the forum, and then the discussion, based on their actions. The president of the local neighbors group participated as well.
  • The report also noted that local government websites had linked to some of the forums and, in one case, the local government had sponsored the start-up of the forum.

These stories suggest a variety of measures that could be applied to e-democracy forums:

  1. How many local government officials are forum members? What percentage of all local government officials are members?
  2. How many of these post, and how often do they post? How many of these posts reflect concrete changes in behavior (meetings scheduled, agenda items added or changed for official meetings, changes to legislation or regulation)?
  3. How many discussions have been used to organize meetings in the community or with government officials?
  4. How many discussions have received links from local or regional newspaper websites?

These measures all need development, and we could likely find booby traps in each. But consider the conclusion of two (hypothetical) reports on community impact of a (hypothetical) forum in Smallville:

Based on web metrics
The forum received 500 pageviews from 200 unique visitors per month.
It had a membership of 128 at the end of the year.
The average length of a visit was 3.5 minutes.
The average visit includes 4 pageviews.

Based on “grow your own” measures drawn from these stories
Two of the five members of the city council became members of the Issues Forum. One joined after a discussion of the city’s response to last winter’s monstrous snowfalls erupted in the forum and led to a delegation of forum members testifying before the council about ineffective snow plowing.

One councilperson posts at least once a week. Three times over the course of the past year, she has responded to questions in the forum or asked for further information. She also introduced an amendment to a local zoning ordinance based on concerns raised in the forum.

A dozen forum members from Smallville South used the forum to organize a meeting with the transportation department to discuss the pothole problem on local streets. They are using the forum to follow-up on the meeting, and an official from the transportation department posts updates on actions taken relating to potholes at least once a month.

The Smallville Gazette has used the forum to solicit feedback on its coverage of the city council.

On average, one out of four of its weekly online issues include a link to one or more Forum discussions.

Which would be more likely to persuade you that Smallville’s online forum had actively engaged the community?

3 Comments on “Grow your own metrics for online engagement”

  1. 1 Tim Bonnemann said at 11:01 am on May 3rd, 2010:

    Good post.

    I agree you need to start knowing your objectives and then align your metrics accordingly.

    While it’s true that quantitative data is often not ultimately sufficient I think there’s a lot more that can be done by just slicing and dicing some of the basics you mention above:

    A total membership of 128 may not be very insightful. But how many of these are active participants (e.g. have logged in and/or commented at least once over the past 30 days)?

    How many are starting new topics? How many are just replying? What’s the comment distribution curve across all participants and how does it develop over time?


  2. 2 Chris Berendes said at 11:31 am on May 3rd, 2010:

    Tim – Thanks for the comment.

    I agree that, once you’ve identified your objectives, digging into the numbers can be very productive. For instance, Steve Clift has suggested that it’s useful to aim for having at least 10% of the households in the community join the particular forum. It’d be interesting to see whether forums that meet that threshold generally do better on the “story-based” measures I suggested. If so, then Steve’s “gut feel” would be validated.

  3. 3 Rob Goodspeed said at 4:12 pm on May 15th, 2010:

    Chris, I agree with you the best approach to take is to look at objectives/goals, and then craft measurements. I (finally) jumped into the original thread – I’d love to hear your thoughts:


Leave a Reply