14 May

Assessing participation in an online learning community

A couple of days ago, I spoke at a PD evening for high school teachers on using technology, specifically about how I use social technologies for connected learning. There was some interest in how I assess students’ contributions to the learning community and I was asked if I would be willing to share the criteria I use for this.

I think it’s important to provide some context for what I do and why I do it this way, so I want to share a bit of background, too.

How I came to assess students’ contributions in online learning communities

In my first year of teaching, I had a real bee in my bonnet about online students’ engagement – with the learning materials, with me, and with their peers. So when I had the opportunity to design a new unit that year, I built the assessment around engagement, in addition to the learning outcomes. It helped that in this particular unit, students were learning about personal learning networks. This meant I had both pedagogical and content-driven motivation to get them to talk to each other, to me and to professionals in industry.

I had a hunch that I would need to assess student contributions in the online learning community in order to successfully foster engagement. And as it turned out, that year, I learned that by assessing student contributions in an online learning community, I could draw them into participating in a meaningful way. Some colleagues and I did a bit of a project about online peer engagement and interviewed students about their experience in the learning community in this unit. We also interviewed students from an accounting unit and an education unit about their online peer engagement. We looked at three different online peer engagement activities across the three units: in the accounting unit, we looked at the use of a discussion forum; in the education unit, we looked at Collaborate-based online tutorials; and in my unit, we looked at the whole approach to the unit, which was essentially what we might now call connected learning (but we didn’t have a name for it at that point – or at least, I didn’t have a name for it). In a nutshell, I ran a WordPress Multisite installation where the students each had their own blog, and used BuddyPress to turn the site into a social network. This approach was inspired by Michael Stephens’ and Kyle Jones’ work with WordPress and BuddyPress with a course at San Jose State University. I also assessed students’ participation in the learning community. Their participation involved commenting on my and their peers’ blog posts, having discussions on the site using the social networking functionality, and engaging on Twitter through conversation and using the class hashtag to share interesting material related to the unit.

In the interviews, my students reported they started contributing to the learning community because they were required to for assessment, but they kept on contributing because they found it valuable. Assessment drew them in, but they kept contributing because it was beneficial. And that’s what it’s all about, right? Creating experiences that are valuable to students, not just experiences they have to engage in to satisfy the course requirements.

But how do you assess contribution to the learning community?

The answer is: it depends on what you’re looking for.

I was looking (and continue to look) for meaningful engagement with the content and robust discussion and conversation. I wanted students to form a personal learning network, both within the class, and beyond. I didn’t just want them to connect with each other, but also with industry. I wanted them to feel empowered to learn by playing with technology and critically reflecting on their practice. I wanted them to support each other, to demonstrate leadership, to role model playfulness and personal investment in ongoing continuing professional development.

In that first year, I developed assessment criteria for participation in the learning community that were fairly basic. I also negotiated the assessment criteria with students (which I do in most of my units), so they had the opportunity to tell me what they thought participation should look like. The next year, after conversations with a colleague who does something similar, I added in criteria about demonstrating leadership. And the criteria have continued to evolve each year.

So what exactly do I assess?

When I grade students on participation, I’m looking at

  • analysis and critical discussion
  • leadership
  • extent of contribution.

I don’t know about you, but I really dislike developing criteria sheets (aka rubrics). I dislike it because I invariably come to use them and find flaws that make it really difficult to mark against them. I also find it tricky sometimes to find the right words to convey what I’m looking for. So I really labour over these things and at the end of each semester, I make notes about what worked and didn’t work, and I iterate them from semester to semester. I also trawl the web and academic databases to look for rubrics or criteria sheets that might serve as inspiration.

At my institution, we use criterion referenced assessment. My personal approach is to write really detailed criteria sheets so that I am completely transparent about what I’m looking for. I spend time explaining the criteria to students (although I aim to write criteria that don’t need explanation), and then I give them opportunity to think about the criteria and negotiate them with me. (As an aside, I find negotiating criteria a really powerful tool to help students take charge of their learning.)

Criteria sheet

Download my criteria sheet for assessing student contribution in online learning communities [Word]. It is by no means perfect, but you might find some of the words useful.

Over the years, I’ve been inspired by rubrics shared by other teachers – primary, secondary and tertiary teachers. I can’t remember all of the rubrics I’ve drawn from but this year, I found two in particular very helpful, and I’ve borrowed words from both. They are both from iRubric:

The practicalities

While I track the volume of students’ posts in our various online spaces, I am more interested in what they contribute, rather than how much they contribute (although the latter is important too). This means I need to look at students’ contributions, not just statistics, to grade them. This is not easy and it’s not quick, but I believe it’s worth the effort. When I use a WordPress blog network where students have their own blogs, in combination with BuddyPress, it’s relatively easy for me to go back and see what students have contributed. This semester, my students’ blogs are all over the web, so it’s a bit trickier to track their contributions.  I have a multi pronged approach this semester, but the main thing I’ve had to think about is how to capture students’ comments on their peers’ blogs in a way I can work with to grade them. I’m doing this by aggregating the comment feeds from my students’ blogs into a single RSS feed, which I’m scraping into a Google Spreadsheet.

If This Then That recipe for scraping an RSS feed into a Google Spreadsheet.

If This Then That recipe for scraping an RSS feed into a Google Spreadsheet.

A colleague of mine is manually tracking contributions this semester by tallying them in a spreadsheet. She is much more disciplined than me though! I wish I could commit to that level of tracking, but the reality is I’d fall behind and I’d invariably give up. So automating works for me!

I’d love to hear your ideas for tracking student engagement in learning communities, and any thoughts you have on assessing contributions effectively. Please share in the comments!

12 May

Periscope for live broadcasting classes


Last night I had a colleague Chris Willems come along to my class to run a workshop on talking to camera. In these workshops, Chris does an exercise where he gets students to stand up in front of an iPhone and talk to camera on a random topic that he gives you immediately before counting you in to start recording.

This workshop is a lot of fun, but it is tricky to make it engaging for the online students who are at home listening in via Collaborate. (Note: We bring online students ‘into’ our on campus classroom using Collaborate.) They can’t see what’s happening when the students in class do the activity. We have problems using video in Collaborate because it chews up a lot of bandwidth – to the point where I just tend not to use it at all because it messes with the quality of the audio. So, seeing students would be talking to a phone camera anyway, I thought we’d have a go at broadcasting the activity via Periscope for our online students to watch live.

Here I am having a go at the talking to camera activity - screenshot from the Periscope replay.

Here I am having a go at the talking to camera activity – screenshot from the Periscope replay.

It worked beautifully and the online students loved it. Not only could they see us, but the audio quality was significantly better than Collaborate (not hard to beat that though really!). I’ve been watching the replay back this morning and I’m amazed that the microphone on my iPhone picked me up when I was at least five metres away from it. We just don’t get that in Collaborate using the lapel mics we have in the classroom. As a result, any discussion that happens in the on campus classroom becomes fragmented and unintelligible for the online students.

The other thing we struggle with in Collaborate is getting online students to watch videos at the same time we do in class. This is realllly complicated. I find uploading videos to Collaborate is almost always problematic so I tend to use clips that are online and send the online students the link for the video so they can watch it at the same time we’re watching it in class. But I also have to remember to turn the microphone off in Collaborate so the audio in the classroom doesn’t get broadcast into Collaborate. With Periscope, I just stuck the tripod in front of the projected image in the classroom and it worked a treat. The audio was good and the students at home could see the video. (Don’t worry, all of these videos were Creative Commons licensed so I wasn’t doing anything illegal!)

I didn’t realise I could set Periscope to automatically save videos to my camera roll (I tend to jump in the deep end and try stuff out without reading the instructions. Sometimes this gets me into trouble, but more often than not it works out okay). When I got home last night and investigated, I realised I could have saved the recordings to my phone at the end of the broadcast, as the video was being uploaded for replay, but I don’t think you can do it after that (or at least I haven’t been able to figure out how).

Verdict? We loved it. The online students were really happy with the quality so we kept it running for the rest of class. I turned the broadcast off while we were all working on an independent activity, and the online students asked me to start it again afterwards.

What’s missing? Conversation. Obviously if my phone is on a tripod capturing me talking, I’m not going to be able to engage with the students at home via Periscope. Viewers can heart the broadcast at various stages and they can write messages in the chat (both of which appear on the replay – nice!), but we can’t have a text based conversation.

What we didn’t love? Students without iDevices couldn’t join us (Periscope is currently iOS only).

Will I use it again? Yup, definitely. It’s not a replacement for Collaborate because of the limitations (no conversation and iOS only), but I think it’s an easy thing to run in addition to Collaborate. It will improve the online experience for students who attend live classes via our virtual classroom tools with virtually no extra effort, and that is a very good thing.

PS. I’m @katiedavis on Periscope.