Demo of online peer-review tool Eli Review

Why do I need another peer-review demo?

If you want to do online peer review, you should be familiar with the variations in the tools. When I did the Peerceptiv demo, I was very interested to see its operation. Its focus on supporting the use of student evaluators to be able to grade each other and its focus on quantitative validity and reliability are admirable.

I think I share Professor Norton’s philosophy about the value of peer review, but my own inclinations are more qualitative, and the Eli Review peer-review platform provides what I would describe as a great peer-review learning experience for students and a more qualitative teaching-support tool for me.

In fact, Eli Review unlocks four of the five research-validated benefits of peer review that Tracy and Peerceptiv identify:

1. Student writing performance improves significantly when students provide peer feedback. Students learn through engaging in teaching their peers as they review.
2. Student writing performance improves more through feedback from multiple peers than through single instructor (expert) feedback.
3. Students who only provide feedback on the work of others perform better than those students who only write.
4. Grades generated from multiple peers tend to be more reliable and just as valid as those generated from a single expert instructor.
5. Peer assessment uniformly benefits weaker and stronger writers regardless if the feedback is generated from weaker or stronger reviewers. There is no discernible improvement from breaking peers into similarly-skilled subgroups.

My sense is that the tilt of the Peerceptiv team’s research is feedback (items 1, 2, 3, and 5 of this list), but the tilt of their application is grading (item 4). Eli Review simply does not engage with grading. I use it as a tool not to assign grades to students, but rather to decide how I’ll teach my next (or current) class with them.

Quick overview of Eli Review and registering

Eli Review provides a platform where writers can get feedback on small chunks of writing (they say as little as a sentence, but I’ve never gone with something that short). It supports the following tasks:

• Writing. Students submit some chunk of text.
• Reviewing. Eli Review and the teacher model and guide effective feedback from peers. (Of course, it is the giving, not the getting of feedback that delivers the greatest pedagogical punch.)
• Debriefing. Teacher and students debrief the experience, talk about problems, questions, etc. And the teacher gets real-time feedback on learning that allows them to make evidence-based decisions on where to go next.
• Revising. Eli Review and the teacher guide the students to plan and make revisions.
• Rinse and repeat. Students can then submit revised texts.

There is much more detail on their site, but I suggest just doing this demo to see how it works. Instructor accounts are free. If your school does not have an Eli Review license, the cost for students is $12.50 for three months, $19.95 for six months, and $29.95 for one year. I simply assign it as a textbook for my class, recommending the six-month term. (There is a two-week grace period at the beginning of a course term, so students who need financial aid to purchase texts have a bit of time.)

To do the demo, you need to create an account (which is free for this course for all users). Go just go to Eli Review to set it up. Then follow these steps (see the graphic):

1. Once you sign up and log in, you’ll see options for two dashboards. Click on the the Student Dashboard, the second tab on the left in the gray bar.
2. In the Student Dashboard, type this course code in the box: avoids174viewer
3. Click the button “Join Course.
4. The course “Legal writing DEMO” will be listed in the table below. Click on the title of the course to enter.
5. In the course dashboard, click on the title of a task to view or submit work.

 

Using Eli Review in LRW pedagogy

My general approach to teaching reasoning techniques and genres in legal communication has six steps:

• 1. Reading. I give students something to read that explains a reasoning technique or writing genre and its constraints. (I don’t give them a rubric or checklist, but I supply info for them to create their own.)
  2. Writing. I ask students to try it. I alert them that I’m not giving them much guidance on how to do it yet, but that trying it before we go more deeply situates them to better understand our discussions later. Students get “check points” for doing the assignment.
  3. Reviewing. I have them review each other, using the things that matter to me when grading, all of which were evidenced in the reading. I ask them to use the “Describe-Evaluate-Suggest” approach for giving feedback (described further below). As long as the students use D-E-S, they get check points for this assignment, too.
  4. Discussing/debriefing. We have a class discussion/debriefing, in which I ask them to rate the reviews they received (based on criteria discussed below). We discuss the results, I show examples, we consider questions, etc.
  5. Revising. I ask students to write using the technique or genre again.
  6. Submission. I grade it, based on the criteria set out in the reading and the discussions.

I tailor the discussion/debriefing  to focus on things that the Eli Review analytics suggest students are struggling with. I use the analytics from Eli Review after the reviewing to identify good examples of student performances writing and reviewing. I share them with students in the discussion/debriefing where we discuss them; I ask students to discuss interesting things they saw in other students’ drafts.

This puts an end to the questions about “the right way” to do something.

The students don’t recognize (and I’m not sure they believe me when I tell them) that giving feedback is where it’s at. But they do recognize the value of seeing that their peers are doing things differently but often just as effectively. This puts an end to the questions about “the right way” to do something. It enhances their rhetorical awareness that the right way depends on the context, and there is often more than one right way.

Writing assignment (due June 3)

[The balance of this post provides instructions, pretty much exactly as I would provide them to the students. The writing assignment is the first that I have 1Ls do in the fall semester.]

Assignment: Write your elevator pitch and post it on Eli Review in the assignment titled Elevator pitch on or before the date specified above. This assignment is worth three points toward your Exercises and Preparation grade. You may write the pitch in Eli Review or write it in your word processor and copy and paste it into Eli Review.

Instructions: Starting today, you should have an elevator pitch, a brief statement about who you are that you will use when introducing yourself in professional contexts. Consider this scenario: You are at the federal courthouse in Dallas. Riding down in the elevator, you are standing next to an older person wearing a black robe. She notes that you look young, eager, and perhaps a little nervous and recognizes you as a law student or maybe a young attorney. She brightly introduces herself as “Judge Marjorie Gutierrez.” After you give your name, she says “Tell me about yourself!”

Your elevator pitch helps a listener in a professional context know where you “fit” in that context. It should quickly identify your current role and where you hold it. It should tell us something about how you fill that role. In the case of a law student, that usually means saying either what kind of law you are interested in or what kind of job you want to take after law school. Of course, you may very well not know the answer to that yet. You should nevertheless express some kind of interest. If you do express an interest in a particular area of law, make sure you have an answer to the common follow-up question: “What got in you interested in X?” It’s embarrassing not to have an answer to that one, if it’s asked, but you don’t need to explain that in the pitch itself unless you think it achieves some other objective.

Your pitch will tell us something about your background and perhaps about you as a person. This might be as simple as saying what your undergraduate training or previous work experience was. Ideally, though, it will tell us something memorable. It should do all this in 30 seconds or less.

Here’s an artificial example:

Howdy! My name is Martin Frankel, but everyone calls me ‘Gus.’ I’m a first-year law student at Texas A&M. I’m most interested in securities regulation, but I’m still pretty open to other possibilities. Law school is a nice change from last year: I spent six months in the Amazon collecting monkey urine on a research expedition for Cornell’s College of Biology. What kind of work do you do?

If asked, Gus would say he got interested in securities regulation while following the trial of a childhood neighbor for securities fraud. The neighbor was acquitted. Though this example elevator pitch is artificial, I actually did have a student who collected monkey urine as part of a research expedition! That was a memorable part of her elevator pitch. Note that Gus’s pitch was short, informative, and memorable. It’s also a nice touch that he asked his listener to reciprocate. Sometimes a conversation like this between a law student and an attorney will result in a networking opportunity.

Your elevator pitch will change over time as your interests and experiences develop. You will want to tailor your elevator pitch for different audiences, too. Whenever you are going into a new situation where you expect folks to want to understand who you are, you should think first about what impression you want to make and then adjust your pitch accordingly. You should also not expect to get through the whole thing. Your interlocutor might interrupt you with questions or comments, and you should let the conversation unfold naturally. You can work the rest of the pitch into the conversation later, at some point where it seems natural; or perhaps you will never finish it, because you find yourself in an actual conversation with the other person. That’s a win!

Peer-review assignment (due June 8)

[Here are the instructions students receive for the first peer-review assignment.]

Assignment: Review the elevator pitches of the three or four of your peers assigned to you on Eli Review in the assignment titled PR of elevator pitch on or before the date specified above. This assignment is worth three points or four toward your Exercises and Preparation grade, one for each peer to whom you provide a complete review (including D-E-S comment, if required).

Instructions: Peer feedback figures prominently in this class. To demonstrate that you understand the skills required to achieve the learning outcomes, you will provide a great deal of peer feedback where you describe, evaluate, and make suggestions regarding your peers’ efforts in this class. According to Seneca the Younger (c. 4 BCE – 65 CE): Homines dum docent discunt. “People learn while they teach.” Contemporary research bears out the wisdom of this classical author. As the creators of the Eli Review software note:

• • “Reading others’ work lets you see what choices they’ve made. That gives you more options as a writer.”
  • “Checking to see if other writers have met the criteria will help you bring those criteria into better focus in your own work. You’ll have a clearer sense of how to succeed by using the criteria on peers’ work and your own.”

Perhaps most importantly to your development as a professional, you should recognize that great leaders give great feedback. Next year, when you are a teaching assistant for this course, or fifteen years from now, when you are a law partner giving feedback to a new associate, your ability to give valuable developmental feedback will be a measure of your value as a leader. Accordingly, the point of peer feedback in this class is not so much for you to get feedback to improve your own writing, but to give feedback to develop and demonstrate your knowledge of the course skills.

It is important to note that your peers are not grading you. The grade I give you on major graded assignments is based on the criteria for the assignment and the standards set out in this syllabus. The points I give you for Eli Review writing tasks and reviewing tasks are based on whether you do them on time and whether you follow the instructions for them.

In Eli Review, getting a glowing or scathing review from a peer on a writing task or getting “stars” from your peers or “endorsements” (thumbs-ups)  from me on your reviews will not change your grade on those tasks. Stars and endorsements do, however, signal that you are getting the material in this class, that you are understanding and learning. Consequently, work to get them!

When you do a peer review, follow the instructions on Eli Review. When it asks you to use the “Describe-Evaluate-Suggest” method, this is what you should do.

• • Describe some aspect of your peer’s performance. When you give feedback, it’s important for your peer to understand that you actually did carefully read what they wrote. For example, “Your elevator pitch included x…”
  • Evaluate it. Compare the described aspect to some evaluative criterion. This can take one of two forms:
    • You can note that the readings or I told you to do something a certain way. For example, “Larson said that the elevator pitch should communicate your role.”
    • Or you can put yourself in the shoes of the audience and identify something you think would matter to them and why. “I think if I were a judge meeting you in an elevator, I might expect y, because of z.”
  • Make a suggestion. The S should be related to the D and E. It should not relate to issues like grammar and writing mechanics.
  • Be respectful. Throughout, you should talk to your peers and about their writing in a respectful way—it’s an essential part of your professional identity.

You can provide these in any order, as long as all three are present. Here are two examples of complete D-E-S comments:

Larson said that the elevator pitch should communicate your role (Evaluative criterion). Your current pitch does not do that (Description), so that’s something I’d add (Suggestion).

Here, the evaluative criterion came from the assignment prompt.

You said you were planning to use this elevator pitch at a reception. I think it might be noisy there and harder to keep the attention of your listener, so you might want to be extra careful about the length of your pitch (Evaluative criterion). Right now, yours is 150 words (Description), which is twice as long as Larson’s example. You might want to shorten it, or at least make sure you put the most important information first (Suggestion).

In this example, the reviewer considered the context where the author would use the pitch and assessed it based on how a listener might respond.

After you give your reviews and your peers review you, you will be able to see their reviews of your work. Please wait until we meet to assess their reviews of your work (i.e., by giving them ‘stars’).

In the second Zoom discussion…

In the second Zoom discussion, I’ll show you how the teacher sees the results of this process, how I use them, and how students give each other feedback on the feedback, at least in my classes. For about 20 minutes of that time (maybe longer if we are having good time), I’ll teach you as if you are my students. After that, the floor will be yours for questions and comments!

If you have questions in the meanwhile, please just email me!

-Brian