WHOOPPEE (For Teaching Team Members)

This article is suited for members of the associated course's teaching team, and explains the basics of using the WHOOPPEE peer grading app from an administrative perspective. (If you're looking for the student experience, please click here.)

Minimum Requirements 

You will need the following before you can use WHOOPPEE: 

  • A laptop or desktop PC
    • Currently tablets and phones are not supported.
  • A current web browser
    • Modern browsers are supported; if Canvas works in your browser, WHOOPPEE should as well.
  • A PDF Reader for PDF metadata inspection
    • Acrobat is recommended for both Windows and Mac users, but Mac users can also use Finder and Preview to inspect the metadata.

What is WHOOPPEE?

WHOOPPEE is the Wharton Online Ordinal Peer Performance Evaluation Engine which allows students to anonymously submit, rank, and review assignments from their peers. The app from an administrative perspective operates through 4 phases: Submissions, Rank & Review, Algorithm, and Results. Its basic workflow is:

  1. Submit Assignment. Assignments (anonymized) are submitted through Canvas.
  2. Anonymity Check. WHOOPPEE checks to see if Step 1 has been completed anonymously.
  3. Random Assignments. Whooppee distributes to the student random batches of submissions from the student's peers.
    • The teaching team (consisting of the instructor and course TAs), at the instructor's discretion, will receive Gold Standard batches. This will be subsequently discussed.
  4. Student Review of Peers. Student reviews the peer submissions, comments on them and ranks them in order of strength.
  5. Peer Comments Available. Once the student has reviewed their peers' submissions, the student's own submission, with peer comments, becomes available to review.
  6. Results. Whooppee displays the results of the assignment (including the student's grade).

WHOOPPEE also provides some helpful context in addition to the grade of the submission itself. Examples include:

  • The strength of the batch that the submission was in
  • The strength of each reviewer based on the rank that reviewer’s submission received (where the Gold Standard would be ranked equivalently to the top student’s)
  • The strength of the student’s own ranking of 5 submissions (where mis-rankings will likely negatively affect the student’s own ranking).

All of this combined helps determine the grade for the assignment.

What is the Teaching Console, and What Does It Let Me Do?

The Teaching Console is a dashboard available for teaching team members that allows for easy viewing of students' progress during the WHOOPPEE process. It will change slightly based on the current phase, but much of the functionality is similar from phase to phase.

  • At-a-glance reporting for student status, with filtering:
    • Submissions Phase
      • Students with successful submissions
      • Students that have not submitted yet
      • Students that have submitted, but have an anonymity check (PII) failure
    • Rank and Review Phase
      • Reviews not started
      • Reviews partially completed
      • Concerns reported from a student on submissions they're reviewing (this is usually something along the lines of "I can tell who wrote this paper")
  • Singular or bulk actions on students, including:
    • Reminding of student to complete assignment (as an editable template) from the app
    • Warning of student to anonymize submission (as an editable template) from the app
    • Singular or bulk e-mailing (no template) from the app
    • Immediate PII concern status clearing, to allow affected students to advance to the Rank and Review phase
      • This should be used for those cases where there is a false positive flag of a submission having non-anonymous information, and the teaching team can verify the submission as valid
  • Report downloads using CSV (comma-separated-value) files, for further manipulation in Excel or other spreadsheet software
  • Student Submission downloads, individually
  • Removal/Deletion of assignment (in case the assignment needs to be set up again)
  • Emulation of student experience, similar to Canvas' emulation feature (for Wharton Computing staff only)

Step-by-Step Guide

Here is an example of someone with administrative privileges going through an assignment with WHOOPPEE.

Intro: Courses and Assignments

  1. Log into the WHOOPPEE app through the link provided in the assignment. You will then see the Courses page, which provides a list of courses. Test course exampleWhen you see your relevant course, click it to advance.
  2. Click on your course to advance to the Assignments page.
  3. On the Assignments page, you'll see a list of Linked (assignments set up for WHOOPPEE) and Unlinked assignments, as well as their due dates and the current Status (or phase) that WHOOPPEE is in. Click the assignment that's relevant to you to advance to the Submissions Dashboard.List of courses for WHOOPPEEBe sure to check the Linked portion and click the assignment; Unlinked ones are not set up for WHOOPPEE.

Submissions Dashboard

  1. On the Submissions Dashboard, you'll see all of the students' names, IDs, submission titles, submission date/time, the current submission status, and a link to download the submissions.Submissions Dashboard example, annotatedThe Submission Dashboard. There are many tools available on each of the dashboards.
    • Other features to take note of, as outlined above in the screenshot:
      • The Trash Can icon next to the assignment name will delete the assignment completely. Please make absolutely sure you want to delete the assignment before clicking it; you'll be asked to confirm, just in case you clicked it by accident. The assignment will then be removed and will have to be recreated; it cannot be retrieved.
      • The blue Refresh button at the top can be used to reload the data without having to refresh your entire browser window.
      • The All Students and Actions drop-down menus allow you to filter students and execute actions on the selected students (respectively).
      • The check marks next to the name allow you to select individual students. Check the box next to the Name column to select all students.
        • For example, you could select the student with the PII Check Failed using the check mark box, then click Actions > Clear Concern to remove that flag from their status.
      • The download buttons (the blue clouds with the arrows) enable you to download individual submissions.
      • If you plan on manipulating this data table in Excel, you can use the Download All Data as CSV button down to get all of the information you see on screen as a CSV file.
  2. Be sure all submissions have been anonymized by first noting the "PII check failed" status and resolving concerns.
    • To make it easier, you can filter the students down by "Non Anonymity".
      • You also should receive an email whenever a PII check fails for a student; it will indicate what metadata was detected, and where.
    • To confirm the presence (or absence) of personally identifying metadata, download the file from the student(s) in question from the dashboard.
      • If you're running Acrobat on Windows, follow the instructions here to inspect the document.
      • If you're on a Mac, simply right-click the file and click Get Info. Scroll down and you should see if any personally identifying metadata is included.
    • If there is metadata, have the student(s) sanitize the data (using either the automatic tool or the instructions provided to do it manually) and re-submit in Canvas.
    • If there is no metadata, tick the box next to the student(s) and select Clear Concern from the Actions menu.
  3. Once you've confirmed all expected students have submitted, you can click the To Peer Review/Rank button to close the Submissions phase. This will advance WHOOPPEE to the Rank and Review phase, and the students will automatically see the page change. 

Rank and Review Dashboard

The Rank and Review Dashboard is very similar in view to the Submissions Dashboard. It is also searchable, filterable, and able to perform bulk actions.  You can also download all the data as a CSV file, as with the Submissions phase.

Peer Rank/Review Dashboard ExampleThe Peer Rank/Review Dashboard will appear when you advance the phase to Peer Rank/Review.

As with the Submissions phase, you can easily see the status of the students. The main status categories will be:

  • Reviews Not Started: The student hasn't started the rank/review process yet.
  • Partially Reviewed: The student started, but hasn't finished, the rank/review process.
  • Concern Reported: The student in question has submitted a concern about one (or more) submissions within the batch they've received; the usual case is that there's something within the submission that could potentially identify the student that wrote it.
    • You should reach out to the student to get more info about the concern and escalate to the faculty member for further guidance if necessary. Once you're satisfied, you can check the box next to the student and click Clear Concern under the Actions menu. The student's status will the revert to the Ranked X/X Commented X/X status (if they're mid-ranking and review).
  • Ranked X/X Commented X/X: This should indicate the student has completed the Rank and Review phase, where X is the number of submissions in the batch. The usual number will be 4 or 5 per batch, so it should look something like the example below.Here's an example of a student with a completed Peer Rank/Review phase.

Note: There is no need for students to rank and review all submissions in one sitting. The student can easily review one 

When all students have completed the Rank and Review phase (all statuses should say Ranked X/X Commented X/X), you can click the Run Algorithm button at the top to advance to the Algorithm phase.

Algorithm Phase

The Algorithm phase uploads the submissions to AWS and runs the WHOOPPEE algorithm. This calculates the final scores for the students.

Once the algorithm is complete, you will receive a notification e-mail. You can then click To Display Results to transition to the last phase, the Results phase.

  • Contingent on how many students are in the course, the WHOOPPEE algorithm can take anywhere from a few minutes to a few hours to run completely. If you are unsure of the algorithm's status, please feel free to reach out to Wharton Computing staff and they will verify it.

Results Dashboard

In the Results phase, no action is required from a student-monitoring perspective, since the WHOOPPEE run is completed at this final phase. Students' statuses should all be marked as Success, aside from those that did not submit before the due date. The results can then be downloaded as a CSV file and viewed by the teaching team to verify the results and provide a more granular perspective.

See the example listed below (note that the Paper Author ID will be populated with the students' anonymous ID):

A spreadsheet with final results data.This CSV file contains a more granular view of the final results, to help explain how the final scores are calculated.

Questions?

Please contact Wharton Computing if you notice a technical issue.