WHOOPPEE (For Teaching Team Members)

This article is suited for members of the associated course's teaching team, and explains the basics of using the WHOOPPEE peer grading app from an administrative perspective. (If you're looking for the student experience, please click here.)

What is WHOOPPEE?

WHOOPPEE is the Wharton Online Ordinal Peer Performance Evaluation Engine which allows students to anonymously submit, rank, and review assignments from their peers. The app from an administrative perspective operates through 4 phases: Submissions, Rank & Review, Analytics, and Results. Its basic workflow is:

1. Submit Assignment. Assignments are submitted through Canvas.
2. Personally Identifiable Information (PII) Sanitation. WHOOPPEE anonymizes the submissions through metadata erasure, and assigns them random titles.
3. Random Assignments. WHOOPPEE distributes to the student random batches of submissions from the student's peers.
   • The teaching team (consisting of the instructor and course TAs), at the instructor's discretion, will receive Gold Standard batches. This will be subsequently discussed.
4. Student Review of Peers. Student reviews the peer submissions, comments on them and ranks them in order of strength.
5. Peer Comments Available. Once the student has reviewed their peers' submissions, the student's own submission, with peer comments, becomes available to review.
6. Results. WHOOPPEE displays the results of the assignment (including the student's grade).

WHOOPPEE also provides some helpful context in addition to the grade of the submission itself. Examples include:

• The strength of the batch that the submission was in
• The strength of each reviewer based on the rank that reviewer’s submission received (where the Gold Standard would be ranked equivalently to the top student’s)
• The strength of the student’s own ranking of 5 submissions (where mis-rankings will likely negatively affect the student’s own ranking).
• Peer responses to the students' reviews of their submissions, to help indicate if the reviews were helpful or not.

All of this combined helps determine the grade for the assignment.

What is the Teaching Console, and What Does It Let Me Do?

The Teaching Console is a dashboard available for teaching team members that allows for easy viewing of students' progress during the WHOOPPEE process. It will change slightly based on the current phase, but much of the functionality is similar from phase to phase.

• At-a-glance reporting for student status, with filtering:
  • Submissions Phase
    • Students with successful submissions
    • Students that have not submitted yet
    • Students that have submitted, but have an anonymity check (PII) failure
      • This shouldn't be an issue, but if there is one, feel free to escalate to Wharton Computing's WHOOPPEE Support.
  • Rank and Review Phase
    • Reviews not started
    • Reviews partially completed
    • Concerns reported from a student on submissions they're reviewing (this is usually something along the lines of "I can tell who wrote this paper")
• Singular or bulk actions on students, including:
  • Reminding of student to complete assignment (as an editable template) from the app
  • Warning of student to anonymize submission (as an editable template) from the app
  • Singular or bulk e-mailing (no template) from the app
  • Immediate PII concern status clearing, to allow affected students to advance to the Rank and Review phase
    • This should be used for those cases where there is a false positive flag of a submission having non-anonymous information, and the teaching team can verify the submission as valid
    • As above, this shouldn't be an issue. But please report to WHOOPPEE Support if you need.
• Report downloads using CSV (comma-separated-value) files, for further manipulation in Excel or other spreadsheet software
• Student Submission downloads, individually
• Removal/Deletion of assignment (in case the assignment needs to be set up again)
• Emulation of student experience, similar to Canvas' emulation feature

Step-by-Step Guide

Here is an example of someone with administrative privileges going through an assignment with WHOOPPEE.

Intro: Courses and Assignments

1. Log into the WHOOPPEE app through the link provided in the assignment. You will then see the Courses page, which provides a list of courses. When you see your relevant course, click it to advance.
2. Click on your course to advance to the Assignments page.
3. On the Assignments page, you'll see a list of Linked (assignments set up for WHOOPPEE) and Unlinked assignments, as well as their due dates and the current Status (or phase) that WHOOPPEE is in. Click the assignment that's relevant to you to advance to the Submissions Dashboard.Be sure to check the Linked portion and click the assignment; Unlinked ones are not set up for WHOOPPEE.

Submissions Dashboard

1. On the Submissions Dashboard, you'll see all of the students' names, IDs, submission titles, submission date/time, the current submission status, and a link to download the submissions.The Submission Dashboard. There are many tools available on each of the dashboards.
   • Other features to take note of, as outlined above in the screenshot:
     • The Trash Can icon next to the assignment name will delete the assignment completely. Please make absolutely sure you want to delete the assignment before clicking it; you'll be asked to confirm, just in case you clicked it by accident. The assignment will then be removed and will have to be recreated; it cannot be retrieved.
     • The Active Refresh button (the circular arrows) allow the data to refresh every minute.
       • You can also use the blue Refresh button below it to reload the data manually without having to refresh your entire browser window.
     • The All Students and Actions drop-down menus allow you to filter students and execute actions on the selected students (respectively).
     • The check marks next to the name allow you to select individual students. Check the box next to the Name column to select all students.
       • For example, you could select the student with the PII Check Failed using the check mark box, then click Actions > Clear Concern to remove that flag from their status.
     • The download buttons (the blue clouds with the arrows) enable you to download individual submissions.
     • If you plan on manipulating this data table in Excel, you can select all of the students using the check box in the top left and use the Download Selected File button to get all of the information you see on screen as a CSV file.
2. Once you've confirmed all expected students have submitted, you can click the To Peer Review/Rank button to close the Submissions phase. This will advance WHOOPPEE to the Rank and Review phase, and the students will automatically see the page change. 

Rank and Review Dashboard

The Rank and Review Dashboard is very similar in view to the Submissions Dashboard. It is also searchable, filterable, and able to perform bulk actions.  You can also download all the data as a CSV file, as with the Submissions phase.

The Peer Rank/Review Dashboard will appear when you advance the phase to Peer Rank/Review.

As with the Submissions phase, you can easily see the status of the students. The main status categories will be:

• Reviews Not Started: The student hasn't started the rank/review process yet.
• Partially Reviewed: The student started, but hasn't finished, the rank/review process.
• Concern Reported: The student in question has submitted a concern about one (or more) submissions within the batch they've received; the usual case is that there's something within the submission that could potentially identify the student that wrote it.
  • You should reach out to the student to get more info about the concern and escalate to the faculty member for further guidance if necessary. Once you're satisfied, you can check the box next to the student and click Clear Concern under the Actions menu. The student's status will the revert to the Ranked X/X Commented X/X status (if they're mid-ranking and review).
• Ranked X/X Commented X/X: This should indicate the student has completed the Rank and Review phase, where X is the number of submissions in the batch. The usual number will be 4 or 5 per batch, so it should look something like the example below.Here's an example of a student with a completed Peer Rank/Review phase.

When all students have completed the Rank and Review phase (all statuses should say Ranked X/X Commented X/X), you can click the Run Algorithm button at the top to advance to the Algorithm phase.

Gold Standard Reviews

Gold Standard Reviews are ranks/reviews conducted by the teaching team. Use the following steps to generate Gold Standard batches for review.

1. Click Gold Standard Reviews at the top of the WHOOPPEE app.
2. Click Generate All Gold Standard Sets to begin the process.
3. Once the sets are generated, you have one of two options:
   1. Use the Excel File Upload Tool above the blue buttons (Generate All Gold Standard Sets and Download Completed Sets) to upload your batches downloaded from WHOOPPEE.
   2. Use the In-App Batches Tool on the page to update your sets from WHOOPPEE itself. This tool will appear once the batches are generated. This method excludes Excel completely.

We will cover both methods below. For now, you will see this page, which has abbreviated versions of the instructions below.

When you click Gold Standard Reviews at the top of the app, you'll be presented with this page (if no batches have been generated yet). Note that there are no batches generated yet in this example.

Using the Excel File Upload Tool

To use the Excel File Upload Tool:

1. Click Download Completed Sets to download the batches.
2. Open Excel and fill in the Rank and Comment sections. You shouldn't need to fill in any other fields; this keeps the integrity of the file intact.
3. When completed, return to the Gold Standard Reviews page, click Browse to select your file, then click Submit to upload the batch.
   • Be sure the file is in .xlsx format.
4. Repeat this process until all batches are completed. Be sure to complete all sets before moving to the next phase.

Using the In-App Batches Tool

To use the In-App Batches Tool:

1. Click and hold the left handle to drag and drop each submission and order them from highest to lowest rank (top to bottom, respectively).
2. You can leave a review by clicking Expand and filling your review in the text box on the right that appears.
3. When your batch is done, click Update Set to complete it.
4. To switch between batches, select the drop-down menu with the listed Batch IDs.
   • Be sure to complete all sets before moving to the next phase.

Here's an overview of what the page looks like, with some instructions embedded. Use the purple instructions for Excel rank and review, and the orange instructions for WHOOPPEE rank and review. Both will provide the same data; it's just a matter of personal preference.

The Gold Standard Reviews page lets you rank and review submissions in two ways. 

Analytics Phase

The Analytics phase uploads the submissions to AWS and runs the WHOOPPEE algorithm. This calculates the final scores for the students.

Once the algorithm is complete, you will receive a notification e-mail. You can then click To Display Results to transition to the last phase, the Results phase.

• Contingent on how many students are in the course, the WHOOPPEE algorithm can take anywhere from a few minutes to a few hours to run completely. If you are unsure of the algorithm's status, please feel free to reach out to Wharton Computing staff and they will verify it.

Faculty Adjustments / Confirmation Dashboard

The faculty Adjustments / Confirmation Dashboard allows the  instructor to view student scores and make immediate adjustments to scores before moving to the Result Dashboard where students are then notified that scores are ready.

• Once a score has been manually updated you will be presented with the (Save) option to accept  changes.
•  To update scores in bulk download the Results Data file make your updates in score column and re-upload in .csv format. Once accepted you can choose (Save) to complete the task.

Adjustments / Confirmation Dashboard allows the  instructor to view  student scores and make  adjustments to scores.

Results Dashboard

In the Results phase, no action is required from a student-monitoring perspective, since the WHOOPPEE run is completed at this final phase. Students' statuses should all be marked as Success, aside from those that did not submit before the due date. The results can then be downloaded as a CSV file and viewed by the teaching team to verify the results and provide a more granular perspective.

See the example listed below (note that the Paper Author ID will be populated with the students' anonymous ID):

This CSV file contains a more granular view of the final results, to help explain how the final scores are calculated.

Technical Problems?

Because of the way web browsers interact with web applications, sometimes data can become "stuck" or otherwise not function as expected. This happens sometimes in WHOOPPEE, as well.

The simplest way to resolve most technical issues is to perform the following steps:

• Refresh your web browser.
• Clear your browser's cache and/or cookies.
  • Click the link for your particular web browser for instructions (it will open in a new tab):
    • Chrome / Firefox / Safari / Opera / Edge / Brave
• Attempt to use a different web browser, since different web browsers can render pages in slightly different ways.
• Reboot your device, or use a different one. WHOOPPEE should cache your progress in the cloud, so you should be able to pick up where you left off.

Troubleshooting

Questions?

Please contact Wharton Computing if you notice any technical issues not resolved by performing the steps in the previous section.