Gradebot Teacher's Manual

Git repositories

Gradebot uses git to store the teacher's grading scripts (called homework) and the students' programs (called program). Their repository addresses are:

Homework
metastasis@{server}:homework/{courseID}/{homeworkID}/grade
Program
metastasis@{server}:user/{username}/{courseID}/{homeworkID}

For example:

A teacher may check in and out any homework and the program of any student, even after the homework deadlines.

Note: If a teacher wishes to check in his own program, he must register himself as a student in this course. The same user may be both a student and a teacher.

Grading scripts

Each homework, consisting of multiple grading scripts, has the following directories and files, some of which are mandatory.

build/init (mandatory)
Script for compiling the student program.
run/init (mandatory)
Script for running the student program.
grade/init (mandatory)
Script for grading the student program.
src/ (optional)
Contains the source of the reference program, and scripts for compiling the reference program and creating test inputs.
share/ (optional)
Contains scripts shared by all the above directories
Makefile (optional)
For compiling the reference program and creating test inputs.
config.yaml (optional)
Contains configuration options for LXD, which the gradebot uses for sandboxing student programs. This file is in the YAML format and currently recognizes the following optional keys:
config.limits.memory
The maximum allowed memory for the LXD container. This string value must end in the kB, MB, GB, TB, PB or EB suffix. (Note: due to a bug in LXD in Ubuntu 16.04, this option is currently ignored.)
devices.root.size
The maximum allowed disk size for the LXD container. This string value must end in the kB, MB, GB, TB, PB or EB suffix. (Note: due to a bug in LXD in Ubuntu 16.04, this option is currently ignored.)
timeout
The value specifies an integer timeout in seconds. The gradebot kills a grading script (build/init, run/init, or grade/init) if it fails to finish within this timeout.

Grading process

This sections describe how the gradebot invokes the grading scripts.

When grading scripts are submitted

After the teacher pushes his grading scripts, the gradebot checks out this repository. If the root directory of the repository contains Makefile, the gradebot runs

make COURSEID={courseid} HOMEWORKID={homeworkid}

in the root directory of the repository.

Advice: A typical Makefile builds the reference program in src/. Then, it runs the reference program on each input in run/input/ and saves the output in grade/output/.

When a student program is submitted

After the student submits his program, the gradebot runs in three passes.

  1. Build: The gradebot runs build/init.
  2. Run: The gradebot runs run/init.
  3. Grade: The gradebot runs grade/init.

The command line arguments to all the above init scripts are (without the curly braces)

{username} {courseID} {homeworkID}

In every pass, the grading script can access other directories listed in Table 1, where the Mounted as column shows the mounting point of each directory visible to the grading script.

Table 1: Directories mounted and visible to grading scripts.
RepositorySubdirectoryMounted asAvailable duringWritable?
BuildRunGrade
Grading scriptsbuild/gradebot/buildNo
run/gradebot/runNo
grade/gradebot/gradeNo
share/gradebot/shareNo
Student program/gradebot/programYes
Provided by the container/gradebot/ioYes

The directory mounted at /gradebot/io is for the grading scripts to send data to the gradebot via the following files. This directory persists across all the three passes. E.g., any file written under in this directory during the build pass will be accessible to the run and grade passes.

Security warning: Since the student program may run during the build and run passes, it may modify any file in the /gradebot/io directory. Therefore, the grading scripts should defend against unauthorized modification to the above files. I recommend the following approach.

Debugging grading scripts

Debugging Makefile is easy, since you can run "make" on your local machine

To debug build/init, run/init, grade/init, you would need to set up the mounted directories in Table 1 above.

You can view gradebot's errors during grading in Set up course and homework -> Debug logs on the web interface.

Security

The gradebot divides the grading process into three passes to enforce security properties.

  1. Build. Since the grading script may invoke the student's Makefile, which may run malicious code, the gradebot does not mount the run (typically containing test inputs) or grade (typically containing reference test outputs) directories to prevent the Makefile from stealing the test inputs or outputs. Otherwise, a malicious Makefile might read the test inputs or outputs and then echo them to stderr, which the grading script typically sends to the web interface to help students debug their compilation problems.
  2. Run. The gradebot doesn't mount the grade directory to prevent the potentially malicious student program from stealing the test outputs in this directory. To prevent the malicious program from stealing the test inputs in the run directory, the grading script should not send the stdout or stderr from the student program to the web interface.
  3. Grade. The grading script should invoke no student prorgam during this pass to prevent malicious program from interfering with how the grading script calculates the grade.