Bypassing the autograder with a single line of code

Previous Hack

As demonstrated in this previous page, there are many ways to easily bypass the CodeHS Autograder.

Before, I discovered that the CodeHS autograder only verifies that an assignment is complete by sifting through stdout for a __unittests__ keyword followed by a JSON object which contains testcase info. This meant that we could inject a payload in our java code as follows:

System.out.println("__unittests__{\"tests\":[{\"test\":\"Exploit Autograder\",\"success\":true,\"studentOutput\":\"3.14\",\"message\":\"woot woot!\",\"solutionOutput\":\"3.14\"}]}");
if (true) throw new RuntimeException();

I recently discovered that it’d be easier to use a static code block with System.err and System.exit() instead, so that I could place this code in the body of any loaded class:

static {
    System.err.println("__unittests__{\"tests\":[{\"test\":\"Exploit Autograder\",\"success\":true,\"studentOutput\":\"3.14\",\"message\":\"woot woot!\",\"solutionOutput\":\"3.14\"}]}");
    System.exit(1);
}

Drawbacks

Although this hack means that we don’t actually have to write the java code necessary to actually solve the problem, we are still required to provide java code that is able to compile, which oftentimes is the majority of the work in an assignment.

I wanted to find ways to make the autograder print the __unittests__ data without having to even run java.

First Attempt

A quick sift through the ps ax logs revealed that the autograder uses commandline javac and java commands to compile and execute java code respectively.

My first approach was to try to upload a fake Grader.java (or Java 8-compiled Grader.class) file onto the server, which would immediately spit out autograder results, without running any student code. However, the autograder overwrites files on the server (such as Autograder or Grader), making these attempts futile.

I even tried removing write permissions from Grader.java using chmod u-w Grader.java, but that resulted in an error in the javascript server, which caused the autograder to hang.

run.sh

One day, I was searching the internet for ways to change the CodeHS Java version (because Java 8 is literally prehistoric). I stumbled upon some CodeHS articles which explained that you could set your main file to a shellscript named run.sh. This would make it so that the “Run” button would execute the bash script, but the “Check Code” button would still run the same javac and then java commands with Grader.java to check your code.

Another crucial detail of this exploit was the fact that CodeHS reuses the same cloud computing session between “Run” and “Check Code” instances, as long as your connection doesn’t time out. This meant that whatever modifications made by my shell script would apply to the autograder, as long as I ran both on the same server session.

Breaking javac

I wanted to change the behaviour of the linux javac command so that it wouldn’t even have to compile my code. Using the shellscript, I examined the $PATH env variable to see if there was anything I could use:

echo $PATH
/home/karel/.jenv/shims:/home/karel/.dotnet:/usr/local/jenv/bin:/usr/bin/python3.11:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

This looked very promising. It looked like the .jenv/shims folder was stored in my user’s home directory, which meant that I had permission to write to them. This is where java stores its utility executables:

ls -l ~/.jenv/shims
total 216
-rwxrwxrwx 44 karel karel  389 Dec  1 01:01 java
-rwxrwxrwx 44 karel karel  389 Dec  1 01:01 javac
# many files ommitted for brevity

Running which javac then confirmed that the java compiler was indeed stored in the user home directory! This meant that all I had to do was replace javac with a phony executable that would print the __unittests__ data instead of compiling my broken code.

I needed to write this command into the javac file:

echo "__unittests__{\"tests\":[{\"test\":\"Exploit Autograder\",\"success\":true,\"studentOutput\":\"3.14\",\"message\":\"woot woot!\",\"solutionOutput\":\"3.14\"}]}"

To do that, I put this command into my run.sh file:

echo "echo \"__unittests__{\\\"tests\\\":[{\\\"test\\\":\\\"Exploit Autograder\\\",\\\"success\\\":true,\\\"studentOutput\\\":\\\"3.14\\\",\\\"message\\\":\\\"woot woot!\\\",\\\"solutionOutput\\\":\\\"3.14\\\"}]}\"" > $(which javac)

Results

So, with my run.sh file, I pressed the “Run” button, and then the “Check Code” button to test my hack. Success!

It works!1 This means that you can literally rewrite the java compiler to just spit out fake autograder results without even havking to write a single line of java code! The entire assignment can be bypassed with a single line of bash code in run.sh:

echo "echo \"__unittests__{\\\"tests\\\":[{\\\"test\\\":\\\"Exploit Autograder\\\",\\\"success\\\":true,\\\"studentOutput\\\":\\\"3.14\\\",\\\"message\\\":\\\"woot woot!\\\",\\\"solutionOutput\\\":\\\"3.14\\\"}]}\"" > $(which javac)

  1. This exploit only bypasses the autograder if you hit “Run” before “Check Code” ↩