Reduct is a game designed to measure the basic computer science skills of incoming CS students. It functions similarly to Scratch, in that instead of typing code, players assemble colourful blocks to create scripts, with the goal of solving simple problems and testing understanding of concepts such as recursion and first-class functions.

An intermediate level in Reduct. The goal of the player is to get the pieces on the board to turn into the pieces in the goal box at the top.

It’s based on a similar game of the same name written by Professor Andrew Myers at Cornell, but the version of Reduct that I deployed for this experiment was a complete rewrite of the project in TypeScript. I rewrote the project for a handful of reasons:

• To add type-safety to the codebase: the original code was written in JavaScript by a professor who teaches functional programming languages. Modifying the code was extremely difficult.
• To make it easier to add new levels quickly: I needed to be able to iterate on the game’s levels quickly once I collected feedback, so I introduced a new YAML level format that I could quickly edit and deploy.
• To add data collection: The original Reduct game was not really designed to allow instrumentation. Despite being written in a colorful functional style, it used a lot of rather opaque global state.
• To change the rendering engine: The original Reduct used the HTML <canvas> for rendering. I considered this unnecessary as this wasn’t a highly graphics-intensive game, and I wanted to defer to the browser for hit-testing, rendering, and all of the things a browser already provides, rather than trying to maintain reimplementations of these things that worked 98% of the time.

The new implementation

Reduct is a complex enough game that rewriting it took a good fraction of the summer. And I had a deadline: the CASE (Cornell Advanced Standing Exam) for computer science students was taking place at the end of the summer, and the game needed to be ready by then for my experiment to work.

To speed things up, I got to work with the tools I was familiar with: namely, React and Redux. Because Reduct is a puzzle game where each move has a predictable impact on the game’s state, it was well-suited to Redux. Every move was implemented as an action or sequence of actions dispatched to the store:

• Dragging a 2 into a (_ + _) block dispatches an action which reparents the 2 to create (2 + _), followed by a reduce action which does nothing
• Dragging a 5 into a (2 + _) block dispatches an action which reparents the 5, followed by a reduce action which reduces (2 + 5) to 7

By ditching the canvas in favour of a DOM-based implementation, I could implement most of the game’s logic using simple HTML drag-and-drop events. The trickiest parts were ensuring that the logic of the game was consistent, and adding animations.

The deployment

Each move was logged using Google Cloud logging so that I could see the traces of each session in real time and download the logs in bulk for processing. This turned out to be super valuable when the CASE did roll around, because there were bugs:

My body still remembers those three days when the CASE was taking place. Using Sentry and Google Cloud I was able to trace and fix the issues almost immediately, but dreading the next email definitely took years off my life.

Despite a few pretty bad crashes, most of the sessions went off without a hitch, and I ended up with around 90 playthroughs of Reduct to analyze.

The analysis

Unfortunately, this post is going to end on a rather anticlimactic note. In preliminary analysis, this experiment showed a mild positive correlation between Reduct scores (such as time to completion, or number of levels completed) and traditional pen-and-paper CASE scores. However, if there is one thing I hate doing, it is statistics, and I quickly lost my motivation to continue this project now that the software engineering part was over.

As such, I cannot state anything conclusively about whether this game was good for placing students in the appropriate computer science course. Instead, what I gained from this project is:

• The ability to build and deploy a large project by myself: After all said and done, Reduct comprises over 14,000 lines of code (excluding comments and blank lines). This was the first time that I created such a large project, and I did this while also managing coursework and a summer internship.
• The ability to troubleshoot bugs in real time based on end-user complaints: End-users are not very good at telling you exactly what went wrong, so good instrumentation was crucial here.
• The ability to work with stakeholders to get something into users’ hands: Getting Reduct endorsed alongside the official CASE exam required negotiation. This was my first time launching a project that needed a green light from someone else.

Acknowledgements

On this project, I’m grateful for coding assistance from Ian Tomasik and guidance from Professors René Kizilcec, François Guimbrètiere, and Andrew Myers.