Headhunter: A Reprogrammable Head-tracking Midi Controller

Gefei Tan

alt text

Introduction

Meet Headhunter: a head-tracking MIDI controller. It allows the user to control a MIDI-compatible instrument using… well, their head.

Headhunter utilizes the sensors already built into smartphones to track head movements, then translates that data into corresponding MIDI messages using Max. It is fully reprogrammable, meaning the user can easily customize Headhunter to map specific head wobbles to whatever MIDI parameter they want.

Inspiration

My inspiration for Headhunter comes from Keith Jarrett, one of my favorite jazz pianists. (Herbie Hancock also ranks high on my list, as hinted by the name Headhunter.)

Besides his constant and notorious moaning, Mr. Jarrett is known for his dramatic head movements while playing the piano.

alt text

These movements, often deemed distracting, inspired me to think: What if these “redundant” movements could actually control some aspect of the sound being played? By integrating these gymnastics into the musical performance, they become engaging for the audience, providing auditory feedback synchronized with the visuals. This also offers keyboard players significantly more freedom, enabling actions like bending notes or adjusting filters while both hands are busy playing chords—no extra knobs or sliders required.

Process

To bring this idea to life, I began with a proof of concept.

First, I researched head tracking and discovered an open-source software called OpenTrack. It supports various head-tracking inputs, including smartphones, making it accessible to musicians without the need to wear dedicated, sci-fi looking headsets.

Next, I worked on converting the yaw, pitch, and roll data from OpenTrack into a format Max could recognize. I decided to use the built-in virtual joystick output protocol to encode the head-tracking information into a virtual joystick that Max could then process.

The final step was figuring out how to properly map three virtual joystick inputs to MIDI messages. These inputs are integers from 0-65535, whereas MIDI note bend messages and continuous control messages are values from 0-127. A simple linear mapping would not be practical: first, it is physically painful to move one’s head far enough to reach value 0 or 65536, and second, if the tracking is too sensitive, it becomes impossible to stay in tune and accidental changes are inevitable. After extensive trial, error, and neck stretching, I created several dedicated JavaScript objects in Max to handle these complex mapping rules.

Future Work

Currently, changing hyperparameters in the mapping rule sets requires editing the JavaScript code directly. In the future, I aim to create user-friendly sliders or knobs in the Max patch for easier parameter tuning.

Additionally, combining all the separate software components into a single executable file would significantly improve the user experience.

Finally, I plan to replace the smartphone with a dedicated orientation sensor, making the headset lighter and eliminating the need to strap a phone to one’s forehead.

Video Demo

In this demo, I am playing a Rhodes Piano and an ARP Solina string ensemble. The volume of the ARP Solina is set to 0 at the beginning.

The yaw axis (shaking head “no”) is mapped to pitch bend, creating a vibrato/out-of-tune effect as I turn my head left and right.

The pitch axis (nodding “yes”) is mapped to the volume of the ARP Solina, so the synth string sound fades in and out as I look up and down.

Music written by me

How to Setup Headhunter

Note: This method works only on Windows. The main issue with macOS is the complexity of virtual joystick support, potentially requiring actions like compiling drivers from source and installing kernel extensions. If virtual joystick support is achieved on macOS, Headhunter could work on this platform as well.

Install OpenTrack, Max, Vjoy, and loopMIDI on your PC by following the links provided in the Credit and Download section below.
Follow this link to setup and connect your smartphone to OpenTrack. Make sure OpenTrack responds to your smartphone before proceeding to the next step.
Change the output protocol in OpenTrack to “Virtual Joystick”.
Create a new MIDI channel in loopMIDI to connect Max to your instrument.
Download the Max patch and js files here. Set all ctlout and bendout objects to the MIDI channel you created in Step 4. Select vjoy device as your input, and click poll 10 to start the Max patch.
Set the channel you created in step 4 as the input of your VST. Or simply select “All” as your input.
Start OpenTrack and your smartphone as described in Step 2. You should see numbers changing in your Max patch.
Determine your “zero position” and reset the tracking in OpenTrack.
Now you can start to use Headhunter!