Color Based Musical Instrument

I presented my paper “The Development of a Color-based Musical Instrument” at the 2018 IEEE R1 Student Conference. The conference was hosted at the New York Institute of Technology, on Long Island.

The paper is based on my work with Dr. Craver. We wanted to create a musical instrument that allows a user to point at the color of a surface and generate a unique tone. You could make any surface into an instrument or create any instrument you want by printing off different colors of paper.

Figure 1 Color Image with Saturation thresholds

Pictures of the surface are taken with a Raspberry Pi cameras enclosed in a shrowd. The shrowd lights the surface and allows you to select what part of the surface to sample.

Figure 2 HSV Color Space

The RGB color space samples from the camera are converted to the HSV color space. In HSV, the base color “hue” is a single angle from 0-360 degrees. In order to a tone to the color, we only need to estimate what this angle is and then map angles to tones.

Figure 3 Circular Color Distribution

By only capturing pixels where the saturation is above a certain threshold, we can easily both isolate the sample region and eliminate any glare caused by the led lights. A saturation threshold is also used to filter out when the surface isn’t colorful enough to estimate.

Figure 4 88 Key MLE

We use the Von Mises distribution to derive our maximum likelihood estimator.

\[f(x|\mu,k) = \frac{e^{cos(x-\mu)}}{2\pi I_0(k)}\]

Von Mises is a circular normal distribution. The possible colors are broken up into 88 discrete tones, based on the 88 tones on a piano, and MLE is used to estimate which tone to assign the sample.