A musical instrument that converts colour into sound: The Synaesynth
A simulated synaesthetic instrument — colours, captured live, translated into MIDI tones.
A simulated synaesthetic instrument — colours, captured live, translated into MIDI tones.
Designer and developer. Industrial design, electronics, Arduino programming, software development, user testing. Timber casing designed and built by Elliot Henkel. Original circuitry contributed to by Sonny W. Photography by Philip Pille.
Synaesynth is an experimental instrument that translates the colours of your life into music. It achieves this through a unique blend of software and hardware, creating a simulated synaesthetic experience. This project sits at the intersection of Product Design, Interactive Art Practices, and Human Computer Interaction (HCI), exploring a tangible interface between the digital and physical worlds.
Synaesynth operates by capturing colour data from a video feed and converting it into musical tones. The software presents a grid-divided video feed, matching the hardware's button grid. Pressing a pad captures the colour of the corresponding video segment, which is then algorithmically translated into a MIDI note, which can be sent to a synthesiser.
The device has nine function keys, a 16-button matrix, and an LCD screen, housed in custom casings made from japan black and Fijian mahogany. It runs on 5V DC and communicates with the software via USB.
The project simulates synaesthesia — the neurological condition in which two sensory pathways involuntarily overlap — as a creative instrument for orchestral production, performance, and exploration.
Initial software prototypes were developed in Python, GML, and Processing, with Processing ultimately chosen for its user-friendliness. Research also included exploring mobile Raspberry Pi setups and web-based controls. User surveys conducted on Reddit helped refine the colour-sound conversion rules, with a focus on expectations from musicians and laypersons alike.
User testing with a hardware prototype was undertaken with bedroom DJs, beat-makers, and producers. Filmed sessions, screen captures and audio recordings, along with interviews and participant sketches, informed key feature developments.
Synaesynth contributes to generative art and electronic music, enhancing discussions on augmented reality and emerging media. It allows artists to create context-responsive works, and in live performances, it can create an audio-visual feedback loop. The project aims to inspire reflections on cross-sensory perception and the modern challenge of sensory overload.
This project was made possible with help from Elliot Henkel, Philip Pille and Sonny W.
