Interfaces 1 – MIDI

As they are widely available and well known, MIDI interfaces can serve as a starting point for exploration by translating physical gestures into synchronized data streams that simultaneously control both sound generators and visual rendering systems.

MIDI CCs can quiete easily be mapped to parameters of sound as well as visual elements. When MIDI is received in TouchDesigner or Processing, the data can be broadcasted via OSC to suitable sound software as SuperCollider, Max, or PureData.

Modern MIDI interfaces often incorporate additional sensor data through MPE (MIDI Polyphonic Expression) or OSC integration, allowing for more nuanced and expressive audiovisual performances that respond to subtle gestural variations.

But also the standard MIDI data can be treated by programming and software for more complex data structures and computations.

Interactive systems with MIDI in Processing

Receiving MIDI in Processing

import themidibus.*; //Import the library that lets us talk to MIDI devices
MidiBus myBus; // Create a variable that will connect to our MIDI controller

void setup(){
  size(400, 400); // Create a window that is 400 pixels wide and 400 pixels tall
  MidiBus.list(); // Show all MIDI devices connected to the computer in the console
  myBus = new MidiBus(this, "MIDI Mix", "MIDI Mix"); // Connect to a device called "MIDI Mix"
  // The first "MIDI Mix" is for input (receiving messages)
  // The second "MIDI Mix" is for output (sending messages)
}

void draw(){
  background(255); // Fill the window with white color (255 means white)
  // This happens 60 times per second to keep the window clear
}

void controllerChange(int channel, int number, int value) {
 // This function runs automatically when you move a fader or dial on the MIDI controller
 println("Fader/dial: channel "+channel+", number "+number+", value "+value); 
 // Print information about which control was moved and its new value
 // channel = which group the control belongs to
 // number = which specific fader or dial was moved
 // value = position of the control (0-127, where 0 is all the way down/left and 127 is all the way up/right)
}

void noteOn(int channel, int pitch, int velocity) {
 // This function runs automatically when you press a button on the MIDI controller
 println("Button on: channel "+channel+", number "+pitch+", velocity "+velocity); 
 // Print information about which button was pressed
 // pitch = which button was pressed
 // velocity = how hard the button was pressed (usually 127 for full press)
}

void noteOff(int channel, int pitch, int velocity) {
 // This function runs automatically when you release a button on the MIDI controller
 println("Button off: channel "+channel+", number "+pitch+", velocity "+velocity); 
 // Print information about which button was released
}

Passing MIDI on via OSC

MIDI data to SuperCollider

Processing

import oscP5.*;
import netP5.*;
import themidibus.*; //Import the midibus library
MidiBus myBus;       // The MidiBus
OscP5 oscP5;          // Declare an instance of the OscP5 class to handle OSC communication (receiving messages).
NetAddress myRemoteLocation; // Define the destination for outgoing OSC messages
int dotX;             // X-coordinate for the visualization dot
int dotY;             // Y-coordinate for the visualization dot

void setup(){
  size(400, 400);                  // Set the size of the window
  MidiBus.list();                  // List all available MIDI devices to the console - useful for debugging connections
  myBus = new MidiBus(this, "MIDI Mix", "MIDI Mix"); // Connect to the MIDI Mix device for both input and output
  oscP5 = new OscP5(this, 12000);  // Initialize oscP5 to listen for incoming OSC messages on port 12000.
  myRemoteLocation = new NetAddress("127.0.0.1", 57120); // Target SuperCollider's language port for OSC communication
  dotX = width/2;     // Initialize dot position to center of the window
  dotY = height/2;    // Using width/height properties avoids hardcoded values
}

void draw(){
  background(255);    // Refresh the canvas each frame to prevent trails
  fill(0);            // Set fill color to black for the dot
  ellipse(dotX, dotY, 10, 10); // Draw the dot at current position with fixed diameter
}

void controllerChange(int channel, int number, int value) {
 println("Fader/dial: channel "+channel+", number "+number+", value "+value); // Log MIDI CC data for debugging
 switch (number){     // Switch based on the controller number rather than multiple if statements
   case 16:           // Controller 16 (likely a specific fader on the MIDI Mix)
     dotX = (int) map(value, 0, 127, 0, width); // Map MIDI range (0-127) to window width, cast to int
     OscMessage oscX = new OscMessage("/pitch");  // Create OSC message with address pattern "/pitch"
     oscX.add(value); // Add the raw MIDI value as the first argument in the OSC message
     oscP5.send(oscX, myRemoteLocation); // Send the OSC message to the specified address/port
     break;
   case 17:           // Controller 17 (another fader/knob on the MIDI Mix)
     dotY = (int) map(value, 0, 127, 0, height); // Map MIDI value to window height
     OscMessage oscY = new OscMessage("/pitch");  // Same address pattern used for Y position
     oscY.add(value); // Use the unmodified MIDI value for the OSC message
     oscP5.send(oscY, myRemoteLocation); // Send to the same destination (SuperCollider)
     break;
 }
}

void noteOn(int channel, int pitch, int velocity) {
 println("Button on: channel "+channel+", number "+pitch+", velocity "+velocity); // Log MIDI note events
 // Function triggered by MIDI note-on messages, currently just logs data without taking action
}

void noteOff(int channel, int pitch, int velocity) {
 println("Button off: channel "+channel+", number "+pitch+", velocity "+velocity); // Log MIDI note-off events
 // Function triggered by MIDI note-off messages, currently unused but available for expansion
}

SuperCollider

Interactive systems with MIDI in TouchDesigner

Process MIDI inputs from the Touché device

Send the processed data to SuperCollider via OSC

Example: AV System

The type of interface and the technology affects how interactions are possible and how performers are able to manipulate digital materials. We can explore this notion by building an audiovisual system that employs two very different interfaces: