Interfaces 3 – Audiovisual Lutherie
Digital lutherie is the art of designing and building electronic musical instruments and sound-generating tools using digital technology. Like traditional luthiers craft physical instruments, digital lutherie creates musical instruments through software, hardware interfaces, and digital signal processing. It combines programming, sound design, and interface design.
Examples:
Creating software instruments (virtual synthesizers, samplers)
Designing hardware controllers and interfaces
Developing new ways to interact with sound through technology
Building custom instruments using platforms (Pure Data, Max/MSP, SuperCollider)
Exploring novel approaches to sound generation and manipulation
Augmented by sensors: orientation sensors / accelerometer: example Terrain Study (see below)
"Smart" Instruments:
Integrated – wearables as interfaces:
Augmented sound: Microphone with ML-Software that interprets phrases, pitches etc to produce/add sound. Similar approaches can be used for audiovisual instruments.
1 Hand Tracking – Leap Motion
The Ultraleap sensor (former Leap Motion Sensor) is a hardware device designed for hand tracking and motion control. It uses infrared cameras and LEDs to detect and track hand movements and gestures in 3D space. The sensor creates a hemispherical interaction area above the device. It allows to control systems using hand movements without touching any physical controls. Initially targeted at desktop computer control, it was used in virtual reality, augmented reality, robotics, and can of course also be used to design audiovisual instruments.
Applications
Mimu Glover (commercial)
Geco Multi-dimensional MIDI expression through hand gestures
https://www.youtube.com/watch?v=fwn7_Czh5Y0&t=1s
2 Body Tracking, Environment – Kinect and others
3 DIY with sensors: from Augmented Musical Instruments to Audiovisual Instruments
Sensor-enhanced musical instruments utilize sensors to transform player interaction and musical output in innovative ways.
Hardware
Controller: ESP32 Wemos Lolin32
Sensor: Bosch BNO055 / Adafruit Breakout
Battery: LiPo / Li-ion 3.7V
Controller
The ESP32 is a low-cost microcontroller system-on-chip with wireless connectivity options. It has integrated Wi-Fi and Bluetooth, allowing for wireless communication in IoT and embedded applications. The ESP32 can be powered by a battery, including deep sleep which draws only microamps of current, making it suitable for portable and battery-operated devices. Its has also been used widely in arts.
Sensor
BNO055 chip by Bosch.
Board by Adafruit.
Software
Arduino/ESP32 Library by Adafruit:
WIFI messages can be broadcasted by using the UDP protocol.
UDP is a connectionless protocol, meaning that messages are sent without negotiating a connection and that UDP does not keep track of what it has sent.
It has no handshaking dialogues and thus exposes the user's program to any unreliability of the underlying network.
there is no guarantee of delivery, ordering, or duplicate protection.
// -------------------------------------------
// Esp32 Code to send orientation data
// of the BNO055 sensor via UDP.
// -------------------------------------------
#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BNO055.h>
#include <utility/imumaths.h>
#include <WiFi.h>
#include <WiFiUdp.h>
// I2C Clock Pin: 4
// I2C Data Pin: 0
// WiFi credentials
const char* ssid = "";
const char* password = "";
// UDP settings
const char* udpServerIP = "192.168.0.61"; // Replace with your computer's IP address
const int udpPort = 8888;
WiFiUDP udp;
unsigned long previousMillis = 0;
const long udpInterval = 5000;
#define BNO055_SAMPLERATE_DELAY_MS (100) /* Set the delay between fresh samples for the BNO055*/
Adafruit_BNO055 bno = Adafruit_BNO055(55, 0x28, &Wire); //(by default address is 0x29 or 0x28)
void setup(void) {
Serial.begin(115200);
Wire.begin(0,4); // Important! Define the I2C Pins manually
// wait for serial port to open
while (!Serial)
delay(10);
// Initialise BNO055 Orientation Sensor
if(!bno.begin())
{
Serial.print("BNO055 not found.");
while(1);
}
delay(1000);
bno.setExtCrystalUse(true);
// Establish WIFI Connection
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("WiFi connected, IP address: ");
Serial.println(WiFi.localIP());
// Establish UDP
udp.begin(udpPort);
Serial.print("UDP started on port ");
Serial.println(udpPort);
}
void loop() {
/* Get a new sensor event */
sensors_event_t event;
bno.getEvent(&event);
/* Display the floating point data */
Serial.print("X: ");
Serial.print(event.orientation.x, 4);
Serial.print("\tY: ");
Serial.print(event.orientation.y, 4);
Serial.print("\tZ: ");
Serial.print(event.orientation.z, 4);
Serial.println("");
sendUDPMessage("RotX:"+String(event.orientation.x)+",RotY:"+String(event.orientation.y)+",RotZ:"+String(event.orientation.z));
unsigned long currentMillis = millis();
// Check WiFi connection
if (WiFi.status() != WL_CONNECTED) {
Serial.println("WiFi disconnected. Reconnecting...");
WiFi.begin(ssid, password);
return;
}
// Send message at an interval (currently deactivated)
if (currentMillis - previousMillis >= udpInterval) {
previousMillis = currentMillis;
sendUDPMessage("RotX:"+String(event.orientation.x)+",RotY:"+String(event.orientation.y)+",RotZ:"+String(event.orientation.z));
}
/* Wait the specified delay before looping and requesting next data */
delay(BNO055_SAMPLERATE_DELAY_MS);
}
void sendUDPMessage(String message) {
udp.beginPacket(udpServerIP, udpPort);
udp.print(message);
bool success = udp.endPacket();
if (success) {
//Serial.println("UDP message sent: " + message);
} else {
Serial.println("Failed to send UDP message");
}
}Cheaper but less accurate option is e.g. the MPU-6050, 3-axis gyroscope and acceleroscope. https://www.az-delivery.de/en/products/gy-521-6-achsen-gyroskop-und-beschleunigungssensor
TouchDesigner
The network of operators in TouchDesigner needs to process the following program blocks:
Receive the UDP messages from the sensor
Process the received data
Map the processed data to visual processes
Send the data processed or raw via OSC to sound software (e.g SuperCollider)
Last updated