Interfaces 3 – Audiovisual Lutherie

Digital lutherie is the art of designing and building electronic musical instruments and sound-generating tools using digital technology. Like traditional luthiers craft physical instruments, digital lutherie creates musical instruments through software, hardware interfaces, and digital signal processing. It combines programming, sound design, and interface design.

Examples:

• Creating software instruments (virtual synthesizers, samplers)

• Designing hardware controllers and interfaces

• Developing new ways to interact with sound through technology

• Building custom instruments using platforms (Pure Data, Max/MSP, SuperCollider)

• Exploring novel approaches to sound generation and manipulation

Augmented by sensors: orientation sensors / accelerometer: example Terrain Study (see below)

"Smart" Instruments:

FIGURE 1. Pictures of existing smart instruments: (a) the Sensus Smart...ResearchGate

Integrated – wearables as interfaces:

A Touch on Musical Innovation: Exploring Wearables and Their Impact on New Interfaces for Musical ExpressionMDPI

Augmented sound: Microphone with ML-Software that interprets phrases, pitches etc to produce/add sound. Similar approaches can be used for audiovisual instruments.

1 Hand Tracking – Leap Motion

The Ultraleap sensor (former Leap Motion Sensor) is a hardware device designed for hand tracking and motion control. It uses infrared cameras and LEDs to detect and track hand movements and gestures in 3D space. The sensor creates a hemispherical interaction area above the device. It allows to control systems using hand movements without touching any physical controls. Initially targeted at desktop computer control, it was used in virtual reality, augmented reality, robotics, and can of course also be used to design audiovisual instruments.

Applications

Mimu Glover (commercial)

MiMU — Glovermimugloves.com

Geco Multi-dimensional MIDI expression through hand gestures

GECO - Music and sound through hand gesturesuwyn.com

https://www.youtube.com/watch?v=fwn7_Czh5Y0&t=1s

2 Body Tracking, Environment – Kinect and others

3 DIY with sensors: from Augmented Musical Instruments to Audiovisual Instruments

Sensor-enhanced musical instruments utilize sensors to transform player interaction and musical output in innovative ways.

Hardware

• Controller: ESP32 Wemos Lolin32

• Sensor: Bosch BNO055 / Adafruit Breakout

• Battery: LiPo / Li-ion 3.7V

Controller

The ESP32 is a low-cost microcontroller system-on-chip with wireless connectivity options. It has integrated Wi-Fi and Bluetooth, allowing for wireless communication in IoT and embedded applications. The ESP32 can be powered by a battery, including deep sleep which draws only microamps of current, making it suitable for portable and battery-operated devices. Its has also been used widely in arts.

Sensor

BNO055 chip by Bosch.

Smart Sensor BNO055Bosch Sensortec

Board by Adafruit.

Adafruit 9-DOF Absolute Orientation IMU Fusion Breakout - BNO055adafruit

Software

Arduino/ESP32 Library by Adafruit:

GitHub - adafruit/Adafruit_BNO055: Unified sensor driver for the Adafruit BNO055 orientation sensor breakoutGitHub

WIFI messages can be broadcasted by using the UDP protocol.

• UDP is a connectionless protocol, meaning that messages are sent without negotiating a connection and that UDP does not keep track of what it has sent.

• It has no handshaking dialogues and thus exposes the user's program to any unreliability of the underlying network.

• there is no guarantee of delivery, ordering, or duplicate protection.

User Datagram ProtocolWikipedia

// -------------------------------------------
// Esp32 Code to send orientation data
// of the BNO055 sensor via UDP.
// -------------------------------------------

#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BNO055.h>
#include <utility/imumaths.h>

#include <WiFi.h>
#include <WiFiUdp.h>

// I2C Clock Pin: 4
// I2C Data Pin: 0

// WiFi credentials
const char* ssid = "";
const char* password = "";

// UDP settings
const char* udpServerIP = "192.168.0.61";  // Replace with your computer's IP address
const int udpPort = 8888; 
WiFiUDP udp;

unsigned long previousMillis = 0;
const long udpInterval = 5000;   


#define BNO055_SAMPLERATE_DELAY_MS (100) /* Set the delay between fresh samples for the BNO055*/
Adafruit_BNO055 bno = Adafruit_BNO055(55, 0x28, &Wire); //(by default address is 0x29 or 0x28)

void setup(void) {
  Serial.begin(115200);
  Wire.begin(0,4);          // Important! Define the I2C Pins manually

  // wait for serial port to open
   while (!Serial)
    delay(10);              

  // Initialise BNO055 Orientation Sensor 
    if(!bno.begin())        
  {
    Serial.print("BNO055 not found.");
    while(1);
  }
  delay(1000);
  bno.setExtCrystalUse(true);

  // Establish WIFI Connection
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected, IP address: ");
  Serial.println(WiFi.localIP());

  // Establish UDP
  udp.begin(udpPort);
  Serial.print("UDP started on port ");
  Serial.println(udpPort);
}

void loop() {

  /* Get a new sensor event */
  sensors_event_t event;
  bno.getEvent(&event);

  /* Display the floating point data */
  Serial.print("X: ");
  Serial.print(event.orientation.x, 4);
  Serial.print("\tY: ");
  Serial.print(event.orientation.y, 4);
  Serial.print("\tZ: ");
  Serial.print(event.orientation.z, 4);
  Serial.println("");
  
  sendUDPMessage("RotX:"+String(event.orientation.x)+",RotY:"+String(event.orientation.y)+",RotZ:"+String(event.orientation.z));

  unsigned long currentMillis = millis();
  
  // Check WiFi connection
  if (WiFi.status() != WL_CONNECTED) {
    Serial.println("WiFi disconnected. Reconnecting...");
    WiFi.begin(ssid, password);
    return;
  }
  
  // Send message at an interval (currently deactivated)
  if (currentMillis - previousMillis >= udpInterval) {
    previousMillis = currentMillis;
    sendUDPMessage("RotX:"+String(event.orientation.x)+",RotY:"+String(event.orientation.y)+",RotZ:"+String(event.orientation.z));
  }

  /* Wait the specified delay before looping and requesting next data */
  delay(BNO055_SAMPLERATE_DELAY_MS);
}

void sendUDPMessage(String message) {
  udp.beginPacket(udpServerIP, udpPort);
  udp.print(message);
  bool success = udp.endPacket();
  
  if (success) {
    //Serial.println("UDP message sent: " + message);
  } else {
    Serial.println("Failed to send UDP message");
  }
}

Cheaper but less accurate option is e.g. the MPU-6050, 3-axis gyroscope and acceleroscope. https://www.az-delivery.de/en/products/gy-521-6-achsen-gyroskop-und-beschleunigungssensor

TouchDesigner

The network of operators in TouchDesigner needs to process the following program blocks:

• Receive the UDP messages from the sensor

• Process the received data

• Map the processed data to visual processes

• Send the data processed or raw via OSC to sound software (e.g SuperCollider)