Digital lutherie is the art of designing and building electronic musical instruments and sound-generating tools using digital technology. Like traditional luthiers craft physical instruments, digital lutherie creates musical instruments through software, hardware interfaces, and digital signal processing. It combines programming, sound design, and interface design.
Augmented sound: Microphone with ML-Software that interprets phrases, pitches etc to produce/add sound. Similar approaches can be used for audiovisual instruments.
1 Hand Tracking – Leap Motion
The Ultraleap sensor (former Leap Motion Sensor) is a hardware device designed for hand tracking and motion control. It uses infrared cameras and LEDs to detect and track hand movements and gestures in 3D space. The sensor creates a hemispherical interaction area above the device. It allows to control systems using hand movements without touching any physical controls. Initially targeted at desktop computer control, it was used in virtual reality, augmented reality, robotics, and can of course also be used to design audiovisual instruments.
3 DIY with sensors: from Augmented Musical Instruments to Audiovisual Instruments
Sensor-enhanced musical instruments utilize sensors to transform player interaction and musical output in innovative ways.
Hardware
Controller: ESP32 Wemos Lolin32
Sensor: Bosch BNO055 / Adafruit Breakout
Battery: LiPo / Li-ion 3.7V
Controller
The ESP32 is a low-cost microcontroller system-on-chip with wireless connectivity options. It has integrated Wi-Fi and Bluetooth, allowing for wireless communication in IoT and embedded applications. The ESP32 can be powered by a battery, including deep sleep which draws only microamps of current, making it suitable for portable and battery-operated devices. Its has also been used widely in arts.
WIFI messages can be broadcasted by using the UDP protocol.
UDP is a connectionless protocol, meaning that messages are sent without negotiating a connection and that UDP does not keep track of what it has sent.
It has no handshaking dialogues and thus exposes the user's program to any unreliability of the underlying network.
there is no guarantee of delivery, ordering, or duplicate protection.