Further group project developments
It has been a good week. We’ve made some great progress on our group project, and I’ve been able to contribute a lot more than I thought. Here’s what I’ve been up to:
Giving the AI letter detector a purpose
Like I considered in the previous post, I made a Rand-O-Drummer that listens to the letters being sent from the AI letter detector and uses them as a seed for generating loops. My code was only designed to send out what it sees back to the website where you draw letters, so I had to make some changes to also send it via OSC to the drummer.
I’ll spare you the eyesore of the abhorrent back-end code I wrote to make that happen, but here’s how the Max patch looks:
Once that was done, the little ecosystem surrounding the letter detector was essentially complete.
Guidonian Hand
To make things more interesting and performative, we wanted my mouse movements as I drew letters to make sounds too. One of our group members showed us a program called HandPose-OSC that tracks the user’s hand using the webcam and serves the data via OSC. I figured we could make hands the general theme of the piece. After some thinking, another group member had taken inspiration from the Guidonian Hand, a medieval way to teach music theory by associating notes with parts of the hand.
Image credit: Wikipedia
I wrote a Python script that sends the x and y coordinates of my mouse to a Max patch he wrote via OSC, and that then plays a note on a piano based on where the mouse is on the hand. It also sends the speed of the mouse movement (how many pixels it has moved since the last frame), which affects the velocity of the note played.
You can run this yourself, but you’ll first need to install the pyautogui and python-osc libraries.
import pyautogui
import time
from pythonosc import udp_client
ip = "10.0.0.100"
port = 5005
client = udp_client.SimpleUDPClient(ip, port)
x, y = pyautogui.position()
while True:
new_x, new_y = pyautogui.position()
dx, dy = new_x - x, new_y - y # Delta X and Y since last frame
distance = (dx ** 2 + dy ** 2) ** 0.5 # Pythagorean theorem
client.send_message("/mouse", [new_x, new_y, distance])
x, y = new_x, new_y
time.sleep(0.05) # 20 frames per second
Final thoughts
That covers pretty much everything that I’ve contributed to the project. We will be doing some final testing and figuring out how to display it all in a performance setting. When we are done, I’ll write one final post on this explaining some of the other group members’ contributions, and how it all came together. I’m excited to see the final result!