Due Thursday, Oct 17

1. Develop a p5.js sketch with ml5.js, using one or more of the models we’ve covered in the first half of the semester. Here are some ideas to get you started:
   • Create a simple movement-controlled game using hand, body, or face keypoints. Think about how different gestures might influence gameplay or character actions.
   • Design a musical instrument that responds to hand, body, or face keypoints, allowing gestures to control sound parameters such as pitch, volume, or effects. Explore creating melodies or soundscapes.
   • Build an augmented mirror or photo booth by overlaying p5.js elements on real-time video. Use segmentation or keypoint data to add visuals that respond to movements.
   • Make an interactive painting tool controlled by hand, body, or face keypoints. Experiment with effects like trailing particles or dynamic brushes that change shape and color based on gestures.
   • Chain models by using the output of one model as input to another with ml5.neuralNetwork(). For example, capture body or hand poses to create a custom classifier that drives an interaction. How could regression be applied to modify continuous elements like color gradients or speed?
   • Use transfer learning with Teachable Machine to train a custom image classifier on objects from your surroundings. Experiment with using these classifications to trigger interactions, visuals, or sounds in your sketch.
2. Document your project in a blog post and come prepared to share your work for ~5 minutes. You are not required to have a slide deck, however, you might find slides useful to help you plan and structure your demo. Add a link to your documentation to the ml5.js Project Step 2 Wiki page.

• [x] Consider including:
  • [x] Title
  • [x] Sketch link
  • [x] One sentence description
  • [x] Summary (250-500 words)
  • [x] Visual documentation such as a recorded screen capture / video / GIFs.
  • [x] Inspiration: How did you become interested in this idea? Quotes, photographs, products, projects, people, music, political events, social ills, etc.
  • [x] Process: How did you make this? What did you struggle with? What were you able to implement easily and what was difficult?
  • [x] Code references: What examples, tutorials, references did you use to create the project? (You must cite the source of any code you use in code comments. Please note the following additional expectations and guidelines at the bottom of this page.)
  • [x] Next steps: If you had more time to work on this, what would you do next?

Tittle:

Heart Pose <3 (Gesture-Driven Interactive Features)

Sketch Link:

https://editor.p5js.org/TamaraBueno/sketches/voipADOxz

One Sentence Description:

An interactive p5.js sketch using ml5.js Hand Pose to recognize hand gestures and trigger visual elements such as particles and dynamic hearts.

Summary:

This project explores the intersection of gesture recognition and interactive visuals using p5.js and ml5.js. The aim was to create an engaging experience where hand poses control visual feedback. Utilizing ml5's Hand Pose model, I trained the system to recognize specific hand shapes, such as a small heart pose, which triggered corresponding visual effects. The first recognized gesture activates particle trails, while the second gesture generates a dynamic heart shape that grows in response to the gesture.

The core challenge was implementing a mechanism where visual elements were triggered only when a specific hand pose was detected. This was achieved by using conditional statements and flag variables. For example, particles are only displayed when the system recognizes the "small heart" pose. A combination of online tutorials and trial-and-error helped me integrate the heart shape and particle effects.

The final component of the project was adding an introductory instruction screen using mode-switching in p5.js. The user is guided to press "Enter" to start the interactive experience. This intuitive design ensures a smooth user experience, allowing them to interact with the sketch seamlessly.

While the basic functionality is in place, there are opportunities for expanding the interaction possibilities, such as adding additional gestures and more complex visual responses.

Visual Documentation + Inspiration + Processes:

Inspiration:

I became interested in this project by exploring ways to use gesture-based interaction as a creative tool. The idea of using hand gestures to control visuals, like creating a heart shape with one's hands, resonated with me because it’s a universal symbol of affection and expression. Watching interactive visualizations and the tactile nature of drawing apps inspired me to think about how physical gestures can influence art in digital environments. Using ml5 Hand pose, create a program that gathers and recognizes poses and releases certain interactive features based on trained result.