Recap From Day 042

In day 042, we continued with video features: Optical Flow. We learned that by estimating optical flow between video frames, we can measure the velocities of objects in the video. In general, moving objects that are closer to the camera will display more apparent motion than distant objects that are moving at the same speed.

Today, we will look at how fast to send features.

Features: How fast to send them?

The easiest way to use machine learning to control a real-time system is to use our classifier or regression models to compute a new set of output values for every time we receive a new input feature vector. This means that the faster we send features, the faster our outputs can change, but also the more computation we have to do.

Source

So how fast do we need to send features really depends on what we want to control. If we’re controlling audio for instance, we may find that we need a very high rate of control in order for the audio to not sound choppy. When we’re listening for changes in pitch, our ears and brain tend to be pretty sensitive. We want to update the pitch at a very fast rate.

If we’re updating pitch, a rate of 10 to 20 milliseconds gives us something that we’ll perceive as smooth. If we are updating other audio parameters, things like volume or spatialization, we may find that there’s a different rate at which we start to perceive some choppiness. Likewise if we are controlling animation, depending on what we’re animating, if it’s changing color or position, and depending on the magnitude of the change, we again may need to find the rate at which we need to send updates in order to perceive things as changing smoothly.

If we really need to execute our machine learning more slowly than once every say, 20 milliseconds if we’re controlling pitch, we could alternatively do some smoothing in between these values withing our synthesis program. We need to keep in mind that if we’re building a gestural controller where we’re changing sound or image as we move our body, we don’t just care about how things sound or look. We care about how it feels.

We may find that if something takes longer than say 20 milliseconds or so to respond to our actions, we’ll feel like there’s some delay. Of course the best way to know how fast we need to perform computation in order for our system to behave how we want, is to try it out.

Awesome. That’s all for day 043. I hope you found this informative. Thank you for taking time out of your schedule and allowing me to be your guide on this journey. And until next time, be legendary.

Reference

*https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists-v/sessions/sensors-and-features-generating-useful-inputs-for-machine-learning*