FFT in FFGL

[gabor_papp]

Hi,

I'm trying to explore how I can get FFT data into FFGL plugins. I'd like my plugins to respond audio in a rich way, so a 256-band FFT would be nice. These are my ideas, although none of them are perfect:

1. using the FFT from Resolume, by limiting it to distinct ranges and sending it to separate parameters. This won't work for high number of bands, but easy to achieve.
2. writing an FFT plugin that calculates FFT using the audio input and renders this into a texture, which other plugins can use. Each band would be represented by one pixel or one column of pixels, where the intensity relates to the energy level of the FFT band.
3. sending the FFT texture similar to above using Syphon from an application that calculates the FFT itself.

Is there a way to get the FFT data from Resolume to the plugin somehow? It would be less computationally expensive than calculating the FFT again each frame. Getting the FFT as a texture would be a nice feature. For example. Shadertoy (https://www.shadertoy.com/) uses the same technique, so shaders can obtain FFT through a texture sampler.

I would be grateful for suggestions, ideas. Thanks.

-Gabor

[Sadler]

According to Alex May:
viewtopic.php?f=14&t=10354

Afraid there is no way to do this inside a FreeFrame plugin, so afraid not...

This was specifically for getting audio from the Resolume composition. But Alex obviously succeeded in getting audio into his FFGLs from the default audio input.

[Joris]

The texture approach certainly has a lot of appeal. It totally makes sense when your workflow is completely based on shaders, like Shadertoy.

Our feeling is more towards adding a specific parameter type to the FFGL spec that contains the 256 bands as an array of floats. This way it's still open to use it both for CPU and GPU based effects. It has the added UI advantage that it can also be used for passing composition and clip FFT.

We've been playing around with the idea before, but never had the time to properly implement it. If you're interested in speccing such a parameter type, we can then push it to be part of the FFGL spec. Feel free to drop a line over to mail@resolume.com to talk about it.

[gabor_papp]

Thanks for the suggestions! Extending the FFGL specification is probably the best idea indeed.

Adding a new parameter type for the FFT is a bit too specific. Maybe it would be better to add array support similarly to the vvvv extended FreeFrame specification (http://vvvv.org/documentation/freeframe ... cification). This would have the advantage that multiple parameters could receive different FFT's as you mentioned.

The other option would be adding a new function like setTime called setFFT for example. This is a simpler extension, although the plugin could work only with one kind of FFT data at a time, which is still a big step ahead for FFGL I think.

[j4s0n.c]

...Is there a way to get the FFT data from Resolume to the plugin somehow? It would be less computationally expensive than calculating the FFT again each frame...

I would suggest checking out Magic Music Visuals to run alongside Resolume via spout/syphon. It does FFT, and runs shaders, and 3D models, and...

Otherwise:

1) Put any effect on a clip or the comp, one with enough parameters to do what you want. Bypass it.
--If you want to use the same parameters from an effect your are actually using in Resolume, don't bypass, just use that.
2) Receive OSC in PureData or whatever thing like that you know. Convert to MIDI.
--Enable OSC mapping mode in Resolume to see what you need to route.
3) Use loopMIDI/MIDI Yoke/Copperlan (Win) or IAC core MIDI (OSX) to send that MIDI to whatever your generating with.
4) I use this to get the beat sync'd looping out of Resolume, but It'd probably work for FFT too...