Audio to MIDI signal

[GusanOrate]

Hi. I'll try to be clear about my problem.

I've got a music visualization project for which I need to control Resolume parameters with some audio files. The idea is to take the separate instrument tracks of a song and use each one to control parameters in different clips that are mixed together. This is so that each instrument looks as a different clip and/or effect.

I'm looking for the easiest and cheapest way to do this What I've done is that I put each sound track as the audio for the different clips. The bass with a background texture, the guitar with a high contrast line figure and the voice with colored dots that revolve. Connecting the opacity of each clip to it's own sound as FFT input, I get a simple sound-visual correspondence effect: when the bass sounds, the texture appears, when the guitar sounds, the line appears, etc, and when they sound together they show together.

While this is okay I would really like to use the separate drum audios (kick, snare and overs) to affect different parameters of effects applied to the whole composition. The easiest way I've figured to do this is to use each audio as a MIDI signal. I've found that you can easily convert audio to midi tracks so what I'm looking for now is a way to play those files and produce a live MIDI signal that I can map to resolume. Any ideas on software that can do that?

This is the way I've found around this project, but I'm open to suggestions on how to achieve this. Hope that was clear enough.

Thanks in advance!

[Arvol]

I built something like this in Touch Designer. Sometime's I get stuck out at FOH house VJ'ing without a break for 12 hours. No food and sometimes no water....

I created a TD patch that takes an incoming audio signal from FOH and breaks it down into 6 audio bands, each can be set to the desired freq. And then I can set an amplitude threshold that sends out a value from 0-100.

I then take those values and merge them into a table, and then create a midi/osc/dmx output that pulls from that table.

This way, I can output about 20 different parameters from a single audio track. So if I walk away for 10 min to get some food or water, The show still looks sort of good haha. When there's no audio, the screens are black and when there is music, I assign those values to speed, FX, and color. So the visuals still sync up to the audio.

If you already have the midi as you converted the audio, you just need something to sync the midi tracks you created to the rest of the set.

This could be achieved in Touch Designer:
Have your midi track loaded in TD.
Have a midi or osc IN device loaded in TD and set it coming from Resolume (outputting midi or osc).
Have a midi or osc OUT loaded into TD and link the midi file and out directly. (be sure to set the proper playback speed and settings)
Link the midi/osc device (coming from Resolume) to the midi file's play button.

Now when you play something in Resolume and you link that same clips button via midi/osc to TD when you click play in Resolume, it should also trigger the midi file in TD. that would then output midi notes right back to resolume.

This could get really clutters really quick. So my recommendations would be to convert the midi notes to osc insides TD.

you could also create a playlist in TD and control that via osc as well so each song can be selected and cued up.

the other thing to keep in mind here is the latency from playing the audio to the midi file getting processed to actual playback time (rather or not it's in sync with the audio at this point)



The project is do-able, just going to take some time to program it all.
Also keep in mind. TD is not free for non-commercial use, so if you plan to make money on this, you would have to purchase a copy of that.

If you plan to go this route, feel free to PM me or continue this topic on the TD forums or FB page


There might be some other options out there as well. This is just the route I typically take since TD is a great melting pot of different protocols and can be easily integrated into my setups.

[KoenTanghe]

Hi,
typing this from my phone on an island with crappy internet access, so i hopd this comes through...
I’m not sure how wel VST2 plugin support is in Resolume, but if you’re on Windows, you could try my KTDrumTrigger plugin for this (Google is -still, at least for now- your friend). You put it on an audio channel and it sends out MIDI events, including an amplitude level if you want that.
You will need to find out how to route MIDI output from a VST2 plugin in Resolume (or ask around). Can’t test it on my PC myself right now, sorry.
Good luck!
Koen

[Arvol]

Hi,
typing this from my phone on an island with crappy internet access, so i hopd this comes through...
I’m not sure how wel VST2 plugin support is in Resolume, but if you’re on Windows, you could try my KTDrumTrigger plugin for this (Google is -still, at least for now- your friend). You put it on an audio channel and it sends out MIDI events, including an amplitude level if you want that.
You will need to find out how to route MIDI output from a VST2 plugin in Resolume (or ask around). Can’t test it on my PC myself right now, sorry.
Good luck!
Koen

Couldn't get it to load in V6, but it loads in V5. I'll try to play with it later tomorrow. Looks promising. The only thing now is getting a live audio source into Resolume.