Analyzing and Manipulating Music Values

The most essential thing in AVS is to get the values of the music and setup the presets response to the music. Superscope, DynamicMovement, Texer, etc. offer almost unlimited space for coding mostly used for amazing effects but what about the music beyond simple beat and getosc/getspec?

I searched this forum for some sophisticated methods to analyze the music input but it is difficult to find and most of the time hidden in general coding discussions.

So I think we need a special thread on this particular theme collecting techniques for setting up music response.

Want to start with a try to react to the loudness of the music using a floating average of the music values over a certain amount of frames. Goal was to setup a var "speed" for example to be used for a rotation increasing and decreasing with the music loudness.

fdelta = number of frames used for floating average
bufstart = number of the first megabuf to store the values
speedf = factor to adjust the average found to your needs


INIT:
speed=0;fdelta=50;bufstart=50;speedf=1.0;

FRAME:
pt=bufstart-1;
loop(fdelta,exec2(assign(pt,pt+1),assign(megabuf(pt),megabuf(pt+1))));
assign(megabuf(bufstart+fdelta),getspec(0.05,0.9,0));
pt=bufstart;
vsum=0;
loop(fdelta,exec2(assign(pt,pt+1),assign(vsum,vsum+megabuf(pt))));
vsum=vsum/fdelta;
speed=vsum*speedf;

Interesting stuff using a buffer array to store the 'average'.

If feeling brave enter the world of beat detection and other methods in this thread.



A bit MIlkDropy at times but we have a few other devolpers in there too from other plugins.

Well i mostly used this very simple method in Extra Dimension, which has it's flaws, but seems to work ok most of the time.

per frame

now=gettime(0) ; sec=now-last ; fps=1/sec ; last=now ;

vol=vol*0.95+getspec(0.5,1,0) ;

speed=(min(3,vol)/fps)*0.1 ;

ts=ts*(1-sec) ; t=t+ts



per beat

ts=(rand(51)-25)*speed

Well first there is the whole speed fixed on fps line,
then it tries to figure out the current volume. The use of min function there is to prevent some sudden sound peaks that might make the dm/ssc/texer II/etc. spin extremely fast, i have tried to figure out the most common values & top values of the volume with the help of the debugging window and noticed that with this method the value of "vol" hardly goes beyond 3.

Well anyways, i know this is probably not what you mented or wanted, but this is currently working for me quite nicely.

I would gladly hear also what other people use for sound reaction in their presets 👍

Tried your stuff tuggummi and it's working allright but the response is a little bit jumpy. The advantage of using a floating average (do not know if this is the correct english term for German "gleitender Durchschnitt") is that we get a flowing increase and decrease of the speed and we do not have to mind about short jumps in the volume. Also it is more "exact" since "vsum" will never become bigger than 1.

But I like the idea to have an eye on the fps. I think this could/should be implemented in my approach.

Ok, here is the code for the floating average based on time/fps.

timebase = time in seconds for building the average
bufstart = first megabuffer to store the values
vsum = the average

The attached example shows 3 Graphs:
1. red = original values
2. yellow = values on a timebase of 0.2 seconds
3. green = values on a timebase of 2.0 seconds

What is interesting here is that Graph 2 and 3 nicely show the beats. Perhaps this could be used to build a custom beat detection.


CODE___________________________________________

INIT:
timebase=2.0;bufstart=50;

FRAME:
now=gettime(0);getit=above(now,timeout);
fps=if(getit,fct,fps);reg10=fps;
fct=if(getit,0,fct+1);timeout=if(getit,now+1.0,timeout);
fdelta=timebase*(fps+1);
pt=bufstart-1;
loop(fdelta,exec2(assign(pt,pt+1),assign(megabuf(pt),megabuf(pt+1))));
assign(megabuf(bufstart+fdelta),getspec(0.05,0.9,0));
pt=bufstart;
vsum=0;
loop(fdelta,exec2(assign(pt,pt+1),assign(vsum,vsum+megabuf(pt))));
vsum=vsum/fdelta;
speed=vsum*speedf;

Actually you can keep an average value much more efficiently 😉. Every frame, just add the new value to an accumulator, and subtract the oldest value from the accumulator.

Originally posted by UnConeD
Actually you can keep an average value much more efficiently 😉. Every frame, just add the new value to an accumulator, and subtract the oldest value from the accumulator.

I thought this is exactly what I am doing here.Is there a better way to manage the values?

i use the same method here, altho i dont think there is a better way....maybe....

"I thought this is exactly what I am doing here.Is there a better way to manage the values?"

As far as I can see, you're adding them together every frame, and even shifting them around all the time, rather than just overwriting the oldest value with the newest.

Originally posted by UnConeD
As far as I can see, you're adding them together every frame, and even shifting them around all the time, rather than just overwriting the oldest value with the newest.

Ok, I agree the sum could be done without a loop like this:


vsum1=vsum1-megabuf(bufstart)+ megabuf(bufstart+fdelta);
vsum=vsum1/fdelta;

but when testing this I dicovered some little jumps in the average. I think because of the changing fdelta since this is related to timebase > fps which changes a little over time. So I have to average the fps too.

Still can't figure out how to dynamically keep and retrieve the oldest value without this loop 🙂

how about this method:

val=100;

fc1=(fc1+1)%val;fc2=val-fc1;
assign( megabuf(fc1), getspec(.1,.1,0));
tot=tot+megabuf(fc1)-megabuf(fc2);
avg=tot/val;

example preset:

Yeah thats very nice, synth, but there is a small bug. I do not think the results are 100% correct since you do not always subtract the oldest value. fc1 and fc2 are "getting close" ah... do not how to say this in english... like this example:

fc1: 48, 49, 50, 51, 52 ...
fc2: 52, 51, 50, 49, 48 ...

but I think this will fix the thing

fc1=(fc1+1)%val;fc2=(fc1+1)%val;

My first own beat detection

Thanx to synth-c for this great setup to build the average. Without this little bug it is working great.

Just added my fps detection to make the average run over time.

Rovastars hint on this beat discussion made me start a desperate try to build up a my own beat detection. Attached you'll find the result. Not too bad for a first try but of course still some things unsolved. The setup does no beat prediction at all since this is very dangerous ground. If prediction fails - a fact in all songs with changing bpm or difficult beats not detected by the setup - you'll get beats at unwanted points. So I think better rely on the pure input.

Difficult beats for this setup are for example a soft base drum behind a loud guitar solo. Don't know how to filter these out. Good prediction would help here but again: if it fails I think it is better to get no beat at all...

The graphs in the attached example show:

green => the average over time
yellow => the original input value
red => mybeat
white => beat found by AVS


The things behind this beat detection:

0. Getting the input value
1. Building a running average of the input value over a given amount of time
2. Comparing the input value with the average and stating a beat if certain conditions are true


Conditions/Assumptions:

- The input has to be a certain amount over the average to indicate a beat

- The next beat is only possible if the input has returned to/below the average

- The amount of the input to be over the average is intially set but must be dynamic too to suppress too fast beats and allow beats if no beats are found over a certain time

Dynamic adjustment of the amount of the input to be over the average

- To prevent too fast beats after a beat the next beat is suppressed by a linear decreasing additional dumping value over a short amount of time


- To get beats if no beats are found after a certain time the intial dumping of the input is lowered on the increasing time after the last beat until a beat is found.

Here is a new version of my beat detection:

- Used different input "mval" to better detect bass beats

- v_dump adjusts dynamically on volume

hmm this has some problems with the trance music i'm listening to...
doesnt detect some of the beats

and in one song it doesnt detect beats at all (Usual Aspect - Mr Blue(Progressive Mix) that is)
maybe you should only watch the lower bands for beats...

Originally posted by TomyLobo
maybe you should only watch the lower bands for beats...

Yeah I know it is not perfect yet. Works well with a lot of songs but these lower beats are difficult. In mybeat03 I tried to concentrate on the lower bands but still not 100% OK. Perhaps check out lowering v_damp to almost zero.

A lot depends on the bands used for input. So I will do more experimenting with "mval".

What I wanted to get is a less hyperactive beat detection than the one AVS offers since in visualizations too fast beats sometimes ruin the effects. And skipping beats using custombeat setup will fail to get some of the essential beats in a song.

one could skip fast beats, like colormap does
that is beats that are less than 0.1 sec from each other or something