Welcome again to the course on Audio Signal Processing for Music Applications. In the last lecture we talked about stochastic signals and how to model them. With that we can now extend the sinusoidal and harmonic models that we talked about before, into the concept of sinusoidal or harmonic plus residual or stochastic modelling. We will decompose sounds into these two parts. The sinusoidal or harmonic one and the residual or ideally a stochastical one if this residual is the stochastic signal. So in this lecture, we will be combining all the models we have been talking about until now. We'll put together the sinusoidal or harmonic modelling with the idea of residual component. For that we will need to talk about the subtraction of the sinusoidal, the harmonics in order to obtain the residual. And we will talk about a system that puts these together into the harmonic plus residual system. Then we will introduce the stochastic model and we'll put together the sinusoidal and or harmonic models with the stochastic ones for the residual. So in order to do that, we'll need to talk about how to model the residual as in a stochastic component. And finally we will make an example of this system that combines the harmonic plus the stochastic analysis into an analysis synthesis system. The sinusoidal plus residual model is an extension of the sinusoidal model. We assume that a sinusoidal model is not able to represent the whole sound and that there is residual component that is perceptually relevant. So it is a sound that needs to be kept. So in here we see in this equation that y, the signal, is modeled with these sum of sinusoids time bearing sinusoids with amplitude and frequency. And we have xr which is our residual. So the whole sound is ys, our sinusoidal component, and xr, our residual signal. And xr is simply computed by subtracting the sinusoids from the original signal. And this is what we'll be talking about in this lecture. Okay, there is much better, at least from our point of view, to show this model from the spectral point of view, because that's where we're going to be developing all these models from. So here we see the sum of the sinusoids as the sum of the transform of the windows shifted to a frequency and scale to the amplitudes of these sinusoids plus the spectrum of this residual component. And the spectrum of this residual component and now, this is how we're going to do it. It's obtained by subtracting the spectrum of the sinusoids from the spectrum of the original signal, okay? So let's show an example exactly how these will work. This is a one frame of a sound and here we can show the different steps involved these harmonic plus a residual analysis. On the left top, we see our window frame of a flute sound, okay? So it's just few periods of a flute sound. And then below that, we see the harmonic analysis that we do from the spectrum. So we do the spectral analysis the peaks, and we select the peaks with these blue crosses that are really the harmonics of that particular sound. And below it we see the actual phase of the spectrum with the crosses and the phase of these harmonics, okay? And then what we do is we have to synthesize these harmonics. [COUGH] And this is what we see on the right side with the light red and light cyan color. So the light red is the synthesized harmonics of the sound of that particular frame. Of course this is a different FFT size, the shapes of these lobes is different because the window is different. This is a window using the synthesis. So this is the synthesized spectrum and then we have to subtract these spectrum from the original spectrum. Strictly speaking, we don't subtract it from the spectrum on the left, we subtract this from another generated spectrum that has the same parameters so that we can subtract the two of the same size and the same window size. And then it will subtract this synthesize sinusoids or harmonics from the original one. We get this dark red and dark cyan color. Okay, and this is the residual spectrum in magnitude and phase representation. And if we take the inverse of that, we see the residual signal in the tandem and that's what we see on the top-right plot. In which we see the original flute sound of course with the right windowing and the right size that we have in the synthesis. And we see the residual signal, this dark blue one. And again, this is not just an error signal, this in fact is a relevant component, it's a relevant part of the sound that we want to recover. So the whole system, if we put together all this analysis in a frame by frame type of thing and put it together into a whole analysis synthesis system. We get this block diagram in which we start from the signal x[n], then we window it, we compute FFT, obtaining the magnitude and phase spectrum, we detect the peaks. Hold up those peaks, we find the fundamental frequency, and once we have this fundamental frequency we can identify the harmonics of that sound. And we can synthesize those harmonics with the window. Okay, so we have another spectrum, Yh, that can be subtracted from the original signal. But in order to do that, we need to recompute another spectrum of the original with a window and a size that is comparable with the size that we use in the synthesis. So we will choose a window size that normally will be 512 samples, we'll use a window so that the shape of these X[k] that we now compute can be easily subtracted From Yh. So it's just a complex subtraction and we get capital Xr, which is our residual spectrum, okay? And then this residual spectrum can be added to the harmonic spectrum. And then, we can compute the inverse FFT and do the Overlap-add iterating over the whole sum. We can see an example of the analysis of a particular sound using the harmonic class residual model. So here, we took the flute sound that we have heard before, and so on top, we see the spectrogram of these flute sound and superpose we see the harmonics that have been obtained. So let's listen to these harmonic synthesis. [SOUND] Okay, and then these harmonics are subtracted from this background spectrogram and we obtain these lower sort of a plot which are the spectrogram of the residual component. So let's just now listen to this residual that has been obtained. [NOISE] Okay, it's soft but it's clearly very relevant. It's basically the breath noise of the instrument which is an important part of the characteristics of the sound. But this residual component is a complete sound. Therefor, it's difficult to process. We need to model it. So we need to be able to approximate it with a particular modeling approach. And of course, we will need to model it with the stochastic model. So let's introduce the stochastic approximation of these residual components. And let's present this idea of the sinusoidal plus stochastic model. So very similar to what we saw before. So we have the signal to be the sum of sinusoids plus the stochastic signal. Now this is stochastic signal or stochastic component is not just the subtraction of the sinusoids minus the original signal but it's actually the result of a modeling approach. So, below here, we see the equation of the modeling of this stochastic component. The stochastic component is the result of filtering wide noise with the impulse response of the approximation of these residual signal. So we have the impulse response of every frame of this residual signal, and we obtain this impulse response that approximates a spectral shape of that. So in fact, it's much better to visualize this model in the frequency domain. And so here we see on the top, the equation of the sum of the sinusoid, the sum of the analysis windows plus the spectrum of the stochastic component. And now this stochastic component is this idea of a filter white noise but in the frequency domain, is the product of the approximation of the absolute value of the residual signal multiplied by e to the j and the phase of the random, the white noise set of random numbers, okay? So, the magnitude spectrum is the approximation of the residual and the phase spectrum is the white noise. Basically, the phase spectrum of the white noise, this is the concept of the stochastic approximation that we saw in the previous lecture, okay? So, with these, we can actually see how in a single spectrum, we actually perform this stochastic approximation of the residual. So we start on the top with the spectrum of a signal, the harmonics. And then below the light red is the synthesized spectrum, the mYh. And then this is subtracted from the original spectrum, again it's a spectrum that will have to be recomputed. And then we obtained the next curve which is the mXr which is the residual spectrum, okay? And this residual spectrum can be approximated with a smooth curve which is the mYst, which is this sort of line approximation of this residual. And this is going to be our stochastic model. So, we can put it together into an analysis synthesis system and it's very similar to what we saw before. So we start from the signal. We compute FFT, we find the peaks, we find the harmonics, we synthesize them in the frequency domain and we subtract them from another spectrum of the original signal recomputed to be able to subtract it. And then what is new in this model is the stochastic approximation of the residual. So we take this residual spectrum, we run it through the stochastic approximation, approximation module and then we can synthesize. And we can synthesize the stochastical by, basically, the idea is filtering white noise, but in the implementation is basically taking the phases of random numbers and applying the stochastic approximation magnitude spectrum. And this results into a spectrum that then can be sum to the spectrum of the harmonics and perform exactly the same thing that we have done before. So, perform the inverse of FFT and the Overlap-add. So let's now see an example of a complete analysis synthesis of a particular sound. So we're taking this saxophone sound, let's listen to that. [MUSIC] Okay, and then below it we have the two representations that we have obtain the harmonics and the stochastic component, the spectragram of the stochastic component. Let's listen to the harmonics. [MUSIC] Now we may not appreciate what is missing but when we listen to this stochastic approximation, [NOISE] we have to make it a little bit louder in order to actually listen what is going on. Well, with these two components, we basically have analyzed and modeled the original signal and we can put them together and generate this synthesized sound, [MUSIC] That captures most of what is perceptually relevant in the sound. So, for these topics that I discuss this lecture, there is not that many references in terms of tutorial or sort of more introductory material. But, there is quite a bit of articles that have been proposing different strategies to analyze sinusoids, obtain residuals, approximate the residuals, etc. So in this link that I put here on the website of the MTG, I have kept some articles, well quite a bit of articles that have been published related to these issues. So feel free to go there and you can sort of find those articles. And that's all basically in this lecture. We have covered the most advanced models that we will be presenting in this course. We basically combine all the previous models, developing a variety of analysis and synthesis techniques that can be applied to many sounds and for many applications. In the next lecture, we'll focus on how these models can be used to transform sounds. So I think we're going to start having fun in doing some interesting new sounds. So I hope to see you then, bye bye.
