Welcome back to the course single processing from music applications. This is the transformation week, the week that we talk about the potential of the spectral modeling approaches we have been discussing in the past few weeks for the application of manipulating sounds of transforming different aspects of sounds. And in these programming lectures, I want to go over some of the code of the SMS tools package that implement these transformations. So in this lecture, I want to explain the stft transformations. So the file that implements some very basic transformations using the stft. So we'll be basically implementing this block diagram, in which after the analysis of a sound in a particular frame where we have the magnet and the phase of a sound. We can apply some transformations to these spectral data before we actually do the inverse for you transform and re-synthesize basically a new sound. So let's go to a script that I wrote that makes a very simple operation on one single DFT of a sound, on one frame, so we'll be doing some filtering to one particular frame. Okay, so we will be using this sound, the oboe sound and after importing all the models and packages that we need we read the sound file, the oboe sound file. Into the array X, then we get the window, hanning window with a given size. Over here we have the same size for the window and the one that we'll be using for the FFT, n. Okay, and then we just take one fragment of the oboe sound, 512 samples, the size that we defined, okay, and then also defined a filter, okay? And the concept of defining a filter In the frequency domain is basically defining a shape in db scale, because we're going to be applying in the db scale that will be added to the spectrum of the residual signal. So it has to be a smooth shape, we don't want to have very strong discontinuities that will create distortions, so in fact what we are defining here is a filter that is in fact the shape of a hamming window. So we're going to be designing a hamming window as a filter in the frequency response. And in order to apply it in db, we multiply it by minus 60 so that we have a shape. In fact, this is an invert hamming window, so this will be used to do, basically, a band stock filter. So we're going to attenuate certain frequencies using the shape, we're going to see that by plotting these curves. Then we take the magnitude and phase spectrum of the signal that we chose. And in order to apply the filter we basically decide where we're going to apply the filter, the filter is 30 samples long. So we're going to affect 30 samples of the spectrum, and we decide that the center is going to be the sample 40. And then now we copy the input spectrum to this my, and we're going to manipulate or we're going to change the 15 samples around this 40th sample. So the filter will be apply to the 30 samples that are around this 40th precision, and then we do the inverse DFT. Okay, so let's run this code first, we are in this directory in the work space, we can just run test. Okay, so we have executed the script, now we can keep plotting the different things. So we first can plot the wave form that we are going to be computing the DFT off, so this is the windowed fragment, okay. We can plot the filter, so if we plot the filter, okay. This is the shade that we're going to be applying, so in fact it already attenuates, because it's a hamming window so there is a step here, so it doesn't start at zero. And since we have subtracted 60 to it, it goes down to 60 and then goes up again, okay? So this is going to be the shape, the 30 samples that are going to be applied into a magnitude spectrum. Then the magnitude spectrum that we're going to be using is this, okay, this is the oval spectrum of 256 samples the magnitude spectrum, the positive values. And now the modified version, well let's maybe we can even plot here on top of these the modified one, so we can plot the m y spectrum here. Okay, so this green spectrum is the modified version in which everything is the same except this middle part, this is the part where we applied the window. And it clearly was affected very little at the boundaries and very much at the center. So we apply a smooth shape and therefore, we did a band stop filter. We kind of reduced, attenuated the frequencies that are around this area and of course then we obtained the output sound, which is the y signal. Okay, this is the modified input signals, so in fact if we plot on top of this, the original signal, okay, that green one is original signal, and of course the blue one has been attenuated, and it's shape is quite different because we have attenuated certain frequencies much more than others. So this is a filtering operation that can be easily done, and then in the stft transformations file that's within the transformations directory we can find this code, the stft filtering as it's implemented to be applied for a whole sound. In fact, it's the same idea, this is the definition of the function in which the input is the input sound, sampling rate, a window of 50 size and hop size. And then we pass it this filter which is an array that is going to be applied to the magnum spectrum. And then basically the core is this while loop that does exactly what we have been showing for one particular frame, perform the DFT, it applies the filter, and then it applies the inverse DFT. Okay, now let's talk about another operation that I believe it's a little bit more interesting which is the morphing that we talked about. So in this script, we are going to be morphing two sounds, the sound of a rain with the soprano sound, but again, one frame. Okay, in order to do this morphing, what we're going to do is apply the magnitude spectrum of one into the other or basically interpolating the two magnitude spectra, how do we do that? Well, we do exactly the same thing that we did for one single sound, we defined the window, we choose these two fragments. They have to be, in this case, the same size because we're going to be applying the same operation on both sounds. We're going to be interpolating the two magnitude spectra so we have to start from the same window size, or at least at the end have to be of the same size. Then we take the two spectra of the two sounds, of sound 1 and sound 2 and the core part of the morphing is in this next section. So what we're going to do is half one smooth version of one sound so that we only apply the generous shape of one sound into the other. And so that's a going to be the second sound is going to be the one to be smooth by one giving factor point 2. So we use the resampling, the kind of downsampling aspect to obtain a smoother version of the x2 spectrum. And then we at sample it again with interpolation to get the complete spectrum with a smoother representation. Of course we can just have no smoothing by putting a value of one. And then we're going to basically combine the two, a linear combination of the two. We are in a db scale, so we can just add them, otherwise, we would multiply them. In which there is this balance factor that defines how much of each is in the mix, basically. This point seven, it means it's point seven is apply to the second sound and the rest, so point three is applied to the first sound. And then of course, we take the inverse DFT from the resulting magnitude spectrum with the phase of the first signal, okay? So the phase will be of the first signal, and the magnitude spectrum is a combination of the two. Okay, let's run this again, let's run Test1, okay? And now let's keep showing the different sort of variables that we have, let's start with the magnitude spectrum. So let's plot the magnitude spectrum of sound one mx1, okay? And I had it open, so we have to close it here and let's open it again, okay, so this is the magnitude spectrum of the rain sound. So it's quite noisy, it's of course it, the case in the high frequencies, so the low frequencies are stronger but the case as the frequencies go up. And we can plot on top of that the second spectrum which is this soprano voice which is clearly different, soprano voice has this very clear shapes which relate to the formants. We are not so interested in this type of morphing to get a good resolution on the harmonics, we're interested in the general shape. Okay and now we can plot on top of these the smooth version of the second, so we applied a smooth factor of point two. So mxX2 is smooth two is the interpolated version of that, so if we put mx2 smooth two, that's going to be the smooth version of this voice sound. So, now this is the red line is the one that we'll be interpolated with the blue line, okay? And now with the balance point seven, the resulting spectrum is m y that also can be plot on top of these. So m y, okay, will be, now it's the blue line and it's clearly an interpolation between the two. So since this point seven is closer to the red line than to the blue line, if we had instead of point seven we had let's say point two or point one, let's say that and let's run the test1 again. And now if we plot on the same plot the m y, okay? Now it's this purple curve, and it clearly is much closer to the range sample because we have specified point one, okay? So hopefully that gives you an idea of this interpolating between two magnitude spectrum and of course then when we plot y the output this will be the resulting fragment of the sound that captures this interpolation or this morphing aspect. And if we go to the stft transformations and there is another function that performs that for a whole sound, the stft morph. Okay, so stft morph receives two sounds x1 and x2 and here we have the possibility of controlling windowing differently. And then there is the smooth and the balance and it does exactly what I showed in the case of two individual frames. Okay, and now we can show the file that basically is the one that is call from the interface the transformation interface that has one function name and that it receives two sounds and plots the spectrum and it plots the resulting morphing. So it has some default parameters and it does all the things that we have been doing, so basically for the whole sound, so it reads the sounds, it gets the windows. And then it performs the stft analysis for the whole sound and then it does the morphing between the two sounds in this case the stft is done so that we can visualize the spectrum of the signal. Okay, and then morphing we do again the analysis of the output sound so we can visualize the spectrum of the input and the output sound in the plots. And then it writes the resulting sound files, okay, so we can import this file so the stft morph function. We can import it as a name which for example ST and now we can call this main and of course we can change the parameters. So the default parameters where ocean sound with the speech-male, with this particular window. So for example let's change the balance so now it was point 2, so it was closer to the ocean than to the speech-male, let's make it so that the balance of f is very much close to the speech, let's type it right once come on balance, okay, f and let's put for example point 9, okay, and let's run this. Okay, so now we have the input sound, the spectrogram of the input sound and this is the spectrogram of the output sound, okay? And then, the actual output sound, so, clearly here, the output sound has captured quite a bit of the shape of the speech. So, if we can play it, it has save a file into this directory, so, we can just type play and Ocean stft morph. >> You hear me, they don't lie at all. >> Okay, so clearly he has taken the duration of the ocean sound, because that's the basic sound from which we're taking the analysis from. And then the magnitude spectra has been modified by these smooth spectra of the speech male sound. Okay, and that's all I wanted to say, so, we have seen how within the SMS tools there is all these code for being able to do morphing between two sounds using the short time fourier transform, and the code is quite simple. In fact, it is maybe the simplest type of transformation that we can do with this spectral tools and it can be quite useful and interesting for quite a few types of applications. So that was all, this is the first programming lecture. So in the next one we will talk about the Sinusoidal Model. How to implement transformations in the Sinusoidal Model using these same SMS tools and the particular code that implements them. So I'll see you next lesson, bye-bye.