Welcome to the course on Audio Single Processing for music applications. This week is the week of the Short Time Free Transform. And in the previous programming class, we talked about windows. We talked about how to get a window, how to display it, how to compute it's spectrum. And how to visualize the kind of things that we talked about in the theory class related to windows. And now, in this second programming lecture, I want to build the whole short time for your transform system and talk about the implementation we have in the semestral's package. So we will basically, implementing the whole STFT equation shown here. So basically, we'll be iterating over a sound x and we will be computing at every frame, so the frame number is l. So we'll be computing frames of the sound at each frame location we will be multiplying the fragment of sound with the window, w. And therefore, at the end we'll have a sequence of spectra. A sequence of in this case, complex spectra. And we see it as a block diagram so we start from the signal, x of n, we multiply one fragment with our window so we get a windowed signal. That's what we compute the FFT of and then, of course, we compute the magnitude and phase representation of the spectrum. And then, we can undo the whole process by calling the inverse FFT, therefore obtaining the windowed synthesize sound and what we call overlap at strategy. We will be adding together these fragments into a whole signal. And hopefully, if we have done the things correctly, especially if we have chosen the window and the hop size correctly, these output signal will be identical to the original. So let's look at the code of the STFT that we have in the SMS tools package. Okay, so this is a filed called stft.py and in it we have three functions. One there is a STFT, that does the whole analysis and synthesis. And then, we just have the analysis and synthesis separate, so sometimes it's useful. So we have the STFT-anal function and the STFT-synth function for the inverse part. But let's talk about the complete analysis, synthesis, so let's talk about the STFT function. As input parameter is it has the X, our array with the complete sound, we need to do the sampling rate of that sound. Then we need to pass it, the window we're going to be using W then the size of the FFT capital N and the hop size, how are we going to hop through the sound. And basically, the core of the function is a loop In this case, a wild loop that keeps advancing through the sound by a pointer, this pin. Pin is a pointer that keeps advancing by age, by capital age, by the hop size. So, pin gets advanced at every iteration by capital H until it reaches the end which is the end of the sound. And within that, we basically choose the fragment of the sound that we want to analyze and then we call the DFT anal function. So this is the function that we already discussed last week which is the DFT model, it comes from the package DFT model, okay. So we don't have to worry now here about the zero phase windowing about the zero padding, etc. Everything is done within this DFT-anal function and it receives the fragment of the sound, the window, and the FFT size. And it returns the positive site of the magnitude and phase spectrum, okay. After that, we can do the inverse DFT also calling the DFT-synth function. And therefore, that will return a window fragment of the sound and then we can put it all together, overlap and adding the sound and creating the overall output array. Okay, and apart from that the rest are basically initialization and ways to better analyze the sound and to make sure that the pointer Is can start at the very beginning of the sound. And can end at the very end of the sound, so I'm not going to go into detail of that but please feel free to go over that and understand the scope. Okay, so now we have this function so that returns sound and let's use it. Let's write some code to use this function. So this is a little script that I wrote to call the STFT aal, in which I first have to get out the packages required including the actual STFT function, so this comes from the models and directory. And then, I can get a file, define the window and then, define the size of the window, the FFT size and the hop size. And now, we can create the file with wavread, get the window and call the STFT function. And this will fill up two variables mX and pX which will include the sequence of magnitude and phase spectrum. Okay, so let's run, it let's run test3. Okay, that's quite quick, so it analyse the sound and it adapting all these variables. So, let's go through them, so maybe the first thing we can do is to listen to the sound. So we can just do the exclamation mark play and the name of the sound, which is this flute sound, and we can listen to that. [SOUND] Okay? So this is the sound of the flute and now, we can go through the different variables that we have computed. So for example, the first one is the sound that we read, the X array, so this is the whole sound, the array of the whole sound, so it's more than 80,000 samples. Okay, then of course we can plot the window that we are computed which is a humming window. And then, we have the output array of magnitude and phases but those are very big variables. They are very big matrixes of values, so in fact, if we put mx.shape it will tell me the dimensions of the ray. Basically, the number of frames and the number of samples in the spectrum that we have. So in fact this tells me that we have 238 frames and each frame has 512 samples. These 512 samples are the samples of the magnitude spectrum of half of the magnitude spectrum. So now, we can access individual frames of that, so if we put for example mX, and we access for example the frame number 50. And we plot the whole spectrum of that 50th frame, these are the values of the magnitude. And of course we can, if we call the function plot with that variable, we will plot the magnitude spectrum of the particular location. Okay, so this is the flute sound, we see the harmonics and we see how the higher values are much softer and it's quite noisy. Of course we can also plot the phase spectrum of that same location, this is the phase. The ungraph phase, so we can see all these a nice looking face shape, okay. Of course we want to visualize the whole spectrum, so in order to visualize the whole spectrum the plotting command that can do that is the what is a the function is called pcolormesh. Pcolormesh it plots a matrix as a kind of a three dimensional shape, so if we give it p and mX it will basically plot all the frames. But it maybe plotted in a different way that we are accustomed to because the vertical is the axis is the time. So in fact, we see time going from bottom to top and the frequency is on the horizontal axis. In fact a better way to plot this with colormesh is transpose it first, so we can call the transpose function. So this will transpose the mX in a different way, so that we can see it in a more meaningful way. And so now we see time going from left to right and frequency going, or the frame, yeah, frequency going from bottom to top. Okay, so that's basically how we compute the magnitude and the phase spectrum using STFT and how we can plot it. And there is a function file that implements all that and makes plots in a nice way and that's the STFT_function. And this is the one that is call from the graphical interface that we also use and here it has one function called main and it goes through the kinds of things we did and it does both the analysis and synthesis. Okay, so the main has its input values all the input parameters and then it reads the sound file, it gets the window It performs analysis. This is exactly what we did now but then it does the synthesis and it output the synthesis file into a location with a given name and then it writes it, okay? By the way, here this file I just copied into the work-space but I had to change the relative path of a number of these names. So for example, I have to change the path of the software models or I have to change the output file name because in here it. It was using particular directory to store the sound files, so here, I just write the directory on this directory. Okay, so let's execute this, that's why I have it in this directory. So, it's STFT function, so I can just run STFT function, okay. And it plots, well, the input sound, the magnitude, the derivative of the phase and the output sound. And this has created a file which is called what is it called is called piano stft.wav and of course, from here again, I can play so, I can say play stft, I know piano, piano Stft.wav. [MUSIC] Okay, and it's basically the same than the original sound. Of course now, from this file I can really change and make the things differently if I want. So feel free to change how the plots are shown but these are quite nice way to plug the things with the correct access, with the correct labels, with the correct scaling. So that we can better visualize the analysis and synthesis of using this TFT and that's all I wanted to say. So basically, we have gone through the short-time Fourier transform, the implementation of the short-time Fourier transform. So make sure that you understand the actual equation and what are the signal processing considerations that are involved in that. And then, of course we use a number of packages from Python together with the SMS tools that we have already developed an implement the DFT and the SDFT so we have been calling those functions. That's all, so with this programming class hopefully, you have gotten an idea of how to implement the STFT and how can we use the functions of the SMS tools to actually adapt it to our own particular needs. So please feel free to play around with that and make sure that you understand the STFT also from this programming perspective and that's all. Thank you very much.