In the last course of our specialization, Overview of Advanced Methods of Reinforcement Learning in Finance, we will take a deeper look into topics discussed in our third course, Reinforcement Learning in Finance. In particular, we will talk about links between Reinforcement Learning, option pricing and physics, implications of Inverse Reinforcement Learning for modeling market impact and price dynamics, and perception-action cycles in Reinforcement Learning. Finally, we will overview trending and potential applications of Reinforcement Learning for high frequency trading, cryptocurrencies, peer-to-peer lending, and more.
Quantum Equilibrium-Disequilibrium
Loading...
Del curso dictado por New York University Tandon School of Engineering
Overview of Advanced Methods of Reinforcement Learning in Finance
20 calificaciones
New York University Tandon School of Engineering
20 calificaciones
Curso 4 de 4 en SpecializationMachine Learning and Reinforcement Learning in Finance
De la lección
Reinforcement Learning for Optimal Trading and Market Modeling
Conoce a los instructores
Igor Halperin
So, in the last video,
we spoke about challenges with modelling defaults with classical financial models.
If we summarize it,
we can simply say that the GBM model
directly contradicts the data because it does the meet defaults.
You don't even have to compare predictions of the model versus
actual stock volatility patterns as is usually done in empirical tests of the GBM model.
The model misses something very important namely, defaults.
Therefore, it cannot be a complete model as a matter of principle.
True defaults are rare events but they're
critically important for functioning of financial markets.
If you use the GBM model or a similar model for stocks,
we need to use something else such as credit spread models to incorporate credit risk.
Some sort of account for credit risk is important
for trading in at almost every time horizon,
exclusion possibly only a very short time,
whereas you're trading, intra-day trading.
So, as we see,
if we have a mathematical model with a simple formulation,
it doesn't mean yet that it's simple to use and
practice or that it's all conclusions makes sense.
So, not to contribute to essentially a long discussion of whose methods are better,
I cannot resist the temptation to tell a joke I read recently on that topic.
The joke is about a biologist,
a physicist and a mathematician,
who sit at the bar and observe two people entering a house across the street.
After a while, three people emerge from the house and leave.
Then the biologist says,
the population has replicated.
The physicist says, it's an error in the measurement.
The mathematician says, if now another person enters the house,
it will have exactly zero number of people inside.
So, okay.
Coming back to finance,
we spoke in this lesson and the previous ones about challenges of
financial modelling when using the classical concepts of competitive market equilibrium,
and related models such as the Geometric Brownian Motion,
CAPM or the Black-Scholes option pricing model.
Recall that a market in this approach is assumed to be insulated
from the outside world as it doesn't receive any new money or new information from it.
Yet, the GBM model essentially predicts that in this world,
assets will grow indefinitely large.
The very existence of the market is not
explained in the competitive market equilibrium models.
But instead, is just postulated.
I have already mentioned an alternative paradigm
to how markets can function called equilibrium,
disequilibrium, which was a term suggested by Amihood and co-workers in 2005.
In this picture, a market has a continuous access to new capital and new information.
Market-makers provide liquidity in an amount which is optimal for them,
which impacts market prices.
Investors invest or withdraw capital from the market again
in an amount that is optimal for them, and everything moves.
The only thing that is equilibrium about this,
is that it looks more or less the same pretty much all the time.
In physics, this is called a steady non-equilibrium state.
Now, what we will do next is consider
a model that can implement such a state of equilibrium,
disequilibrium in the market.
The model has to do with what we discussed in
the last week of our course on reinforcement learning,
and also to what you did in your course project for this course.
So, let's let Xt be total market capitalization of a firm.
For convenience, we can scale these by
an average market cap over the whole observation period for example.
So, we can think of Xt as
dimensionless variable whose values would be an average of the order of one.
Now, we can consider a discrete-time dynamics described in equation 17 shown here.
These are essentially the same equations that we used in
our previous course when we considered at various portfolio optimization problems.
Previously, Xt was a total dollar value
of the stock position in an individual investor portfolio,
and U sub t was a change of
this position in asset at the beginning of the time interval t,
from t to t, plus delta t. Now,
the equations are the same.
But because we apply them to the market as a whole,
the meaning is different.
Now, Xt stands for the total market cap
of the form while the U sub t is the total injection or withdrawal,
depending on the sign of money in the market at time t. So,
the first equation says that the price process is determined by first,
a new injection of money Ut in the market,
in the beginning of the interval t,
to t plus delta t. Then,
the new value Xt plus Ut grows at the rate rt.
The formula for the rate rt is given in the second equation,
where rf is the risk-free rate,
Zt are signals, w are their weights and the theorem mu,
times ut describes a linear market frictions.
So, that mu is the friction coefficient.
Now, please note that the money supply Ut can also
be zero here in a particular period or negative,
for let's say in a particular period or in all periods altogether.
Now, note that if we set Ut equal zero in this equations,
identically, we exactly recover the Geometric Brownian Motion model.
But, what if you want to keep Ut different from zero?
What should we use for it then?
Well, in general, Ut is the amount of money that
the outside investors put or withdraw
from the market depending on the market conditions and on outlook.
So, it should depend on the market value Xt,
plus it can depend on other things such as signal Zt.
We can take a simplest functional form for Ut as
a function of Xt that is shown here in equation 18.
It says that Ut can be taken as a quadratic function of Xt with a zero intercept.
We can view this function as
either low order Taylor expansion of some other more complex function.
Or alternatively, we can view it as utility function of outside investors in the market,
which would be similar in spirit to the picture of equilibrium,
disequilibrium of Amihood and coworkers.
Now, if we assume that parameter lambda is positive but very small,
then the behavior or function Ut will be mostly driven by
the first term as long as Xt is much smaller than the ratio of phi over lambda.
If parameter phi is larger than zero,
money is pumped in the market.
Otherwise, money is withdrawn from the market.
Now, if you take this expression for Ut and substitute it in our equations 17,
then drop towards, which are proportional to delta t squared,
and then proceed to a continuous time limit,
you will get the equation 20.
Because this equation describes
dynamics in the spirit of the picture drawn by Amihood and co-worker.
I call these dynamics quantum equilibrium- disequilibrium model.
The meaning of word quantum will be more clear a bit later.
But looking at the equation,
it has four parameters.
If you don't count the weights of the signals as it is.
So, the parameters would be sigma, kappa,
theta and G. The three new parameters,
G, kappa, and theta are defined here in equation 21.
So now, if we keep the sign of mu positive,
the mean reversion parameter kappa here can be of
either sign depending on the sign of a phi and the value of lambda.
So, by playing with different combinations and values of these parameters,
you can actually obtain lots of
very interesting dynamics and
this is something that we'll be doing the next week. See you then.
Explora nuestro catálogo
Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.
Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.
© 2019 Coursera Inc. Todos los derechos reservados.
