So, now we're at level four.
At level four, you have evidence of a change in outcomes and you have proof of causality.
You've done a real experiment or a randomized control trial.
Those are the same thing by the way.
A real experiment means that you've actually done
a randomized control trial often referred to as an RCT.
And because you've done a real experiment or a randomized control trial,
you can actually claim we caused this effect.
You've got proof that you're the reason things are better,
your products, your services,
your business have really made the difference.
The key to being able to show that is randomization,
random assignment to conditions.
So let's talk about randomization.
When you do a randomized control trial,
then you randomly assign an intervention.
Again, training program maybe it's a fertilizer,
maybe it's a medical screening program,
you randomly assign this program,
this intervention to some study participants and not to others.
So, how exactly do you do randomization?
You might literally flip a coin,
you might use a random number generator.
So, in one way or another you're randomly determining who gets your products or services.
Randomization ensures that people are equally
likely to be assigned to the treatment or the control condition.
And statistically speaking, the two groups,
the experimental or treatment group and the comparison or control group, they are equal.
They have for example the same wealth,
the same average intelligence,
the same average age,
the same average fitness,
the same politics, the same distribution of genders.
These two groups are absolutely the same going into your intervention.
Of course, it's a good idea to actually check
this and to be sure that randomization worked.
But this is what random assignment is designed to do.
It's designed to make these two groups absolutely equal,
no difference in the groups overall before they get the benefit of your product,
your service, your operations.
So what this means is that at the end of the intervention when you assess outcomes,
you know that any difference you see between
the two groups was caused by the intervention.
It's the only thing that's available to explain the difference you see.
And when you're able to do this comparison,
the control group tells us what the experimental group would have looked like,
if they had not actually received the intervention in question.
So, randomization overcomes what researchers call,
the problem of self selection.
When you don't randomly assign people to treatments,
to programs they are likely to self-select into those programs,
and that self-selection makes it really hard to tell how effective your program is.
So for example, people who really love to exercise,
they may self-select into an exercise group.
And people who are really committed to saving a lot of money,
they may self-select into a financial services or financial savings education program.
Or people who are really ambitious,
may self-select into a training program to improve their businesses.
When you have that kind of self selection,
you don't actually know whether your fitness program,
your financial savings, services or products,
your training in business program,
you don't know if any of those things made a difference or these people were just super
motivated and they would have been as good as they are after training,
even if they had never gone into
your training program or whatever kind of program you have.
So, as you're hearing about
these randomized control trials, you may be thinking, "Well, wait a minute.
Is it really ethical to randomly assign people to conditions?"
In some cases it's absolutely
impossible to and unethical to randomly assign people to conditions.
We don't for example,
randomly assign people to smoke when they're pregnant.
We don't randomly assign people to drive
without seat belts or to drive over the speed limit.
So sometimes, we know this thing is dangerous,
we're not going to randomly assign it to people.
What about if you think something's really successful and positive,
is it ethical to randomly assign people not to get this benefit?
The fact of the matter is, we really do
experiments in randomized control trials to learn whether something is effective.
We don't actually know for certain it's effective,
that's why we do the randomized control trial in those circumstances it's ethical.
Usually we need to have people prior consent to be part of the study,
so we do have to think about ethics,
but again we don't know the program is effective.
If we think it's harmful,
we really shouldn't randomly assign people to conditions.
If we think it may be helpful,
then an experiment is a great way to prove that.
Let me just give one brief example of
how we do this random assignment that I think is pretty cool.
Sometimes you have programs where you have
lots and lots of people wanting to be part of the program,
wanting to get the product,
wanting to get to the service,
you actually have greater demand than you can fill.
In those circumstances you can actually randomly assign
half the people to get the product or get the program and the other half don't get it,
or they don't get it now,
they'll get it in the future.
That's a great way to do a randomized control trial.
So. Let's take an example of a recent randomized control trial,
what's called a field experiment,
a real experiment in the field that I think is quite interesting, quite impressive.
Also gives you some sense of why people say, "Whoa,
randomized control trials can be really expensive and complicated to do."
This is a study that was published recently in Science,
a very prestigious journal.
It's in our supplemental reading,
so you can read all about this study.
Campos and Frese and others were
interested in a real problem
in the world and they wanted to know if they could make a difference.
So what's the problem?
The problem is, around the world,
millions of people are earning a living as
a self-employed business owners and they're actually not earning a good living.
They aren't very successful,
they make a subsistence living.
Their businesses aren't thriving or growing.
And for these people,
yes they're self-employed but they're really living in poverty,
in some cases extreme poverty.
So, the challenge here is,
what would help the self-employed business owners become more successful?
And a common answer is,
they need business skills,
they need better training in the fundamentals of business.
So that's what we try to give.
It turns out that the results from those kind of
training programs they're not very good. They're pretty mixed.
So it's not clear that basic business skills training actually works very well.
Campos and his colleagues tried a different approach.
And their approach is called, personal initiative training.
In a nutshell and I'll say more about this in a moment,
the training program trains people to plan,
to set goals, to strategize about what they can do to overcome
problems and how they can put their plans into action.
So, people are trained to be thinking about if a problem arises for my business,
customers aren't coming in,
I can't get a bank loan to grow the business.
They're trained to be thinking,
if I have a problem like this,
what am I going to do about it?
How am I going to think about and figure out plan B?
So, Campos and his colleagues wanted to know,
is this training program,
the program and personal initiative,
is it really effective? Does it work?
To do this, they did a randomized control trial in Togo, Africa.
So, a major, really quite huge randomized control trial.
They had 500 microenterprise owners,
owners of tiny businesses,
they had 500 business owners who were in the control group.
They had another 500 microenterprise owners who were not in the control group.
They were in a comparison condition.
They got a traditional business education program.
And finally they had a third group of
500 microenterprise owners who received the personal initiative training.
So this is 1500 microenterprise owners
randomly assigned to the control condition to the comparison condition,
which is this leading business program,
or to the experimental program which is the personal initiative training program.
1500 people in Togo, Africa,
that's a lot of people who are receiving these two training programs.
These training programs last 12 half-days,
they are delivered in classrooms with a maximum of 20 business owners per class.
The trainers are teaching in a mixture of French and local languages,
and the business owners are also getting mentoring sessions where
individual coaches are coming to
their businesses for a few hours each month over a period of four months.
All in all, that's a very big intervention,
a complicated program to run and they're tracking people for
over two years following the training programs.
So, what did they learn? Does personal initiative training actually work?
Actually, the title of Campos and his colleagues article in Science says it nicely.
The title of the article is "Teaching
Personal Initiative Beats Traditional Training
in Boosting Small Business in West Africa."
Yes, training in personal initiative works.
How well does it work?
Personal initiative training increased profits by 30 percent on average compared with
a statistically insignificant 11 percent increase
for those who received the traditional training.
In other words, personal initiative training
is three times as powerful in improving profits.
This is a big deal that when you train people in this mindset you actually get
a 30 percent increase in the profitability of their businesses over time.
The training program is cost effective,
paying for itself within a year.
So, if you're like me,
you might want to know a little bit more about what is
this personal initiative training that's so effective.
So, let me describe it in a little bit more detail.
The essence of personal initiative is self-starting behavior,
long-term orientation and persistence.
So, what do these things mean for entrepreneurs?
Self-starting behavior means doing
something that differentiates your business from other businesses.
I'm thinking about places I've seen around the world where every business looks
the same and for me as a customer it's not that enticing to go shop there.
So, self-starting behavior in this context,
means doing something that makes
your business different and presumably better than other businesses.
And the program is training entrepreneurs to differentiate themselves.
Long-term orientation means making plans into the future and
thinking about business opportunities and threats a year from now,
not just this afternoon or next week or this month, but a year from now.
And persistence means not giving up
when problems occur or you have a failed business project.
Instead, it's how do I learn from mistakes and develop
my plan B and my plan C. So as I said, self-starting behavior,
long-term orientation, persistence, all of these things in
this randomized control trial have made a real difference in profitability.
So, a great example I think of an RCT.
So as we're thinking about whether it's worth it to do an RCT,
it's useful to ask some questions.
One of those questions is, "Well,
when is the right time to do an RCT?"
Turns out the timing is important.
It's not a good idea to do an RCT too soon.
When you're doing a full randomized control trial,
the program that you're offering should be piloted, designed,
you've tweaked it, you've fixed it,
you worked it through and you are really ready to roll it out.
You have a pretty strong sense of exactly what the program is,
how it works and you have a strong suspicion that it's going to be effective.
So, if it's still in the pilot phase,
don't do a randomized control trial.
Conversely, if it's really late and the program is all scaled up,
you've already scaled the program,
probably not worth doing an RCT,
very difficult to change it at that point.
So, mid-stage in development,
that's a great time to do an RCT.
When is it a bad idea to conduct an RCT?
Well, obviously it's a bad idea if a randomization isn't feasible.
So, if you can't randomly assign people to conditions, you really can't do an RCT.
If it's unethical to randomly assign people to these conditions,
also don't do an RCT.
Implementation fidelity is an important idea around doing an RCT.
You need to have fidelity to the plan.
Campos and his colleagues needed to know that
this training program was going to be rolled out consistently,
it was going to be faithfully implemented.
That's implementation fidelity.
So, it's a bad idea to conduct an RCT when
you don't know that you have implementation fidelity.
It's a bad idea to do an RCT if the sample is too small,
a lot of work to have too small of sample size.
And obviously, if you don't have enough time or money to do a good job with an RCT,
it's not worth trying to do one.
So, what are some of the criticisms of doing RCTs?
Well, I've already hinted at some of the challenges here.
They are expensive and time consuming.
Another criticism of RCTs is they may not be generalizable.
I might show through
my randomized control trial that the program works really well in Togo.
But can I generalize it to Philadelphia?
So, this question of generalizability isn't really
a fair criticism of experiments because truth be told,
generalizability is a question in all kinds of research.
The same issue arises if I use lean data collection,
if I conducted interviews, whatever I do,
wherever I'm collecting data,
there's this interesting question of, "Well,
what if you collected data from different people in a different setting?"
It can be hard to know what the results will generalize.
We get better evidence of generalizability the more we do
studies in different places and show that the results generalize.
And that said, we get strong clues about generalizability
when we think about the underlying mechanisms and the boundary conditions.
Now, how is this program working?
In some ways this brings us back to the logic model,
which allowed us to think through the whole chain.
What are the mechanisms that make,
in this case for example,
personal initiative so successful?
Do entrepreneurs in wherever we want to bring this program,
could they benefit from personal initiative training?
These are the kinds of questions that help us guess at generalizability.
When researchers do randomized control trials,
one of the things they'll tell you is,
these randomized control trials can really influence future policy.
A great randomized control trial changes things if we
can show in this randomized control trial that there are real benefits.
It's an interesting question about whether
randomized control trials really drive policy change.
Sometimes that's the case, sometimes it's not.
One of the challenges is that,
academic researchers, people like me,
are the folks who do randomized control trials and we may
be really good at the research skills,
we may have strong research skills to do these randomized control trials,
we may or may not be as good at advocacy and driving policy change.
So, it's worth thinking about if you're reading randomized control trials,
if you're doing them, can I use this randomized control trial?
Can I use these effects to drive policy change? All right.
Wrapping up, our randomized control trials,
really the gold standard?
Clearly they're the gold standard for establishing causality.
They're great for precision.
They're great for communicating in a clear straightforward way.
What's the difference between the people who got my treatment and the people who didn't?
We saw in the Campos study,
personal initiative training raise profits by 30 percent,
that's three times more effective than traditional business training.
That's a clear and important and powerful result.
On the other hand, they're expensive to do,
they're time consuming to do and sometimes they're not so good
at giving us all the nuances of who benefited most,
what was their experience and how will these results generalize.
I would say that there's no one perfect research design.
Randomized control trials are great,
but I'm a big advocate of using all the data
and many different research designs to learn as much as we can about impact.
