SPEAKER 1: Please help me in welcoming Cedric.
CEDRIC BEUST: Thank you.
OK, this works.
Well I still hope this presentation is still cool by
[? Alan's ?] standards even if it's not
exactly what he expected.
So, the title is pretty long.
It's "Beyond JUnit, TestNG and Testing The Next Generation."
I promise I'm going to explain all those items one by one.
And here's a little bit what this presentation is
going to look like.
It looks a bit funny from here.
I'm not even going to tell exactly what
TestNG is right now.
I'm going to ease you into it, if you're not
comfortable with it.
I'll start by talking about JUnit which I assume a lot of
you know and then I'll slowly introduce TestNG and we'll
dive into details.
By the end and depending on how much time we have, maybe
I'll skip on some of the end slides so we can have some
time for questions and answers.
I'll start with JUnit.
Who here is familiar with JUnit?
Can I have a little show of hands?
I'm not surprised.
Everybody who programs in Java and is a decent Java
developer, knows about JUnit.
Something strange has been happening lately in the
I don't if you guys remember, maybe ten years ago or maybe
so those of you who were there before--
testing was considered kind of a low level activity.
Something that developers didn't really want to have
anything to do with.
No, you write the code but no, I'm too good, I
don't test my code.
And at the very least, maybe I'm just going to write the
code and toss it over the wall to a QA department and tell
them, test my code for me and it's-- that's
not worthy of me.
And please don't bother me, there are bugs.
Well this has changed.
I don't know exactly when.
in 2000, 2001--
it's hard to know exactly-- but at some point, testing
started making its way to our developer's habits.
It became part of what we're doing that, when you write
code, we should probably also provide tests.
A lot of them were early build but it is really part of our
consciousness, that it's really--
we need to be professional about it and we need to
So a lot of things have happened.
We can definitely credit the people who were behind the
extreme programming movement.
They really tried to make test a first class citizen of
engineering-- software engineering.
Test-driven developmental also has become quite popular--
I'll talk a bit more about this later.
And, of course, there were something else.
There was JUnit, which came out at, I think, around 2001.
So what happened?
In 2001 when JUnit came out, we did not have a standard way
to test Java code.
It's hard to remember it.
I can't even remember how I was writing test before that.
The truth is I probably was not.
But until JUnit came out, there was nothing.
We were all doing it in an ad hoc way,
using our own framework.
And we were basically reinventing some of the things
that JUnit formalized when it came out.
So it doesn't need much introduction.
I'm not going to spend too much time on this.
I'm pretty sure you all know JUnit well, although I will be
testing you on this very soon.
It's a very simple testing framework, the concepts are
easy to understand.
You have these set up tear down classes, you have those
test methods that start with the word test. And it took the
Java world by storm.
There were such a need for a standard Java testing
framework that, within a few months, there was no question
that anyone who was testing Java code was using JUnit.
So it's a very easy to put in place, very easy to
You can pretty much read two or three pages on how it works
and you can get up and running and you
write your first tests.
The concepts are simple, set up, tear down.
You have test runners.
You have this concept of a suite, which can have a
hierarchy of tests that are smaller and smaller.
And it has a huge ecosystem.
There are a lot of add-ons available to do pretty much
anything you want, to test GUIs, to test database, to
There's a wealth of material, of books and articles and
conferences like this one, plenty of experts on
newsgroups and mailing lists.
The resources to find JUnit documentation are really,
OK so here's the test. This is a very simple
to JUnit test case.
It has two test methods, one field and these two test
methods do exactly the same thing.
They increment the field and then they try to verify that
the field is equal to one.
Just give you a few seconds to understand
what's going on here.
And I'm going to ask you, a quick show of hands again, who
thinks that these tests pass?
I have one, two shy hands, three hands.
Who thinks that this test fails?
A lot more.
Who is not paying attention?
So, that's most of you.
OK I am not going to do a demo.
You can trust me on this.
Or you can just quickly run it on your laptop.
The truth is that this passes.
So to the two or three who raised there hands, you
actually know JUnit pretty well.
Those who raised hands to the second question, you might
have a few things to learn about JUnit and I'm hoping to
dispel some of those things.
Let's go back again.
How can this pass?
Isn't it a bit surprising?
If, in Java, I do something like this, I have a field and
I cull two methods, and each method
increments is the field.
By the time I'm done, this field should be equal two.
But if you run it was JUnit, it's equal to one.
What's the only explanation?
Well, my class must be reinitialized There must be a
new instance that gets created every time.
It's pretty easy to verify.
It's exactly what's happening.
JUnit is reinstantiating your class every single time before
it invokes a test method.
I was bitten by that.
I mean that was a bug that really baffled me.
And then I went into the JUnit code and all that.
I found out this thing that made it obvious that the
behavior was that.
And then I wondered, it can't be a bug.
JUnit's been around for years.
If it was not designed to be this way, we
would know about it.
So I did some more homework exchange of emails with Erich
Gamma and a bunch of other people.
And they all confirmed, yes, it's by design.
And here is the reason, we want your test methods to be
independent from each other.
We don't want them to have to rely on a field having a
certain value that has been set by previous tests.
You should be able, at all times, to be pick a test
method and run it.
And it doesn't depend on anything.
All right, that sounds like a good idea.
I can see some value in this.
But there was still something that was nagging me, I was
thinking, but I still want to do this.
I still want it to maintain some states.
Does that make me an evil person?
Was my code really ugly?
Shouldn't it have been doing this?
What's wrong with that?
The example that I was using back then was, I was writing
test for XML file and I had a huge XML file that was
parsing, that took a while.
Out of this, I was building a tree, or a dome, or whatever--
something pretty costly.
And then my test methods were just making sure that the XML
file was correct.
But because of that, I had basically to reparse my file
every time before I invoked a test method.
I said well that doesn't seem right to me.
It's a read only file.
I'm just going to be create it once and then I want to run my
test. So what do I do?
Well, I'm going to work around this little clever thing that
JUnit has put in my way.
And I'm going to use a static field.
I'm going to make sure that it's only initialized once.
And then the other times, it's not going to
All right, so that works.
That's why I know static fields work.
But I started feeling a bit uncomfortable about this
because now I am trying to work around things,
introducing statics, which I don't really like a lot.
Introducing statics has some downsides.
It's been hard to make it threat safe.
You never really know when it gets initialized.
If you run the same thing several times, in the same
class loader, in the same JDM, the state is going to remain--
which is exactly what JUnit was trying to avoid.
So I started pining over those work arounds, and the static
things I noticed were everywhere in our code base
because of that.
And some of the things I noticed as months went by, I
started examining how I felt about JUnit.
How do I write an individual test method?
I have 20 test methods in my class.
One of them is failing and I just want to run that one so I
can debug what's going on.
Back then, when we didn't have that much support in IDEs,
there were not a lot of solutions for that.
I was just commenting out those 19 out of 20 methods, so
I would just run that method.
You can do a little bit better with IDEs now.
But I thought again, shouldn't the testing framework help me
a little bit there?
Can I just ask JUnit please run just this test method
instead of having to recompile?
So all this let me down a path where I started questioning a
little bit what I was doing with JUnit.
And then actually whether there I was really doing
testing on my code or doing JUnit testing of my code.
My interpretation of what's going on is that in some way
JUnit was a victim of its own success.
It became the de facto standard because there was
But it wasn't really ready to become that.
And also JUnit is a unit testing framework.
Unit testing is really small units of test, at the scale of
the class usually.
It's important to have those tests.
But when it comes down to bigger testing--
there are various names, functional testing, system
testing, integration testing, basically when you're trying
to test how several classes work together--
I started realizing that JUnit didn't really help
me get my job done.
Because it was focused on unit testing it didn't help me for
these other types of testings.
And I wanted to be able to both.
I felt a bit frustrated by the configuration control that I
had, so there are set up and tear down methods which wrap
each test method.
I realized, well, sometimes I want to do a class level.
I want to initialize something for the class once and then I
don't need to initialize again.
How about suite level?
Something that's just initialized once the beginning
of my suite and never again after that.
And I realized, well, set up and tear down, it's o k but I
want a bit more.
It was also getting a bit old.
It hadn't been updated in a while.
This is not quite true, as you know JUnit 4 is out--
but work resumed on JUnit just a year, a
year and a half ago--
which means between 2001 plus and 2005, we were working with
JUnit 3.8 and it hadn't been changed at all.
Hadn't kept up with things like cert and generics and a
lot of other things that we're used to using modern Java.
I thought JUnit was a bit intrusive.
It forced me to write my test and forced me to use certain
names and extend certain classes and use certain idioms
that were not very intuitive as I just showed.
And I also thought that in itself, JUnit.jar the little,
I think, 50k or 60k, probably a bit more now, was really,
I really needed to add a lot of other things in order to
really make my job easier.
OK so that was a quick discussion about JUnit just to
try to make you think a little bit, and make you question
what you're doing.
And now let's try to see there is a way that we can find a
few workarounds around this.
And then, without more ado, I introduce you to TestNG--
which is a testing framework that is the result of all
these little hardships that I felt through JUnit--
which I created three years ago, three and a half years
ago-- just trying to experiment a little bit.
Back then, annotations--
we were still working on annotations.
They were not in the JDK yet.
JDK 5 wasn't out.
So I thought maybe I'll try to play with those annotations
and trying to see all these different things.
So here is, in a nutshell, some of the
main features of TestNG.
Based on annotations, I'll show you an example of that.
Anyone here who is not familiar with Java notations?
Never seen any Java annotations?
OK there are-- most of you are.
Maybe you don't use them, because you are not on JDK 5
but I'm sure a lot of you are aware of what it is and if
anything if you're stuck with JDK 1.4, you're hoping, oh,
gosh I can't use that, but at least I know what they are.
Flexible run time configuration--
I'll explain what that means later.
Another thing, also, that I thought was very important is
being able to put my tests in groups, not just classes, but
really in groups that I can call whatever I want,
I'll show examples of that.
Some other more practical features that I wanted, again,
to do my functional testing on top of my unit testing, such
as dependent methods, being able to test my code and make
sure that it's multi threaded, being able to
run my test in parallel.
And now even laptops now have multi cores and multi CPUs.
And when I run my tests, and all this test methods are put
independent from each other, it would be really silly not
to have them run on separate threads and just leverage what
my laptop has to offer.
So I wanted to build that in also.
And I wanted also think the API guide to be flexible, so
that people feel comfortable putting their own things,
creating new reporters, and new runners and things like.
All right, a bit more code.
Here are two examples of TestNG.
And I did mention that in the list earlier, but TestNG works
both on JDK 1.4 and JDK 5, which might surprise you a
little bit, because I just said it works on annotations.
I said, how can it work on annotations and JDK 1.4?
Well the answer is here.
Just using Javadocs, for those of you who were using
annotations, before they became cool--
with things like XDoclet.
That's how XDoclet introduced annotations to the Java world,
just using Javadoc and using tags.
So it's really easy to simulate exactly what the JDK
5 annotations give us, using these comments.
Of course it's a lot less safe because the compiler has no
idea that these things are here.
You can misspell them and then the tool is
going to miss them.
So it's convenient if all you have, if you have JDK 1.4 and
you can't upgrade, if you have a choice between 1.4 and 5, I
strongly suggest, use 5.
In this example, we have two things.
Let's start with one of at the bottom, which basically says
that the method called server is running is a test. In
JUnit, you would say that just by calling it
test server is running.
But now with annotations, we no longer
need this little trick.
We can call the method whatever we want and we use
the annotation to tell TestNG, hey, by the way, this is a
And the method at the top is what I call a
It's like a set up in JUnit, except that this one is a
It's like a setup, except that it's going to run before you
run any tests on that class.
And it's only going to run once.
If you have ten methods on your class, before that first
method is invoked, we're going to invoke in it and then it's
going to call the ten methods and you can also specify an
after class to do some cleanup at the class level.
Here's what it looks like on JDK 5.
It's a little bit more concise.
But most of all it is type six and you can use your IDE, you
can use completion, you can start typing at before and the
ID's going to tell you all the annotations that you can use.
A lot more convenient, I think more
powerful, also, more readable.
And that's why I recommend this approach if
you have this option.
From now on, I will just show these examples in JDK 5
format, but I whatever you can to do JDK 5 , you can do in
JDK 1.4 too.
So a few remarks on the code that we just saw.
A few things, a few immediate benefits that we have. The
first is that we don't need to extend a base class.
You remember with JUnit, you need to extend this thing
called test case.
This is how you enable the magic of being able to have
set up and tear down and things like that, have the
We no longer need to do that, which is kind of convenient
because extending a base class is a pretty big deal in Java.
We only have one shot at that.
If you want to extend an implementation, you can only
have one base class.
And when you use a framework that forces you to have this
base class, sometimes it gets in your way and you have to
find your way to put your own class in between.
Again no need to start your method with test. You don't
need to call your configuration methods setUp
Again by the magic of annotations, we can just
And something else which is not shown in the example, but
on which I will get back there soon is that test methods can
Yeah, how about that?
Did it ever occur to anyone here that, all right, I love
JUnit and wait a minute, I'd like to pass a parameter to
I can do this in Java.
I can pass a string, I can pass an integer, I can pass a
How do you do that in JUnit?
Well it's pretty convoluted.
It's possible but it's convoluted.
And when I started TestNG, I was thinking, well, I'd like
to do it exactly like I do it in Java, so let's
see where we can--
how far we can take this idea.
Turns out, it works pretty well.
Here are some of the annotations that the TestNG
There are actually a couple more, so it's really not that
many, especially since-- so you have the test when you
have all this configuration before class, after class,
before method, after method.
There are quite a few.
It's not really a problem to have a lot.
It just gives you a lot of flexibility to know exactly
when your initialization needs to take care--
needs to take place, sorry.
If you want to start your application server, it's
obviously something that's going to take it-- to start at
the very beginning of your suite.
If you want to just initialize the state of your class,
you're going to do it in a before method and after a
no, an annotation.
DataProvider is another interesting one that I will
describe very shortly which enables something that is
referred to as data driven testing.
Just to whet your appetite, it's the kind of testing where
the code of your test is actually pretty constant,
pretty simple and what changes is the data that you're
feeding to this test. And Factory, which allows you to
go wild and pretty much create all your test instances, in
case it's not enough to just have them.
Terminology because of all these configuration things, I
had to come up with a pretty strict terminology in order to
match those concepts.
So it's actually pretty simple.
There's just one addition, compared to
what you're used to.
At the very top is a suite, which is what contains your
entire test environment.
And each suite is made of several tests.
And each test is made of several classes and each class
is made of several methods.
And the reason why this little indentation is useful is that
you have configurations that are possible
around each of them.
So we have a before suite, after suite.
We have a before test after test. We have before class,
after class and before method after method.
And even one more than I have listed here.
Again think of it as just a way for you to be very, very
fine grained in the way you can initialize your tests.
A quick list of pretty much exactly what I just said.
So the only thing that we are familiar with from JUnit is
before method and after method--
equivalent in JUnit is set up and tear down.
Everybody is familiar with that.
These other things are actually new in TestNG.
And just for completion--
for completeness, sorry, the before class after class
annotation we're introduced in JUnit 4.
So that slide is no longer quite current.
JUnit 4 has support for before class and after class but, as
far as I know, they still don't have before suite and
before test either.
Another benefit of annotations is that they can have, I'm
going to call them parameters but their real--
the real technical term an attributes.
But it's basically a parameter that you can pass to an
annotation to add some extra information.
This is a bit nebulous for those of you who never used
You can just take a look at the end of the listing at the
bottom, where we see the at test annotation.
But this time instead of being all itself, it actually has
One that is called groups, the other that is called time out,
the other thing that is called depends on groups.
And each of them is assigned a certain value.
You can probably guess what those do.
But I will explain this very soon.
Groups gives you a list of groups that you want this
method to belong to.
Another hint of what you can do is that, instead of having
this method belong to a certain class or certain
package, which so far was the only way that we could really
say this method test databases.
So I'm going to put it in the package called
com.example.test.database So all right,
it's better than nothing.
I know if I want to run my database test, I'm going to
tell JUnit to run everything in the package com example db.
But I realized this is kind of unidimensional.
Very often, I want to tag more attributes to my method.
My method is not just a database.
Actually it happens that method runs pretty quickly.
I like to put it in a group called fast. I'd like also
this method to be part of--
or maybe that one is slow and it runs only weekends.
So I'd like to say, just put it in the group, weekends.
Just to put some categories in the methods and what this buys
us is that once we have compiled our entire system,
then we can switch to the run time part and tell TestNG run
all the fast methods, run all the database methods, run all
the nightly methods.
And what this buys us is that we don't need
to recompile anything.
First of all, it's already burned.
It's already part of the class file.
And second, as you can see, methods can
belong to several groups.
They are no longer constrained by the package part.
And you no longer need to name your package a certain time
and you have this other dimension where the package is
one thing and then the list of groups is another thing.
Parameter is another example.
That's how we pass parameters.
I'll give you an example soon.
It depends on group timeout.
Timeout is a pretty useful one also--
which is basically a way to tell TestNG that if this
method doesn't return within the timeout specified, say 10
seconds, TestNG should abort the method, mark it as a
failure and then move on.
It can be pretty convenient when you are making will test
on heavy network, or under a heavy load and you know, some
things can work.
Or part of the functional specification of the method is
that it should really read your return within one second.
And if it doesn't, then something is wrong and we have
There is a little XML file that is pretty
important with TestNG.
It's basically what contains the entire
description of your tests.
Before I go any further, let me just specify for people who
are a bit allergic to XML, I'm sure there are
plenty in this room--
enough XML, I can't read XML any more--
this is optional.
There are plenty of ways that you can avoid doing this.
The bottom line is that a lot of people who are truly very
hostile to XML end up actually using the XML file because
it's very convenient.
It works very well with TestNG.
But if you don't like it, you can totally avoid it.
The interest of this is that it's very easy to have an
externalized description of all your test classes, all the
groups that you want to run, all the various parameters,
like the timeouts and things that you are
passing to your methods.
It's all contained in this XML file, which you can email your
coworkers when you're trying to have them reproduce a
certain configuration, say hey, can you run this XML
file, you will see it will fail at the fifth implication
of that method.
It's not something that's very easy with JUnit, again,
because it's only described in code.
And you can't really have a capture of an environment
that's failing and then ask someone to rerun the exact
So what this XML file contains the list of all the test
classes and the test methods that you want to run, which
groups you want to include which
groups you want to exclude.
Another little trick, I'm going to show you an example
that's a pretty powerful idea also.
You can also define additional groups to make
those big meta groups.
Also in XML, you can also specify how your
test should be run.
Do you want to run these ten classes in parallel or should
they be run sequentially, should they leverage the
Parameters, if you're passing parameters through your
methods, that's one place where you can specify them.
And one little thing also.
You can run TestNG in JUnit mode.
TestNG can run JUnit tests.
You just in your XML file to let it know, by the way, these
classes, there are JUnit classes so run them in JUnit
mode which is convenient when you're trying to migrate.
People have found this very, very valuable when they were--
they had thousands of JUnit tests.
They wanted to convert to TestNG and they started by
having the XML file contain all the classes in JUnit mode.
And then, one by one, move them in TestNG mode and so
they can really do something very, very gradual.
Here is a simple TestNG.XML that specifies two things.
The first is just the class that we want to run for this
test. It's called simple test. And we're also telling TestNG
that we only want to run test methods that are in the group
called [? Funk ?] test. So if you happen to have other test
methods that are part of a group that is [? Funk ?] test,
those will not be run by this XML file.
You can see also another advantage of the XML file is
that you can have, for example, a TestNG.database.XML
which is only going to run the database groups.
And again you can move this around.
You can have this in your directory and you know that,
when you make a change, and you've just changed something
in the database, there is no point in running the servlet
parts, because you know those are not going to be broken.
So you're just going to run the TestNG.database.XML which
is going to contain something like this except it's going to
include only the database groups.
We have plugins for IDs.
It was very, very important.
It's always been part of our priority to make sure to
provide, first of all, a really useful and powerful
But we also know that without tools, people will not bite.
They will not be interested if you don't have a lot of
integration with popular tools and IDs come at the very, very
top of the list. So we have an Eclipse plugin which is part
of the distribution.
It's not someone else who was doing it.
We're really doing both.
We're really doing both TestNG at the core
also the eclipse plugin.
Same for the IntelliJ plugin, which basically gives you the
same facilities that you have in your
current ID to run JUnit.
You can point to several classes or several methods and
say run them.
And also it supports some of the additional things.
Like you can tell the plugin, say run me the group database
and some of the other things.
Other than that, it has the standard things that you
expect, including the red/green bar.
Big deal-- the red/green bar.
The very first version of the plugin that I wrote didn't
have the red/green bar.
To me it's always been, it's just a little gimmick that
doesn't really-- it's not really that useful.
You can just-- you run it, you look in the console and you
see all right, everything failed, sorry, everything
pass, zero failure, are you happy.
There was an uproar.
People were really, really pissed.
They really, really wanted their green/red bar.
I was happy to oblige, but it was a very interesting a
realization for me.
And also it leverages a bunch of other things from Eclipse
and also IntelliJ, like the quick fixes, where you can
actually easily convert a JUnit class
into a TestNG class.
And things can happen magically with refactoring.
So it's completely safe and you can see exactly what the
It's pretty innocuous.
Here is what it looks like.
This is an example run from Eclipse.
This is the launch configuration in Eclipse.
I'm just skipping through them quickly they're not really
Plus, they are all a bit stretched out of size.
This is IntelliJ and this is IntelliJ as well.
Other tools, again, we really value a lot integration with
You want to make sure that people are comfortable and
they can reuse what they used to.
Maven, personally I'm not a big fan of Maven.
I think it's made my life extremely hard and it's made
the life of people who worked on this integration extremely
hard as well.
But there is no denying that a lot of people are using it.
And to make matters worse, they actually some of them use
version one and others use version two, so we need to
support both., which also comes with a lot of
interesting problems. Spring, we have a few also helper
classes if you're using Spring.
It's very, very easy to leverage TestNG and Spring
with some of those classes.
And other JUnit frameworks which are pretty easy to
integrate with TestNG such as DbUnit.
Usually integration points-- what these frameworks do with
the JUnit is that they just insert
themselves in the hierarchy.
Instead extending test case, you're going to
extend DbUnit test case.
You're going to make sure that you cull their set up and that
you cull their tear down.
And that's how the whole thing works.
So this kind of thing is really easy
to import to TestNG.
Plenty of options if you want to convert from JUnit
or, as I said, if.
You want to run JUnit test, that's possible.
But if you want to take it to the next level, which is
converting your tests.
There is a tool that will just go through your entire code
base and convert everything.
It will do 95% of the work.
There are still two things that it doesn't do, but it
gets you a long, long way toward converting.
And you can also do this directly from your ID if it's
more convenient for you.
You just do it one class per one class.
You see the source file, you see the changes, live, just
exactly what it takes to convert from one to the other.
Here is another one that I haven't talked about which was
also pretty popular, which actually has been picked up by
JUnit 4 as well.
It concerns this topic of testing for failures.
Something that hasn't really been--
it has been overlooked in terms of testing.
Very often when people hear, you should test your code and
all that, they only emphasize the fact that your method is
supposed to return this value when I pass this value.
And you just have a bunch of tests.
But if we want our software to be [UNINTELLIGIBLE] we also
have to define what happens when you fail, what happens
when you're being passed incorrect values.
Sometimes it's part of the functional specification to
say, it should be throwing this exception, it should be
returning this value.
And there are the times where exceptions are thrown and
there's nothing you can do about it.
And this was actually so prevalent, it was actually a
lot of my tests were actually testing for these exceptions--
passing bogus parameters to methods, catching and making
sure that the [UNINTELLIGIBLE] exception we're thrown-- that
I was thinking, this idiom where you invoke some code and
if it goes into the catch then it's OK.
But if it doesn't go into the catch, then it's a failure,
has always kind of blown my mind.
I've never really understood it.
I did it, I repeated it.
Now I could read it and I was aware of what it means, but it
seems to me it didn't make the intention clear.
It's not very clear what we're testing.
So I thought, hey, let's see what we can do with
Can we make it so that it's very obvious that this test
should be throwing an exception.
And the people who read this code shouldn't freak out if
actually there is an exception thrown
and no error is reported.
So here's an example, where we have this function called
When everything goes well, nothing happens.
But when the number is not validated, it should be
throwing a number format exception.
And the way we specify this with TestNG is you just
specify this in the annotation and this way the code inside
the method is really just the API code.
You don't need to surround [UNINTELLIGIBLE]
the catch, and have this weird no up or empty thing.
You just really put API right there.
Here's another thing that I thought JUnit didn't really
make very easy for me.
I touched on this briefly earlier.
It's rerunning failed tests.
Very often our job, I know part of my job, when I come in
the morning, I check the nightly tests and a lot of
them have failed.
And depending on how important they are, my first task is
going to be to go through these failed
tests and run them.
First of all, put myself in an environment that will allow me
to run them.
Not always easy because the nightly tests are usually run
an environment that is not exactly the one
you're working on.
So it can take some time to get up and running.
And then just make sure that you're only running those
You don't really care about all those that were green.
There is nothing to investigate there.
And so I was wondering so I have this testing framework.
During the night, it ran my tests and it noticed that this
And then it's just telling me this in the morning and it's
not helping me anymore.
You're on your own.
Go debug it.
The testing framework knows that method failed.
Can't it help me only a little bit, to allow me to rerun
maybe just that method?
It turns out that the XML file that I mentioned a few slides
ago is actually pretty useful for that.
What happens when TestNG a test suite and some of those
methods fail, it's going to generate another XML file of
itself that will only contain those failed methods.
So the bottom line is that whenever you want to do this,
which is run you test and then run only those tests that
failed, here is how you do it.
Those last two lines at the end is just, first you ran
your tests on your regular TestNG.XML and then you run it
again, but this time on just testng-failed.xml And then you
have at it and you go debug.
And this time you know exactly that only those methods that
failed are being run.
We're going to dive a bit more into detail for another maybe
ten minutes so I'm going to go a little bit faster to only
focus on what I think is important.
I'm noticing some lovely syntax problems here but let's
I'd like to talk a little bit about data providers in data
Here is an example of test that I have to--
that I've had to write at some point.
I wanted to see that our software was reacting
correctly when we were receiving
user agents from browsers.
And there were certain browsers that were supported.
As you know, the user agent string actually-- right, I
just use simple ones here and MSIE, Firefox and all that--
as you know, user agent strings can be like five or
six lines long with semicolons and a lot of other things.
And you need to know exactly which one you support and have
the good-- the current return code.
With mobile phones now hitting our servers all the time we
actually have even more of those strings and we keep
getting new ones every day.
So this database of things that we support and what error
code we need to return changes pretty much every day.
So here is a kind of a naive way to test for this.
We had this verify user agent support method and that has a
bunch of assert equals.
And that just tests all the strings one by one.
The reason why I said this is not going to scale very well
is that, basically since like I said those user agent
strings get added every day, someone needs to go to that
source file, check it out of the source control system add
then recompile them, and check it in and blah blah.
That seems like a lot of work for something that is just
The code itself, we're just evoking the same API get
return code for and we're doing an equal.
That doesn't change.
We should not have to be recompile whenever we want to
change the data.
So here's a better way to do it.
And here is how TestNG lets you do it.
It uses this annotation called data provider.
The idea is that you can hook up your test method with
another method that is going to supply all these strings.
It's going to supply the strings by passing them in
Let's start with the bottom.
We have this at test annotation.
And this new thing called data provider equals user agents,
which is a reference to the method at the top.
Now you will notice that this test method is receiving
that's new, again, so far it wasn't possible to do this.
And those two parameters, the first is the user agent string
that we want to test. And the second is the code.
and so, of course, the code itself is just assert, get
return code for S equals equals code.
This thing is not going to change.
It's really what our test is.
What is going to change is the implementation
of this data provider.
It's just a Java method that returns things that are going
to be cast into those parameters.
So what this means-- this code, if I run it, TestNG is
actually going to run verify user agent four times.
And for each time, it's going to pass first MSIE, then 200,
and then Firefox, 200, and then Safari, 301.
I think the logic is pretty clear.
You could say well OK fine it looks like it's a bit more
decoupled but we still need to modify the Java code whenever
a new user agent appears.
Well, it's not quite true because what we can do now is
that this method, create user agents, doesn't need to have
those strings hard coded.
You can fetch them from a database.
You can fetch them from an Excel spreadsheet.
You can fetch them from a properties file.
You can fetch them from the network,
from pretty much anywhere.
So once you start externalizing this, then
suddenly your tests become completely independent from
the data that are driven.
This way, whoever in charge of logging all those user agents,
make sure you enter them in the database if we don't have
them, can actually be completely separated from the
person was actually working on the tests.
I have plenty of other examples where this has come
very handy and we've had some very creative work from people
on the mailing list who have come up with ideas.
For example, somebody feeding the content of a properties
file to a method, like a comma separated file, except that
the name of this file depends on the name of the method
And it's like ten lines of Java and
basically you want to--
you write your new method, you pass it through parameters,
you just go in that directory, you add a file that is called
the same name as this method and you put all the values
there and blam, your tests automatically pick them up.
So general idea--
we separate the data from the logic of our tests.
And sometimes your tests don't need to that.
When they do, it is very, very convenient to be able to
externalize this data.
The data itself can come from anything that you can read in
Java., which really means anything.
And you can have as many data providers as you want.
You can put them in base classes, you can share them
They are really no limits to what you can do.
that was an interesting one also.
Like I said earlier, laptops now have two cores, two CPUs
and even now Macs have quads and eights.
So we're seeing really--
we're beginning to leverage now those threads where
whenever you run something and you put it in a thread in
Java, it's going to be running in a separate-- on a separate
processor and separate lightweight thread.
So it's no wonder that we want to leverage this
a little bit more.
And on the other hand, how do we test that the code that we
write that is spawning all those threads is
Well the sad news is that there is no way to really
There is no way to make sure that when you have this piece
of code and it's being used by several threads at the same
time, that you're not doing something stupid.
Very often, it's very easy to do something stupid.
But let's say for example what is it wouldn't be the standard
prototype of a method that test that you have a cache and
whenever you put something in this cache, you do it in a
thread-safe way, which is if several threads are trying to
put something in the cache, it's going to be fine.
So here is what it would look like.
First of all, you're going to create a pool of threads.
I don't know.
You probably want to keep some free so that you can change
this value as time goes by or as you're running
on different machines.
You going to create a bunch of workers.
And all these workers are going to simulate the multiple
threads and they're going to try to put
things in the cache.
We're going to launch the pool of workers.
We're going to wait for termination, for all the
cache.puts to be over.
Some of them are going to fail.
That's a possibility too.
So out of 50 threads that we launch in all, 48 are going to
return and two are going to be blocked for some reason.
We need to abort them.
And then when all of this is done we're going to go into
cache and compare it to what it should look like to make
sure that no data has been thrashed.
I've written this kind of code.
It's really not fun, even with the Java [? util ?]
concurrent and things like it's really a lot of work.
And I thought a testing framework would be nice that
could help me a bit to make this easier.
So here's how it works with TestNG.
Again annotations to the rescue.
There were a lot of things that it turns out we can hide
First of all, the logic of this a test method, all it
does is cache.put.
And we've come a long way.
Suddenly we no longer have to do all these things of
creating the pool of workers and waiting for termination
and all that.
However we are asking TestNG to do that for us.
And the way we do this is we have this at test annotation
again and two little attributes that are new.
The first one is implication [UNINTELLIGIBLE] and the
second one is thread pull size.
I think they're fairly self explanatory, but the idea is
that you're basically telling TestNG to invoke this method
It's going to do that 1000 times.
But on top of that, those 1000 times need to come from 10
So TestNG is going to allocate 10 threads, going to start
banging on your methods, then it's going to release one
thread and is going to pick another one.
It's going to invoke it, until your method has been invoked
It's a pretty solid way to actually really put your code
under a lot of pressure, a lot of thread trashing.
Especially so if you help it by putting a lot of yields to
make sure that there is really going to be a lot of
preemption and things like that.
And again you don't really need to
care about the details.
It's basically a way to tell TestNG, please let's see if my
code is thread safe.
Please test it for me.
Again it's not a guarantee.
There is no way to guarantee.
A lot of this is very non deterministic but if you run
this every night, or maybe every day, dozens of times,
for months on and every time your cache is in a good state,
after a while you can be pretty sure that, by then, all
the combinations of preemption have been run and you can be
Of course you can also reuse an attribute that we've seen
earlier, which is time out.
It comes in very handy here, if you want to make sure,
again, that you know some of the threads are not going to
be lagging for too long.
Obviously a cache.put should succeed in
less than 10 seconds.
If it doesn't something really went wrong.
And again, you can ask TestNG to take care of this for you,
which means that after 10 seconds if one of the threads
is still blocked, you're going to say OK, that's it.
I'm going to skip on a lot things.
I'd like to save some time for the Q & A. You will have
access to these slides, I believe, I think they will be
made available by everyone.
I'd like to do a quick wrap up of what we've done.
Oh, I should have changed that.
We haven't just released TestNG 5.0.
We've just released to TestNG 5.6.
The core of TestNG has been stable for quite a while.
As I said, our work has been focused mostly on tools.
We really want people to be happy.
So far they seem to be happy with the core features that we
have. Once in awhile, we had an attribute to one of the
annotations but the changes are fairly minor.
They're all backward compatible anyway.
But we really want to make a top notch experience with IDEs
and with Maven and with Build and Ant Tasks
and things like that.
We have some scripting language support, which I
didn't cover here but there is absolutely a way for you this
to write your tests and also even drive the logic of tests
TestNG using a scripting language, such as BeanShell.
Multi thread testing, distributed TestNG is
something that's in progress.
It's being done there.
There are a couple of initiatives going on here.
One of them is being done with GigaSpaces.
And of course we, as needed, if there's any JUnit framework
that people are very familiar with and that they will want
to with TestNG, it's pretty easy for us to import and
we've done that for quite a few including EasyMock and a
bunch of others.
so I forgot to mention, of course, it's open source, it's
hosted on java.net.
Website is TestNG.org.
And they are basically two people behind this initiative
myself and someone called Alexandru Popescu who is also
been helping me since pretty much day one on this thing and
who has been doing a fantastic job.
well JUnit a lot of good things about it.
It really helped enable the revolution of making testing
popular and making testing possible on Java.
I also think that it's aged a little bit.
And it's lagging a bit behind.
It hasn't kept up with the complexity of the software
that we develop today.
TestNG has a number of benefits over this which I
And really there is one thing that I would like you to take
away from this talk, whether using TestNG or not is an
option for you.
I really want-- and that's how I started TestNG as a thought
which is, how we shake the foundations of this?
How we start looking at things in a different way?
How can we stop looking at things in a JUnit way and try
to see at it more like a testing way.
There something that I want to test. JUnit forces me to do it
in a certain way and are there other ways that I can do it?
Similarly, there are things that you like TestNG but
you're stuck with JUnit and a lot of people are.
Feel free to go on the JUnit mailing list and say hey could
we support this also.
Every little step like this helps the entire community and
the entire Java community to actually raise and actually
have testing frameworks that actually matched the
complexity of what we're doing.
A little plug--
we have a book coming up.
It will be available in November, which will cover a
lot of these things.
It uses TestNG as another framework
underneath all of this.
But it's really a book about advanced testing and its goes
much deeper details about all of these techniques that I
It's been written by myself and Hani, another person who
is working on TestNG.
And with this, I think we're ready for some
questions and answers.
I hope answers.
AUDIENCE: Now this is working.
Right so TestNG looks like a very exciting new development
compared to JUnit.
Now it's also very Java specific apparently so are you
aware of any initiatives or attempts at adapting TestNG to
test code written in other languages than Java.
CEDRIC BEUST: Only very, very small ones.
I know some people try to do it in Ruby but they had also a
few problems doing annotations in Ruby.
There are several clever things you can do with Ruby to
At this point, people were just experimenting and
dabbling with it, so that didn't go very far.
On the C Sharp and Dot Net part, there is no part of
TestNG but there is NUnit, which also was probably one of
the very first to use annotations.
It is more JUnit side and then the TestNG side.
Though the answer to your question overall is no.
OK, so we'll probably tried to steal ideas from you.
AUDIENCE: I have two questions the first one is this is
obviously written before JUnit 4 came out.
Since they used a lot of your ideas in there what do you
consider the biggest advantages of TestNG if you're
making a decision today?
CEDRIC BEUST: Well so JUnit 4 added an annotations, things
like expecting exceptions and before class, after class.
I still think that they are dealing with the legacy and
they need to have a very progressive
evolution over JUnit 3.
And I think they still fell short of what I
was trying to do.
When I saw JUnit 4, there were still plenty of things that
TestNG was already doing that I was really getting used to
that I feel very strongly.
And I tried to push the [UNINTELLIGIBLE] staff and all
the people who work on JUnit to try
to go in this direction.
And more for ideological reasons and philosophical
reasons and also backward compatibility--
they pushed back on that.
They were not really interested.
I think the fact that JUnit is focused so much on extension
via runners, like whenever you want to do something that is
not part of JUnit.
They tell you, write your own runner which basically means
rewriting the entire logic of the engine and
how methods are invoked.
And I'm not very happy about this because it's
very hard to do.
As soon as you do that, you can no longer use your IDE or
anything like that.
You really know in a world of your own.
And I'm kind of disappointed that they tend to push this
complexity on developers whereas they should be taking
care of the complexities so that will be easy to people.
So again, the benefits, there are plenty of things that they
are adding one by one there's the time out, the
multi thread support.
The fact that there is the separation of other groups are
still not in JUnit, by the way, this is probably the most
popular feature in that people use.
They've taken it to crazy, crazy lengths.
There are people are using sometimes dozens of groups and
they'd like to run--
pinpoint just the one that they run.
and I think that's something that wouldn't be against the
I think it doesn't go against unit testing.
It doesn't compromise any of the principles that they use.
So there are still a few things.
I'm not going to go on a long rant.
I think we have more features.
But again, it's a friendly competition.
AUDIENCE: And the other question is, is there any
equivalent in TestNG to writing your own test suite
using logic as opposed to XML?
CEDRIC BEUST: Yes, yes there is a program called
Programmatic API, where you can just create your TestNG
object in Java.
That's used by a lot of people to write
test inside the container.
So they have the tests running in the application server.
And driven by a servlet, for example, and in there, they
just create their own TestNG instance and they do, you
know, that there is programmatic API to add the
test classes, to specify what groups you want to run, to
specify multi thread and all of that.
You can really everything in Java.
AUDIENCE: Something I was trying out with JUnit which I
also didn't see so often in TestNG is I'm passing in these
parameters to my method and this does not quite fit with
the annotations because the annotations are something
fixed and the parameters to something that
changes all the time.
So for example this time out thing, if I have my test
method to import files and if I import one, then I want to
have this time out and if I import 1000 and I want to have
Do you have any idea or something?
CEDRIC BEUST: There are two sides to the
answer to your question.
The first is data providers allow you to pass parameters
that are created in a Java method, by the data providers.
So already from that point, you can pass any parameter
that you can create in Java.
The other point though, modifying annotations is also
something that I read into at some point.
It wasn't initially--
like, typically I have a timeout.
And I have hundreds of methods that have a timeout but is
hard coded to 10,000 milliseconds.
And then time goes by and suddenly this time out is no
longer enough, because the tests are getting complicated.
What do I do?
Do I go through this in all hundreds of methods
and I modify it?
I introduced something called annotation transformers in
TestNG, where basically if you implement one of those--
TestNG, every time that it reads an annotation, it's
going to ask you, do you want to change any value to this
And then it's going to read it.
So by doing that whenever you need to modify an annotation
dynamically, that's really, really easy to do and you
have-- and you can really do a lot of very cool things like
changing the number of invocation counts dynamically,
depending on you know how much CPU you have,
that kind of thing.
So yeah, the answer is all the TestNG annotations can be
changed and modified at run time.
A group's first class objects can you set properties on
groups and set dependencies between groups directly?
CEDRIC BEUST: Yes.
I didn't have time to touch on dependencies but you can have
methods that depend on other methods, but better methods
that depend on groups.
So they can make sure that all the methods that belong to a
certain groups are run first. And then when they've all been
run and they're all passed, then your
methods can be played.
AUDIENCE: The question was, can I say that one group is
dependent on another?
CEDRIC BEUST: Yes.
Yes, there is a depends on groups attribute for test.
MALE SPEAKER: Any other questions?
AUDIENCE: We've been using TestNG in a couple of our
groups, for like eight months or something, and we're very
happy with it.
So I want to plug it.
I saw on your future slide about distributed TestNG and
was curious if you could quickly describe that and when
you might think it would be available?
CEDRIC BEUST: We can take that off line and I can put you in
touch with the GigaSpace person--
people who are working on it and we've been working on it.
The general idea was to being able to nominate a pool of
slave machines that can run your tests.
And then from the master, we take the testng.xml and see
all the ones, all the tests that are independent from each
other that they can be run really, really separately.
Then you put them over there, send them there, get the
results and collate all the results and present them.
That's the overall idea.
I haven't really kept track of what they've been doing so you
would have to talk to them directly.
But I'll be happy to give you their email address.
AUDIENCE: Sounds good.
And we love TestNG.
CEDRIC BEUST: Thank you.
AUDIENCE: Just a quick question.
So most of the time what we do is actually we have a base
framework of JUnit test. And we exchange it, and then
override certain behavior and everything.
So a lot of the data which we override seems to be, like, if
you will move to TestNG would be coming from XML.
Is there a way to set an inheritance path kind of thing
in the XML itself, like, overriding behavior.
CEDRIC BEUST: Set the inheritance path
AUDIENCE: like I mean seven types, like, my exclusion
group changes based on--
it is basically--
I have a base set of tests where I exclude two tests and
somebody's going to inherit that and run that
test so they want--
CEDRIC BEUST: Can we talk?
Let's take it off line.
Meet me after this and you can explain to me more details.
I'm afraid it's going to go very, very deep there.
But yeah, grab at the end of this talk.
MALE SPEAKER: Anyone else?
Or am I just being blind?
MALE SPEAKER 2: All right, Cedric, thank you very much.