Follow US:

Practice English Speaking&Listening with: GTAC 2007: Cedric Beust - Test NG

Difficulty: 0

SPEAKER 1: Please help me in welcoming Cedric.

CEDRIC BEUST: Thank you.

OK, this works.

Well I still hope this presentation is still cool by

[? Alan's ?] standards even if it's not

exactly what he expected.

So, the title is pretty long.

It's "Beyond JUnit, TestNG and Testing The Next Generation."

I promise I'm going to explain all those items one by one.

And here's a little bit what this presentation is

going to look like.

It looks a bit funny from here.

I'm not even going to tell exactly what

TestNG is right now.

I'm going to ease you into it, if you're not

comfortable with it.

I'll start by talking about JUnit which I assume a lot of

you know and then I'll slowly introduce TestNG and we'll

dive into details.

By the end and depending on how much time we have, maybe

I'll skip on some of the end slides so we can have some

time for questions and answers.

I'll start with JUnit.

Who here is familiar with JUnit?

Can I have a little show of hands?

All right.

I'm not surprised.

Everybody who programs in Java and is a decent Java

developer, knows about JUnit.

Something strange has been happening lately in the

testing area.

I don't if you guys remember, maybe ten years ago or maybe

before you--

so those of you who were there before--

testing was considered kind of a low level activity.

Something that developers didn't really want to have

anything to do with.

No, you write the code but no, I'm too good, I

don't test my code.

And at the very least, maybe I'm just going to write the

code and toss it over the wall to a QA department and tell

them, test my code for me and it's-- that's

not worthy of me.

And please don't bother me, there are bugs.

Well this has changed.

I don't know exactly when.


in 2000, 2001--

it's hard to know exactly-- but at some point, testing

started making its way to our developer's habits.

It became part of what we're doing that, when you write

code, we should probably also provide tests.

A lot of them were early build but it is really part of our

consciousness, that it's really--

we need to be professional about it and we need to

provide tests.

So a lot of things have happened.

We can definitely credit the people who were behind the

extreme programming movement.

They really tried to make test a first class citizen of

engineering-- software engineering.

Test-driven developmental also has become quite popular--

I'll talk a bit more about this later.

And, of course, there were something else.

There was JUnit, which came out at, I think, around 2001.

So what happened?

In 2001 when JUnit came out, we did not have a standard way

to test Java code.

It's hard to remember it.

I can't even remember how I was writing test before that.

The truth is I probably was not.

But until JUnit came out, there was nothing.

We were all doing it in an ad hoc way,

using our own framework.

And we were basically reinventing some of the things

that JUnit formalized when it came out.

So it doesn't need much introduction.

I'm not going to spend too much time on this.

I'm pretty sure you all know JUnit well, although I will be

testing you on this very soon.

It's a very simple testing framework, the concepts are

easy to understand.

You have these set up tear down classes, you have those

test methods that start with the word test. And it took the

Java world by storm.

There were such a need for a standard Java testing

framework that, within a few months, there was no question

that anyone who was testing Java code was using JUnit.

So it's a very easy to put in place, very easy to


You can pretty much read two or three pages on how it works

and you can get up and running and you

write your first tests.

The concepts are simple, set up, tear down.

You have test runners.

You have this concept of a suite, which can have a

hierarchy of tests that are smaller and smaller.

And it has a huge ecosystem.

There are a lot of add-ons available to do pretty much

anything you want, to test GUIs, to test database, to

test servlets.

There's a wealth of material, of books and articles and

conferences like this one, plenty of experts on

newsgroups and mailing lists.

The resources to find JUnit documentation are really,

really endless.

OK so here's the test. This is a very simple

to JUnit test case.

It has two test methods, one field and these two test

methods do exactly the same thing.

They increment the field and then they try to verify that

the field is equal to one.

Just give you a few seconds to understand

what's going on here.

And I'm going to ask you, a quick show of hands again, who

thinks that these tests pass?

I have one, two shy hands, three hands.

Who thinks that this test fails?

A lot more.

Who is not paying attention?


So, that's most of you.

OK I am not going to do a demo.

You can trust me on this.

Or you can just quickly run it on your laptop.

The truth is that this passes.

So to the two or three who raised there hands, you

actually know JUnit pretty well.

Those who raised hands to the second question, you might

have a few things to learn about JUnit and I'm hoping to

dispel some of those things.

Let's go back again.

How can this pass?

Isn't it a bit surprising?

If, in Java, I do something like this, I have a field and

I cull two methods, and each method

increments is the field.

By the time I'm done, this field should be equal two.

But if you run it was JUnit, it's equal to one.

What's the only explanation?

Well, my class must be reinitialized There must be a

new instance that gets created every time.

It's pretty easy to verify.

It's exactly what's happening.

JUnit is reinstantiating your class every single time before

it invokes a test method.

I was bitten by that.

I mean that was a bug that really baffled me.

And then I went into the JUnit code and all that.

I found out this thing that made it obvious that the

behavior was that.

And then I wondered, it can't be a bug.

JUnit's been around for years.

If it was not designed to be this way, we

would know about it.

So I did some more homework exchange of emails with Erich

Gamma and a bunch of other people.

And they all confirmed, yes, it's by design.

And here is the reason, we want your test methods to be

independent from each other.

We don't want them to have to rely on a field having a

certain value that has been set by previous tests.

You should be able, at all times, to be pick a test

method and run it.

And it doesn't depend on anything.

All right, that sounds like a good idea.

I can see some value in this.

But there was still something that was nagging me, I was

thinking, but I still want to do this.

I still want it to maintain some states.

Does that make me an evil person?

Was my code really ugly?

Shouldn't it have been doing this?

What's wrong with that?

The example that I was using back then was, I was writing

test for XML file and I had a huge XML file that was

parsing, that took a while.

Out of this, I was building a tree, or a dome, or whatever--

something pretty costly.

And then my test methods were just making sure that the XML

file was correct.

But because of that, I had basically to reparse my file

every time before I invoked a test method.

I said well that doesn't seem right to me.

It's a read only file.

I'm just going to be create it once and then I want to run my

test. So what do I do?

Well, I'm going to work around this little clever thing that

JUnit has put in my way.

And I'm going to use a static field.

I'm going to make sure that it's only initialized once.

And then the other times, it's not going to

reinitialize itself.

All right, so that works.

That's why I know static fields work.

But I started feeling a bit uncomfortable about this

because now I am trying to work around things,

introducing statics, which I don't really like a lot.

Introducing statics has some downsides.

It's been hard to make it threat safe.

You never really know when it gets initialized.

If you run the same thing several times, in the same

class loader, in the same JDM, the state is going to remain--

which is exactly what JUnit was trying to avoid.

So I started pining over those work arounds, and the static

things I noticed were everywhere in our code base

because of that.

And some of the things I noticed as months went by, I

started examining how I felt about JUnit.

How do I write an individual test method?

I have 20 test methods in my class.

One of them is failing and I just want to run that one so I

can debug what's going on.

Back then, when we didn't have that much support in IDEs,

there were not a lot of solutions for that.

I was just commenting out those 19 out of 20 methods, so

I would just run that method.

You can do a little bit better with IDEs now.

But I thought again, shouldn't the testing framework help me

a little bit there?

Can I just ask JUnit please run just this test method

instead of having to recompile?

So all this let me down a path where I started questioning a

little bit what I was doing with JUnit.

And then actually whether there I was really doing

testing on my code or doing JUnit testing of my code.

My interpretation of what's going on is that in some way

JUnit was a victim of its own success.

It became the de facto standard because there was

nothing else.

But it wasn't really ready to become that.

And also JUnit is a unit testing framework.

Unit testing is really small units of test, at the scale of

the class usually.

It's important to have those tests.

But when it comes down to bigger testing--

there are various names, functional testing, system

testing, integration testing, basically when you're trying

to test how several classes work together--

I started realizing that JUnit didn't really help

me get my job done.

Because it was focused on unit testing it didn't help me for

these other types of testings.

And I wanted to be able to both.

I felt a bit frustrated by the configuration control that I

had, so there are set up and tear down methods which wrap

each test method.

I realized, well, sometimes I want to do a class level.

I want to initialize something for the class once and then I

don't need to initialize again.

How about suite level?

Something that's just initialized once the beginning

of my suite and never again after that.

And I realized, well, set up and tear down, it's o k but I

want a bit more.

It was also getting a bit old.

It hadn't been updated in a while.

This is not quite true, as you know JUnit 4 is out--

but work resumed on JUnit just a year, a

year and a half ago--

which means between 2001 plus and 2005, we were working with

JUnit 3.8 and it hadn't been changed at all.

Hadn't kept up with things like cert and generics and a

lot of other things that we're used to using modern Java.

I thought JUnit was a bit intrusive.

It forced me to write my test and forced me to use certain

names and extend certain classes and use certain idioms

that were not very intuitive as I just showed.

And I also thought that in itself, JUnit.jar the little,

I think, 50k or 60k, probably a bit more now, was really,

really minimal.

I really needed to add a lot of other things in order to

really make my job easier.

OK so that was a quick discussion about JUnit just to

try to make you think a little bit, and make you question

what you're doing.

And now let's try to see there is a way that we can find a

few workarounds around this.

And then, without more ado, I introduce you to TestNG--

which is a testing framework that is the result of all

these little hardships that I felt through JUnit--

which I created three years ago, three and a half years

ago-- just trying to experiment a little bit.

Back then, annotations--

we were still working on annotations.

They were not in the JDK yet.

JDK 5 wasn't out.

So I thought maybe I'll try to play with those annotations

and trying to see all these different things.

So here is, in a nutshell, some of the

main features of TestNG.

Based on annotations, I'll show you an example of that.

Anyone here who is not familiar with Java notations?

Never seen any Java annotations?

OK there are-- most of you are.

Maybe you don't use them, because you are not on JDK 5

but I'm sure a lot of you are aware of what it is and if

anything if you're stuck with JDK 1.4, you're hoping, oh,

gosh I can't use that, but at least I know what they are.

Flexible run time configuration--

I'll explain what that means later.

Another thing, also, that I thought was very important is

being able to put my tests in groups, not just classes, but

really in groups that I can call whatever I want,

database, GUI--

I'll show examples of that.

Some other more practical features that I wanted, again,

to do my functional testing on top of my unit testing, such

as dependent methods, being able to test my code and make

sure that it's multi threaded, being able to

run my test in parallel.

And now even laptops now have multi cores and multi CPUs.

And when I run my tests, and all this test methods are put

independent from each other, it would be really silly not

to have them run on separate threads and just leverage what

my laptop has to offer.

So I wanted to build that in also.

And I wanted also think the API guide to be flexible, so

that people feel comfortable putting their own things,

creating new reporters, and new runners and things like.

All right, a bit more code.

Here are two examples of TestNG.

And I did mention that in the list earlier, but TestNG works

both on JDK 1.4 and JDK 5, which might surprise you a

little bit, because I just said it works on annotations.

I said, how can it work on annotations and JDK 1.4?

Well the answer is here.

Just using Javadocs, for those of you who were using

annotations, before they became cool--

with things like XDoclet.

That's how XDoclet introduced annotations to the Java world,

just using Javadoc and using tags.

So it's really easy to simulate exactly what the JDK

5 annotations give us, using these comments.

Of course it's a lot less safe because the compiler has no

idea that these things are here.

You can misspell them and then the tool is

going to miss them.

So it's convenient if all you have, if you have JDK 1.4 and

you can't upgrade, if you have a choice between 1.4 and 5, I

strongly suggest, use 5.

In this example, we have two things.

Let's start with one of at the bottom, which basically says

that the method called server is running is a test. In

JUnit, you would say that just by calling it

test server is running.

But now with annotations, we no longer

need this little trick.

We can call the method whatever we want and we use

the annotation to tell TestNG, hey, by the way, this is a

test method.

And the method at the top is what I call a

configuration method.

It's like a set up in JUnit, except that this one is a

before class.

It's like a setup, except that it's going to run before you

run any tests on that class.

And it's only going to run once.

If you have ten methods on your class, before that first

method is invoked, we're going to invoke in it and then it's

going to call the ten methods and you can also specify an

after class to do some cleanup at the class level.

Here's what it looks like on JDK 5.

It's a little bit more concise.

But most of all it is type six and you can use your IDE, you

can use completion, you can start typing at before and the

ID's going to tell you all the annotations that you can use.

A lot more convenient, I think more

powerful, also, more readable.

And that's why I recommend this approach if

you have this option.

From now on, I will just show these examples in JDK 5

format, but I whatever you can to do JDK 5 , you can do in

JDK 1.4 too.

So a few remarks on the code that we just saw.

A few things, a few immediate benefits that we have. The

first is that we don't need to extend a base class.

You remember with JUnit, you need to extend this thing

called test case.

This is how you enable the magic of being able to have

set up and tear down and things like that, have the

assert functions.

We no longer need to do that, which is kind of convenient

because extending a base class is a pretty big deal in Java.

We only have one shot at that.

If you want to extend an implementation, you can only

have one base class.

And when you use a framework that forces you to have this

base class, sometimes it gets in your way and you have to

find your way to put your own class in between.

Again no need to start your method with test. You don't

need to call your configuration methods setUp

and tearDown.

Again by the magic of annotations, we can just

pinpoint those.

And something else which is not shown in the example, but

on which I will get back there soon is that test methods can

receive parameters.

Yeah, how about that?

Did it ever occur to anyone here that, all right, I love

JUnit and wait a minute, I'd like to pass a parameter to

this method.

I can do this in Java.

I can pass a string, I can pass an integer, I can pass a

Java object.

How do you do that in JUnit?

Well it's pretty convoluted.

It's possible but it's convoluted.

And when I started TestNG, I was thinking, well, I'd like

to do it exactly like I do it in Java, so let's

see where we can--

how far we can take this idea.

Turns out, it works pretty well.

Here are some of the annotations that the TestNG

framework offers.

There are actually a couple more, so it's really not that

many, especially since-- so you have the test when you

have all this configuration before class, after class,

before method, after method.

There are quite a few.

It's not really a problem to have a lot.

It just gives you a lot of flexibility to know exactly

when your initialization needs to take care--

needs to take place, sorry.

If you want to start your application server, it's

obviously something that's going to take it-- to start at

the very beginning of your suite.

If you want to just initialize the state of your class,

you're going to do it in a before method and after a

method class--

no, an annotation.

DataProvider is another interesting one that I will

describe very shortly which enables something that is

referred to as data driven testing.

Just to whet your appetite, it's the kind of testing where

the code of your test is actually pretty constant,

pretty simple and what changes is the data that you're

feeding to this test. And Factory, which allows you to

go wild and pretty much create all your test instances, in

case it's not enough to just have them.

Terminology because of all these configuration things, I

had to come up with a pretty strict terminology in order to

match those concepts.

So it's actually pretty simple.

There's just one addition, compared to

what you're used to.

At the very top is a suite, which is what contains your

entire test environment.

And each suite is made of several tests.

And each test is made of several classes and each class

is made of several methods.

And the reason why this little indentation is useful is that

you have configurations that are possible

around each of them.

So we have a before suite, after suite.

We have a before test after test. We have before class,

after class and before method after method.

And even one more than I have listed here.

Again think of it as just a way for you to be very, very

fine grained in the way you can initialize your tests.

A quick list of pretty much exactly what I just said.

So the only thing that we are familiar with from JUnit is

before method and after method--

equivalent in JUnit is set up and tear down.

Everybody is familiar with that.

These other things are actually new in TestNG.

And just for completion--

for completeness, sorry, the before class after class

annotation we're introduced in JUnit 4.

So that slide is no longer quite current.

JUnit 4 has support for before class and after class but, as

far as I know, they still don't have before suite and

before test either.

Another benefit of annotations is that they can have, I'm

going to call them parameters but their real--

the real technical term an attributes.

But it's basically a parameter that you can pass to an

annotation to add some extra information.

This is a bit nebulous for those of you who never used


You can just take a look at the end of the listing at the

bottom, where we see the at test annotation.

But this time instead of being all itself, it actually has

three attributes.

One that is called groups, the other that is called time out,

the other thing that is called depends on groups.

And each of them is assigned a certain value.

You can probably guess what those do.

But I will explain this very soon.

Groups gives you a list of groups that you want this

method to belong to.

Another hint of what you can do is that, instead of having

this method belong to a certain class or certain

package, which so far was the only way that we could really

say this method test databases.

So I'm going to put it in the package called

com.example.test.database So all right,

it's better than nothing.

I know if I want to run my database test, I'm going to

tell JUnit to run everything in the package com example db.

But I realized this is kind of unidimensional.

Very often, I want to tag more attributes to my method.

My method is not just a database.

Actually it happens that method runs pretty quickly.

I like to put it in a group called fast. I'd like also

this method to be part of--

or maybe that one is slow and it runs only weekends.

So I'd like to say, just put it in the group, weekends.

Just to put some categories in the methods and what this buys

us is that once we have compiled our entire system,

then we can switch to the run time part and tell TestNG run

all the fast methods, run all the database methods, run all

the nightly methods.

And what this buys us is that we don't need

to recompile anything.

First of all, it's already burned.

It's already part of the class file.

And second, as you can see, methods can

belong to several groups.

They are no longer constrained by the package part.

And you no longer need to name your package a certain time

and you have this other dimension where the package is

one thing and then the list of groups is another thing.

Parameter is another example.

That's how we pass parameters.

I'll give you an example soon.

It depends on group timeout.

Timeout is a pretty useful one also--

which is basically a way to tell TestNG that if this

method doesn't return within the timeout specified, say 10

seconds, TestNG should abort the method, mark it as a

failure and then move on.

It can be pretty convenient when you are making will test

on heavy network, or under a heavy load and you know, some

things can work.

Or part of the functional specification of the method is

that it should really read your return within one second.

And if it doesn't, then something is wrong and we have

a regression.

There is a little XML file that is pretty

important with TestNG.

It's basically what contains the entire

description of your tests.

Before I go any further, let me just specify for people who

are a bit allergic to XML, I'm sure there are

plenty in this room--

enough XML, I can't read XML any more--

this is optional.

There are plenty of ways that you can avoid doing this.

The bottom line is that a lot of people who are truly very

hostile to XML end up actually using the XML file because

it's very convenient.

It works very well with TestNG.

But if you don't like it, you can totally avoid it.

The interest of this is that it's very easy to have an

externalized description of all your test classes, all the

groups that you want to run, all the various parameters,

like the timeouts and things that you are

passing to your methods.

It's all contained in this XML file, which you can email your

coworkers when you're trying to have them reproduce a

certain configuration, say hey, can you run this XML

file, you will see it will fail at the fifth implication

of that method.

It's not something that's very easy with JUnit, again,

because it's only described in code.

And you can't really have a capture of an environment

that's failing and then ask someone to rerun the exact

same scenario.

So what this XML file contains the list of all the test

classes and the test methods that you want to run, which

groups you want to include which

groups you want to exclude.

Another little trick, I'm going to show you an example

that's a pretty powerful idea also.

You can also define additional groups to make

those big meta groups.

Also in XML, you can also specify how your

test should be run.

Do you want to run these ten classes in parallel or should

they be run sequentially, should they leverage the

current machine.

Parameters, if you're passing parameters through your

methods, that's one place where you can specify them.

And one little thing also.

You can run TestNG in JUnit mode.

TestNG can run JUnit tests.

You just in your XML file to let it know, by the way, these

classes, there are JUnit classes so run them in JUnit

mode which is convenient when you're trying to migrate.

People have found this very, very valuable when they were--

they had thousands of JUnit tests.

They wanted to convert to TestNG and they started by

having the XML file contain all the classes in JUnit mode.

And then, one by one, move them in TestNG mode and so

they can really do something very, very gradual.

Here is a simple TestNG.XML that specifies two things.

The first is just the class that we want to run for this

test. It's called simple test. And we're also telling TestNG

that we only want to run test methods that are in the group

called [? Funk ?] test. So if you happen to have other test

methods that are part of a group that is [? Funk ?] test,

those will not be run by this XML file.

You can see also another advantage of the XML file is

that you can have, for example, a TestNG.database.XML

which is only going to run the database groups.

And again you can move this around.

You can have this in your directory and you know that,

when you make a change, and you've just changed something

in the database, there is no point in running the servlet

parts, because you know those are not going to be broken.

So you're just going to run the TestNG.database.XML which

is going to contain something like this except it's going to

include only the database groups.

We have plugins for IDs.

It was very, very important.

It's always been part of our priority to make sure to

provide, first of all, a really useful and powerful

testing framework.

But we also know that without tools, people will not bite.

They will not be interested if you don't have a lot of

integration with popular tools and IDs come at the very, very

top of the list. So we have an Eclipse plugin which is part

of the distribution.

It's not someone else who was doing it.

We're really doing both.

We're really doing both TestNG at the core

also the eclipse plugin.

Same for the IntelliJ plugin, which basically gives you the

same facilities that you have in your

current ID to run JUnit.

You can point to several classes or several methods and

say run them.

And also it supports some of the additional things.

Like you can tell the plugin, say run me the group database

and some of the other things.

Other than that, it has the standard things that you

expect, including the red/green bar.

Big deal-- the red/green bar.

The very first version of the plugin that I wrote didn't

have the red/green bar.

To me it's always been, it's just a little gimmick that

doesn't really-- it's not really that useful.

You can just-- you run it, you look in the console and you

see all right, everything failed, sorry, everything

pass, zero failure, are you happy.

There was an uproar.

People were really, really pissed.

They really, really wanted their green/red bar.

I was happy to oblige, but it was a very interesting a

realization for me.

And also it leverages a bunch of other things from Eclipse

and also IntelliJ, like the quick fixes, where you can

actually easily convert a JUnit class

into a TestNG class.

And things can happen magically with refactoring.

So it's completely safe and you can see exactly what the

changes are.

It's pretty innocuous.

Here is what it looks like.

This is an example run from Eclipse.

This is the launch configuration in Eclipse.

I'm just skipping through them quickly they're not really

that great.

Plus, they are all a bit stretched out of size.

This is IntelliJ and this is IntelliJ as well.

Other tools, again, we really value a lot integration with

popular tools.

You want to make sure that people are comfortable and

they can reuse what they used to.

Maven, personally I'm not a big fan of Maven.

I think it's made my life extremely hard and it's made

the life of people who worked on this integration extremely

hard as well.

But there is no denying that a lot of people are using it.

And to make matters worse, they actually some of them use

version one and others use version two, so we need to

support both., which also comes with a lot of

interesting problems. Spring, we have a few also helper

classes if you're using Spring.

It's very, very easy to leverage TestNG and Spring

with some of those classes.

And other JUnit frameworks which are pretty easy to

integrate with TestNG such as DbUnit.

Usually integration points-- what these frameworks do with

the JUnit is that they just insert

themselves in the hierarchy.

Instead extending test case, you're going to

extend DbUnit test case.

You're going to make sure that you cull their set up and that

you cull their tear down.

And that's how the whole thing works.

So this kind of thing is really easy

to import to TestNG.

Plenty of options if you want to convert from JUnit

or, as I said, if.

You want to run JUnit test, that's possible.

But if you want to take it to the next level, which is

converting your tests.

There is a tool that will just go through your entire code

base and convert everything.

It will do 95% of the work.

There are still two things that it doesn't do, but it

gets you a long, long way toward converting.

And you can also do this directly from your ID if it's

more convenient for you.

You just do it one class per one class.

You see the source file, you see the changes, live, just

exactly what it takes to convert from one to the other.

Here is another one that I haven't talked about which was

also pretty popular, which actually has been picked up by

JUnit 4 as well.

It concerns this topic of testing for failures.

Something that hasn't really been--

it has been overlooked in terms of testing.

Very often when people hear, you should test your code and

all that, they only emphasize the fact that your method is

supposed to return this value when I pass this value.

And you just have a bunch of tests.

But if we want our software to be [UNINTELLIGIBLE] we also

have to define what happens when you fail, what happens

when you're being passed incorrect values.

Sometimes it's part of the functional specification to

say, it should be throwing this exception, it should be

returning this value.

And there are the times where exceptions are thrown and

there's nothing you can do about it.

And this was actually so prevalent, it was actually a

lot of my tests were actually testing for these exceptions--

passing bogus parameters to methods, catching and making

sure that the [UNINTELLIGIBLE] exception we're thrown-- that

I was thinking, this idiom where you invoke some code and

if it goes into the catch then it's OK.

But if it doesn't go into the catch, then it's a failure,

has always kind of blown my mind.

I've never really understood it.

I did it, I repeated it.

Now I could read it and I was aware of what it means, but it

seems to me it didn't make the intention clear.

It's not very clear what we're testing.

So I thought, hey, let's see what we can do with

annotations here.

Can we make it so that it's very obvious that this test

should be throwing an exception.

And the people who read this code shouldn't freak out if

actually there is an exception thrown

and no error is reported.

So here's an example, where we have this function called

validate number.

When everything goes well, nothing happens.

But when the number is not validated, it should be

throwing a number format exception.

And the way we specify this with TestNG is you just

specify this in the annotation and this way the code inside

the method is really just the API code.

You don't need to surround [UNINTELLIGIBLE]

the catch, and have this weird no up or empty thing.

You just really put API right there.

Here's another thing that I thought JUnit didn't really

make very easy for me.

I touched on this briefly earlier.

It's rerunning failed tests.

Very often our job, I know part of my job, when I come in

the morning, I check the nightly tests and a lot of

them have failed.

And depending on how important they are, my first task is

going to be to go through these failed

tests and run them.

First of all, put myself in an environment that will allow me

to run them.

Not always easy because the nightly tests are usually run

an environment that is not exactly the one

you're working on.

So it can take some time to get up and running.

And then just make sure that you're only running those

failed tests.

You don't really care about all those that were green.

There is nothing to investigate there.

And so I was wondering so I have this testing framework.

During the night, it ran my tests and it noticed that this

method failed.

And then it's just telling me this in the morning and it's

not helping me anymore.

You're on your own.

Go debug it.

The testing framework knows that method failed.

Can't it help me only a little bit, to allow me to rerun

maybe just that method?

It turns out that the XML file that I mentioned a few slides

ago is actually pretty useful for that.

What happens when TestNG a test suite and some of those

methods fail, it's going to generate another XML file of

itself that will only contain those failed methods.

So the bottom line is that whenever you want to do this,

which is run you test and then run only those tests that

failed, here is how you do it.

Those last two lines at the end is just, first you ran

your tests on your regular TestNG.XML and then you run it

again, but this time on just testng-failed.xml And then you

have at it and you go debug.

And this time you know exactly that only those methods that

failed are being run.

We're going to dive a bit more into detail for another maybe

ten minutes so I'm going to go a little bit faster to only

focus on what I think is important.

I'm noticing some lovely syntax problems here but let's

ignore them.

I'd like to talk a little bit about data providers in data

driven testing.

Here is an example of test that I have to--

that I've had to write at some point.

I wanted to see that our software was reacting

correctly when we were receiving

user agents from browsers.

And there were certain browsers that were supported.

As you know, the user agent string actually-- right, I

just use simple ones here and MSIE, Firefox and all that--

as you know, user agent strings can be like five or

six lines long with semicolons and a lot of other things.

And you need to know exactly which one you support and have

the good-- the current return code.

With mobile phones now hitting our servers all the time we

actually have even more of those strings and we keep

getting new ones every day.

So this database of things that we support and what error

code we need to return changes pretty much every day.

So here is a kind of a naive way to test for this.

We had this verify user agent support method and that has a

bunch of assert equals.

And that just tests all the strings one by one.

The reason why I said this is not going to scale very well

is that, basically since like I said those user agent

strings get added every day, someone needs to go to that

source file, check it out of the source control system add


then recompile them, and check it in and blah blah.

That seems like a lot of work for something that is just

really data.

The code itself, we're just evoking the same API get

return code for and we're doing an equal.

That doesn't change.

We should not have to be recompile whenever we want to

change the data.

So here's a better way to do it.

And here is how TestNG lets you do it.

It uses this annotation called data provider.

The idea is that you can hook up your test method with

another method that is going to supply all these strings.

It's going to supply the strings by passing them in


Let's start with the bottom.

We have this at test annotation.

And this new thing called data provider equals user agents,

which is a reference to the method at the top.

Now you will notice that this test method is receiving


that's new, again, so far it wasn't possible to do this.

And those two parameters, the first is the user agent string

that we want to test. And the second is the code.

and so, of course, the code itself is just assert, get

return code for S equals equals code.

This thing is not going to change.

It's really what our test is.

What is going to change is the implementation

of this data provider.

It's just a Java method that returns things that are going

to be cast into those parameters.

So what this means-- this code, if I run it, TestNG is

actually going to run verify user agent four times.

And for each time, it's going to pass first MSIE, then 200,

and then Firefox, 200, and then Safari, 301.

I think the logic is pretty clear.

You could say well OK fine it looks like it's a bit more

decoupled but we still need to modify the Java code whenever

a new user agent appears.

Well, it's not quite true because what we can do now is

that this method, create user agents, doesn't need to have

those strings hard coded.

You can fetch them from a database.

You can fetch them from an Excel spreadsheet.

You can fetch them from a properties file.

You can fetch them from the network,

from pretty much anywhere.

So once you start externalizing this, then

suddenly your tests become completely independent from

the data that are driven.

This way, whoever in charge of logging all those user agents,

make sure you enter them in the database if we don't have

them, can actually be completely separated from the

person was actually working on the tests.

I have plenty of other examples where this has come

very handy and we've had some very creative work from people

on the mailing list who have come up with ideas.

For example, somebody feeding the content of a properties

file to a method, like a comma separated file, except that

the name of this file depends on the name of the method

being invoked.

And it's like ten lines of Java and

basically you want to--

you write your new method, you pass it through parameters,

you just go in that directory, you add a file that is called

the same name as this method and you put all the values

there and blam, your tests automatically pick them up.

So general idea--

we separate the data from the logic of our tests.

And sometimes your tests don't need to that.

When they do, it is very, very convenient to be able to

externalize this data.

The data itself can come from anything that you can read in

Java., which really means anything.

And you can have as many data providers as you want.

You can put them in base classes, you can share them

among tests.

They are really no limits to what you can do.

Thread safety--

that was an interesting one also.

Like I said earlier, laptops now have two cores, two CPUs

and even now Macs have quads and eights.

So we're seeing really--

we're beginning to leverage now those threads where

whenever you run something and you put it in a thread in

Java, it's going to be running in a separate-- on a separate

processor and separate lightweight thread.

So it's no wonder that we want to leverage this

a little bit more.

And on the other hand, how do we test that the code that we

write that is spawning all those threads is

multi-thread safe.

Well the sad news is that there is no way to really

assert that.

There is no way to make sure that when you have this piece

of code and it's being used by several threads at the same

time, that you're not doing something stupid.

Very often, it's very easy to do something stupid.

But let's say for example what is it wouldn't be the standard

prototype of a method that test that you have a cache and

whenever you put something in this cache, you do it in a

thread-safe way, which is if several threads are trying to

put something in the cache, it's going to be fine.

So here is what it would look like.

First of all, you're going to create a pool of threads.

How many?

I don't know.

You probably want to keep some free so that you can change

this value as time goes by or as you're running

on different machines.

You going to create a bunch of workers.

And all these workers are going to simulate the multiple

threads and they're going to try to put

things in the cache.

We're going to launch the pool of workers.

We're going to wait for termination, for all the

cache.puts to be over.

Some of them are going to fail.

That's a possibility too.

So out of 50 threads that we launch in all, 48 are going to

return and two are going to be blocked for some reason.

We need to abort them.

And then when all of this is done we're going to go into

cache and compare it to what it should look like to make

sure that no data has been thrashed.

I've written this kind of code.

It's really not fun, even with the Java [? util ?]

concurrent and things like it's really a lot of work.

And I thought a testing framework would be nice that

could help me a bit to make this easier.

So here's how it works with TestNG.

Again annotations to the rescue.

There were a lot of things that it turns out we can hide

in annotations.

First of all, the logic of this a test method, all it

does is cache.put.

And we've come a long way.

Suddenly we no longer have to do all these things of

creating the pool of workers and waiting for termination

and all that.

However we are asking TestNG to do that for us.

And the way we do this is we have this at test annotation

again and two little attributes that are new.

The first one is implication [UNINTELLIGIBLE] and the

second one is thread pull size.

I think they're fairly self explanatory, but the idea is

that you're basically telling TestNG to invoke this method

1000 times.

It's going to do that 1000 times.

But on top of that, those 1000 times need to come from 10

different threads.

So TestNG is going to allocate 10 threads, going to start

banging on your methods, then it's going to release one

thread and is going to pick another one.

It's going to invoke it, until your method has been invoked

1000 times.

It's a pretty solid way to actually really put your code

under a lot of pressure, a lot of thread trashing.

Especially so if you help it by putting a lot of yields to

make sure that there is really going to be a lot of

preemption and things like that.

And again you don't really need to

care about the details.

It's basically a way to tell TestNG, please let's see if my

code is thread safe.

Please test it for me.

Again it's not a guarantee.

There is no way to guarantee.

A lot of this is very non deterministic but if you run

this every night, or maybe every day, dozens of times,

for months on and every time your cache is in a good state,

after a while you can be pretty sure that, by then, all

the combinations of preemption have been run and you can be

pretty safe.

Of course you can also reuse an attribute that we've seen

earlier, which is time out.

It comes in very handy here, if you want to make sure,

again, that you know some of the threads are not going to

be lagging for too long.

Obviously a cache.put should succeed in

less than 10 seconds.

If it doesn't something really went wrong.

And again, you can ask TestNG to take care of this for you,

which means that after 10 seconds if one of the threads

is still blocked, you're going to say OK, that's it.

All right.

I'm going to skip on a lot things.

I'd like to save some time for the Q & A. You will have

access to these slides, I believe, I think they will be

made available by everyone.

I'd like to do a quick wrap up of what we've done.

Oh, I should have changed that.

We haven't just released TestNG 5.0.

We've just released to TestNG 5.6.

The core of TestNG has been stable for quite a while.

As I said, our work has been focused mostly on tools.

We really want people to be happy.

So far they seem to be happy with the core features that we

have. Once in awhile, we had an attribute to one of the

annotations but the changes are fairly minor.

They're all backward compatible anyway.

But we really want to make a top notch experience with IDEs

and with Maven and with Build and Ant Tasks

and things like that.

We have some scripting language support, which I

didn't cover here but there is absolutely a way for you this

to write your tests and also even drive the logic of tests

TestNG using a scripting language, such as BeanShell.

Multi thread testing, distributed TestNG is

something that's in progress.

It's being done there.

There are a couple of initiatives going on here.

One of them is being done with GigaSpaces.

And of course we, as needed, if there's any JUnit framework

that people are very familiar with and that they will want

to with TestNG, it's pretty easy for us to import and

we've done that for quite a few including EasyMock and a

bunch of others.

Download information--

so I forgot to mention, of course, it's open source, it's

hosted on

Website is

And they are basically two people behind this initiative

myself and someone called Alexandru Popescu who is also

been helping me since pretty much day one on this thing and

who has been doing a fantastic job.


well JUnit a lot of good things about it.

It really helped enable the revolution of making testing

popular and making testing possible on Java.

I also think that it's aged a little bit.

And it's lagging a bit behind.

It hasn't kept up with the complexity of the software

that we develop today.

TestNG has a number of benefits over this which I

described earlier.

And really there is one thing that I would like you to take

away from this talk, whether using TestNG or not is an

option for you.

I really want-- and that's how I started TestNG as a thought


which is, how we shake the foundations of this?

How we start looking at things in a different way?

How can we stop looking at things in a JUnit way and try

to see at it more like a testing way.

There something that I want to test. JUnit forces me to do it

in a certain way and are there other ways that I can do it?

Similarly, there are things that you like TestNG but

you're stuck with JUnit and a lot of people are.

Feel free to go on the JUnit mailing list and say hey could

we support this also.

Every little step like this helps the entire community and

the entire Java community to actually raise and actually

have testing frameworks that actually matched the

complexity of what we're doing.

A little plug--

we have a book coming up.

It will be available in November, which will cover a

lot of these things.

It uses TestNG as another framework

underneath all of this.

But it's really a book about advanced testing and its goes

much deeper details about all of these techniques that I

just mentioned.

It's been written by myself and Hani, another person who

is working on TestNG.

And with this, I think we're ready for some

questions and answers.

I hope answers.

AUDIENCE: Now this is working.

Right so TestNG looks like a very exciting new development

compared to JUnit.

Now it's also very Java specific apparently so are you

aware of any initiatives or attempts at adapting TestNG to

test code written in other languages than Java.

CEDRIC BEUST: Only very, very small ones.

I know some people try to do it in Ruby but they had also a

few problems doing annotations in Ruby.

There are several clever things you can do with Ruby to

assimilate annotations.

At this point, people were just experimenting and

dabbling with it, so that didn't go very far.

On the C Sharp and Dot Net part, there is no part of

TestNG but there is NUnit, which also was probably one of

the very first to use annotations.

It is more JUnit side and then the TestNG side.

Though the answer to your question overall is no.

OK, so we'll probably tried to steal ideas from you.

AUDIENCE: I have two questions the first one is this is

obviously written before JUnit 4 came out.

Since they used a lot of your ideas in there what do you

consider the biggest advantages of TestNG if you're

making a decision today?

CEDRIC BEUST: Well so JUnit 4 added an annotations, things

like expecting exceptions and before class, after class.

I still think that they are dealing with the legacy and

they need to have a very progressive

evolution over JUnit 3.

And I think they still fell short of what I

was trying to do.

When I saw JUnit 4, there were still plenty of things that

TestNG was already doing that I was really getting used to

that I feel very strongly.

And I tried to push the [UNINTELLIGIBLE] staff and all

the people who work on JUnit to try

to go in this direction.

And more for ideological reasons and philosophical

reasons and also backward compatibility--

they pushed back on that.

They were not really interested.

I think the fact that JUnit is focused so much on extension

via runners, like whenever you want to do something that is

not part of JUnit.

They tell you, write your own runner which basically means

rewriting the entire logic of the engine and

how methods are invoked.

And I'm not very happy about this because it's

very hard to do.

As soon as you do that, you can no longer use your IDE or

anything like that.

You really know in a world of your own.

And I'm kind of disappointed that they tend to push this

complexity on developers whereas they should be taking

care of the complexities so that will be easy to people.

So again, the benefits, there are plenty of things that they

are adding one by one there's the time out, the

multi thread support.

The fact that there is the separation of other groups are

still not in JUnit, by the way, this is probably the most

popular feature in that people use.

They've taken it to crazy, crazy lengths.

There are people are using sometimes dozens of groups and

they'd like to run--

pinpoint just the one that they run.

and I think that's something that wouldn't be against the

JUnit philosophy.

I think it doesn't go against unit testing.

It doesn't compromise any of the principles that they use.

So there are still a few things.

I'm not going to go on a long rant.

I think we have more features.

But again, it's a friendly competition.

AUDIENCE: And the other question is, is there any

equivalent in TestNG to writing your own test suite

using logic as opposed to XML?

CEDRIC BEUST: Yes, yes there is a program called

Programmatic API, where you can just create your TestNG

object in Java.

That's used by a lot of people to write

test inside the container.

So they have the tests running in the application server.

And driven by a servlet, for example, and in there, they

just create their own TestNG instance and they do, you

know, that there is programmatic API to add the

test classes, to specify what groups you want to run, to

specify multi thread and all of that.

You can really everything in Java.

AUDIENCE: Something I was trying out with JUnit which I

also didn't see so often in TestNG is I'm passing in these

parameters to my method and this does not quite fit with

the annotations because the annotations are something

fixed and the parameters to something that

changes all the time.

So for example this time out thing, if I have my test

method to import files and if I import one, then I want to

have this time out and if I import 1000 and I want to have

another timeout.

Do you have any idea or something?

CEDRIC BEUST: There are two sides to the

answer to your question.

The first is data providers allow you to pass parameters

that are created in a Java method, by the data providers.

So already from that point, you can pass any parameter

that you can create in Java.

The other point though, modifying annotations is also

something that I read into at some point.

It wasn't initially--

like, typically I have a timeout.

And I have hundreds of methods that have a timeout but is

hard coded to 10,000 milliseconds.

And then time goes by and suddenly this time out is no

longer enough, because the tests are getting complicated.

What do I do?

Do I go through this in all hundreds of methods

and I modify it?


I introduced something called annotation transformers in

TestNG, where basically if you implement one of those--

TestNG, every time that it reads an annotation, it's

going to ask you, do you want to change any value to this


And then it's going to read it.

So by doing that whenever you need to modify an annotation

dynamically, that's really, really easy to do and you

have-- and you can really do a lot of very cool things like

changing the number of invocation counts dynamically,

depending on you know how much CPU you have,

that kind of thing.

So yeah, the answer is all the TestNG annotations can be

changed and modified at run time.



A group's first class objects can you set properties on

groups and set dependencies between groups directly?


I didn't have time to touch on dependencies but you can have

methods that depend on other methods, but better methods

that depend on groups.

So they can make sure that all the methods that belong to a

certain groups are run first. And then when they've all been

run and they're all passed, then your

methods can be played.

AUDIENCE: The question was, can I say that one group is

dependent on another?


Yes, there is a depends on groups attribute for test.

MALE SPEAKER: Any other questions?

AUDIENCE: We've been using TestNG in a couple of our

groups, for like eight months or something, and we're very

happy with it.

So I want to plug it.

I saw on your future slide about distributed TestNG and

was curious if you could quickly describe that and when

you might think it would be available?

CEDRIC BEUST: We can take that off line and I can put you in

touch with the GigaSpace person--

people who are working on it and we've been working on it.

The general idea was to being able to nominate a pool of

slave machines that can run your tests.

And then from the master, we take the testng.xml and see

all the ones, all the tests that are independent from each

other that they can be run really, really separately.

Then you put them over there, send them there, get the

results and collate all the results and present them.

That's the overall idea.

I haven't really kept track of what they've been doing so you

would have to talk to them directly.

But I'll be happy to give you their email address.

AUDIENCE: Sounds good.

And we love TestNG.

CEDRIC BEUST: Thank you.

AUDIENCE: Just a quick question.

So most of the time what we do is actually we have a base

framework of JUnit test. And we exchange it, and then

override certain behavior and everything.

So a lot of the data which we override seems to be, like, if

you will move to TestNG would be coming from XML.

Is there a way to set an inheritance path kind of thing

in the XML itself, like, overriding behavior.

CEDRIC BEUST: Set the inheritance path

AUDIENCE: like I mean seven types, like, my exclusion

group changes based on--

it is basically--

I have a base set of tests where I exclude two tests and

somebody's going to inherit that and run that

test so they want--

CEDRIC BEUST: Can we talk?

Let's take it off line.

Meet me after this and you can explain to me more details.

I'm afraid it's going to go very, very deep there.

But yeah, grab at the end of this talk.


MALE SPEAKER: Anyone else?

Or am I just being blind?

MALE SPEAKER 2: All right, Cedric, thank you very much.

The Description of GTAC 2007: Cedric Beust - Test NG