Practice English Speaking&Listening with: Google I/O 2011: Accelerated Android Rendering

Normal
(0)
Difficulty: 0

Guy: Good morning, and welcome

to this Android Accelerated Rendering session.

Yes, Chet worked really hard

on that sound effect once again.

Haase: [laughter]

Guy: And I believe, yeah, if you want to tweet

because we're boring, you have hashtags

at the bottom of the-- of the slide

that you can use so we can,

you know, see what you are saying about us later.

If you have feedback,

there's this incredibly-hard-to-parse link,

so go there, tell us what you think about the presentation.

And, for those of you who don't know me,

I'm Romain Guy.

I've been working on the Android UI Toolkit team

for about four years now and, on Android 3.0, I was in charge

of designing and implementing the OpenGL pipeline

that we're going to talk about in this presentation.

And joining me today is Chet Haase,

who's been working on animation

and was helping me on the pipeline as well.

Haase: Oh, I thought I was going to do this in mime.

Okay, so contrary to popular belief,

we've actually been doing GPU stuff for a long time.

People keep asking us for GPU support for the platform.

We--we've actually got it all over the place.

It just may not be quite obvious to you.

For one thing, most of the games are being written using the GPU,

either using the NDK, OpenGL ES1 or OpenGL--

OpenGL ES2 are both supported for the NDK,

or they could be using the STK APIs,

which are basically wrappers around those same libraries.

We also use it for window compositing,

so we do software rasterization of everything,

all the graphics that we draw,

and that gets drawn into a buffer

that is then composited by the hardware,

so there's lots of GPU stuff happening already.

Oh, and including live wallpapers

which were written with an earlier version

of Render Script, which uses the GPU.

So there's all kinds of stuff that's actually using the GPU.

However, the basic UI is not using it.

So everything that we draw on the screen in the framework

when--when you say "Canvas.drawLine" or "Canvas."

you know, "drawRect," or, you know,

any of these shape operations,

all of the stuff that the standard views are using.

That goes through our Canvas APIs.

None of that stuff is accelerated. Right?

And, on top of that, the entire UI that you're using

in all the apps that--

that ship with the core platform are not accelerated,

and that's kind of what people mean,

is when is that stuff going to get faster?

But, do we really need it?

Because all the CPUs are getting faster,

the tablets are coming out with multi-cores.

I mean, there's lots of devices that are coming out

with faster CPUs, more, bigger, better,

lots of memory to use, multi-core,

so do we actually need this thing accelerated,

because maybe we can just depend upon the hardware

that we're getting to speed it up for us.

Guy: And one of the key differences

between tablets and phones is that now we have--

we have much more pixels onscreen,

and the difference is pretty staggering, actually,

between a--a Nexus S that has a high-density display

and the Motorola's tablet that you see here

or the Samsung tablet that-- that you got yesterday.

I think we have more than two times the amount of pixels

on the tablet and, you know, now it's 2011

and you're expecting richer experiences.

You want more animations everywhere,

you want smoother animations, and--

and just in general, you know, you want the UI

to be perfectly smooth and nice to use.

So here's the problem that we've run into.

This is a little graphics-- a little graphic

that shows you the expansion of the number of pixels

since the G1 that shipped in 2008

up until the Motorola Zoom that shipped a couple month ago,

and that's the red line.

So you can see that between the Nexus S and the Zoom,

we've seen a dramatic increase in the number of pixels,

and the yellow line shows you the available bandwidth

that we have to move pixels around

between the CPU and the memory,

and unfortunately, the bandwidth itself has always been, like,

on par with the number of pixels,

so it was good enough, but now we have more pixels

than we can afford to move on--on the bus.

So software rendering is not enough.

So with Honeycomb, we've introduced

a new rendering pipeline for the UI,

so all the standard drawing operations

that Chet mentioned on the Canvas,

for instance "drawLine," "drawBitmap," "drawText,"

are now hardware accelerated if you opt in,

and to give you an idea of the level of compatibility

that we have,

all the standard applications running on Android 3.0

are running with this new pipeline.

So the browser, which includes the web view,

Gmail, contacts, books,

they all run with hardware acceleration turned on.

Haase: They--the acronym that we have here

we're trying to popularize.

It's actually pronounced "guh-poo-ee."

[laughter]

Haase: So it's--it's user interface on the GPU.

That's the whole idea, so--

Guy: I'm sure it will have a lot of success.

Haase: --you can start using that today.

[laughter]

Haase: So it helps to understand

how the rendering model used to work,

and then how it works now under this wonderful

new world of GPU acceleration.

So in the releases prior to 3.0,

a view would be redrawn.

So let's say a button is being drawn and so

its onDraw method gets called,

and then it calls rendering operations,

it calls these drawing operations on the Canvas API.

It's going to draw a border, it's going to draw a background,

gradient fill, or a bitmap,

it's going to draw some text, whatever.

And then we go into the software rasterizer.

We happen to use a library internally called Skia,

and that basically turns these operations into pixels

in the frame buffer. So it, you know, does a little

Bresenham algorithm or whatever it does,

and actually lights up the pixels in the appropriate color

and then it gets copied onto the display.

So all of that's very straightforward.

In the new model, we have this alternative way

of doing it instead.

So we go through the same process

from the views perspective. Nothing changes there.

The button's onDraw method will get called,

it'll issue the rendering calls into the Canvas API,

but instead of going through the software rasterization route,

we go through an alternative path

that turns those into GPU operations,

essentially calls into OpenGL.

So instead of walking the line and--

and casting out pixels, we say,

"Hey, OpenGL, why don't you draw a line for us."

And this is basically the pattern of all the operations,

all the GPU acceleration stuff that we do,

and there's different ways of doing it

that Romain is about to talk about,

but basically everything becomes a series of operations

that we send down to the GPU via the OpenGL ES2 library.

Guy: So with OpenGL,

the only primitives we have available are points,

lines, triangles, and textures.

I mean, it's a simplification of OpenGL,

but that's pretty much what we have to work with.

So we wanted to show you-- we wanted to give you a peek

behind the scenes, like, what happens when we draw

some of the standard elements, UI elements on Android.

So this is a simple ListView running on Android 3.0,

and that list is turned into a combination of those points,

lines, triangles, and textures that I just mentioned.

So for instance, that ListView,

once it goes through the OpenGL pipeline, looks like this.

This is a screen shot I took using one of our debugging tools

that shows us what exactly is-- is used by the GPU.

So you can see the mesh that we're using to draw

all the icons, the pieces of text,

and at the bottom of--

at the top and bottom of the list,

you can see these really wide rectangles.

Those are for the fade regions that you see at the top

and bottom of the list when you scroll them.

Haase: I actually like this version better.

I think it's very geeky-cool.

You can't read the text very well, but it--

it looks really neat. [laughter]

Guy: It's-- it's a new style of UI.

Here's an example of a button.

So we will ignore the text in the--the--

the GPU transformation, but a button uses something

we call "nine patches."

So nine patches are pretty much textures

we stretch in--in--in-- in very specific ways,

and it's made very easy by OpenGL

because we can just map that texture onto a mesh

that looks like this, and OpenGL will do

the work for us of stretching the texture correctly.

The text itself is drawn in a very similar way to path.

So here we have this beautiful blue line with a--

with curves, and to render a path like that,

what we do is we generate a-- an alpha texture, so--

so it's a mask. And we apply this mask

on a very simple piece of geometry, or quads.

We have only two triangles.

And then with our fragmentators,

we colorize the path.

So this is what it looks like.

This is our mask.

This is the--the contents of the OpenGL texture,

and we just draw it on top of the simple piece of geometry.

And text is drawn exactly the same way.

Our--our text-- this is a--

a very common way of drawing text in GL.

You have this mask that contains your font

and you just apply it on very simple pieces of geometry.

For those of you geeks who love OpenGL

and have used OpenGL before, I just wanted to show you

a couple of the shaders that are generated by the renderer,

and if you don't know anything about OpenGL,

you probably don't care about those next two slides,

so we won't go into the details.

But, the--the-- one of the difficulties

of implementing the-- the OpenGL renderer was

we have so many different combinations, you know,

API, that we could not write all the shaders by hand,

so we have a--

a program that generates those shaders at run time.

And I think we have, like, several millions of combinations

of--of shaders possible. So this is an example.

If you draw a piece of text and you colorize

the inside of the text with a gradient,

this is the vertex shader that's generated by the--

the system. And the vertex shader,

for those of you who don't know that,

is applied to every single vertex inside a mesh.

So with a triangle, you have three vertices

and each vertex runs this code.

This is the fragment shader. Again, just an example.

You can see that we have two textures.

Let me show you. Two textures right here.

So that's the texture that contains the font,

and that's the texture that contains the gradient,

and the shader simply combines them

into something that looks awesome.

Haase: So obviously this is really complicated stuff

and we did a lot of work to make this work and--

and there's all these really gnarly meshes being sent down

with texture maps and-- and complicated shaders.

So then the question is, "How do you actually access it?"

What is the-- the complicated API

that you need to be aware of

to enable hardware acceleration in your applications?

So first of all-- no, actually that's it.

Basically, opt in.

The only trick is that you need to choose

to have hardware acceleration for your application,

because, as we'll get into a little bit later,

there may be some stuff that doesn't work quite--

quite correctly for you.

You may be using capabilities of the Canvas API

that we simply haven't enabled on the GPU at this time.

So you need to test your application,

make sure it works for you,

and then go ahead and opt in at the application level

by issuing this in your manifest file.

There's a little bit more detail to it than--than that,

which is that you can enable it at the application level

or at the activity level, or at the window level.

You can also disable it at either the activity

or the application level.

And then, finally, if your overall application

is running with hardware acceleration enabled,

you can disable it for particular views if you need to.

You can't enable it per view,

but once the application has it running,

then you can disable it per view,

and this turns out to be useful for particular situations

where you really want to use GPU acceleration

but your custom view clips against a path

which is not currently supported.

Well, what you can do is just have that single view

use software rasterization.

Everything else in your application

is happily using the GPU.

So here's a little bit more detail

about how to actually opt into this stuff.

So at the application level, as I said,

you can opt in, say "hardwareAccelerated = true."

Maybe there's a particular activity

that can't use it for some reason,

so you can disable it at the activity label--

at--at the activity level by setting the same flag

equal to "false."

You can actually see this in the STK API demos.

There are some API demos that used Canvas APIs

that were not accelerated at the time that they were written,

so the activities opt out by setting this flag to "false,"

even though the overall application

is hardware accelerated.

And so you can also opt in at run time on a window level

by setting this flag,

the "FLAG HARDWARE ACCELERATED" flag.

And, finally, you can disable it at the view level

by using a software layer.

We're going to learn more about layers in a few minutes.

There's more to it than simply being able

to disable hardware acceleration for a view.

But, basically, if you want to draw your view into a bitmap,

you can do this,

and that will use software rasterization.

Guy: And if you plan on using hardware acceleration,

you might run into several issues

where you are using an API

that is not currently supported by the pipeline,

so you might want to use one of those two methods

to check whether hardware acceleration

is currently enabled.

For instance, the team working on the launcher application

for Android 3.0, they wanted the code

to still be able to run on our phones

that don't have hardware acceleration at this time,

so their code contains some of these checks to--

to perform some operations in one situation and--and--

and some other operations in the other case.

So for instance, if the hardware acceleration

is on at the view level, you might want to display,

you know, fancier graphics or enable more animations,

but what's very important is to check the hardware acceleration

at the Canvas level,

because even if your application is hardware accelerated,

the Canvas that's patched to your view for drawing

may be a software canvas.

If the view for some reason is rendered into a bitmap,

then the view will be rendered in software.

So just remember those two methods.

You should not have to use them in many situations.

I haven't used them myself in-- in any place,

I think, so far, but it can be very useful.

And, like we mentioned, we did not--

we do not cover 100% of the existing 2-D API

with the hardware pipeline,

and there are various reasons for that.

We had, you know, time constraints,

there are some APIs that are very rarely used,

if at all, by applications,

and there are also some of the APIs

that we have on the Canvas are difficult to implement

efficiently on the GPU so we-- we had to--to make a decision,

and some of these APIs were left out.

Download that menu.

Haase: Yeah, it--it's important to point out that,

before you get too worried about all the stuff

that we don't support on the GPU,

everything you see on the tablets

that you got yesterday is hardware accelerated

in the home screen, the launcher,

the all apps view, everything in the basic UI,

also everything in the applications we ship,

so that's the level of support that we have

for graphics acceleration, you know.

Everything you're seeing in contacts and Gmail

and YouTube and, you know, all the API--all the--

all the UIs for all these applications

in the home screen itself,

that's all supportable by the GPU.

These are more like corner cases that you should be aware of,

but won't run into with-- with the standard views

and probably won't run into with your custom views either.

Guy: Yeah, if--if you use the standard views

and the standard drawables, you'll be fine.

So this is the complete list of APIs we do not support

in Android 3.0 and I believe 3.1,

so when you clip, you can only clip with the rectangles.

Clipping with "path" and "regions"

is not currently supported.

Drawing pictures is not supported,

although this--

this should be pretty easy to add in the future.

DrawPoints was added either in 3.1 or ICS,

I don't remember exactly, but it's coming.

The DrawPosText and TextOnPath are currently not supported,

but we probably will.

There's no technical reason why we wouldn't.

DrawVertices is the same thing; we can support it,

we just didn't have time to do it.

And on the paints, you can't set a MaskFilter or Rasterizer,

and I don't even remember what the Rasterizer does,

so it's probably not very useful.

Haase: Yeah, I think it rasterizes.

Guy: I'm sure it does. [laughter]

On top of that,

some of the supported APIs have limitations.

So if you clip with a rectangle

and you have a 3-D transformation on the canvas,

the 3-D transform will not be applied to the clipRect.

And there are some operations that you can't do

on the clipRect.

You can't do an "x" or a "difference"

or a "reverse difference."

Those are currently not supported,

and I've never seen an application use them.

DrawBitmapMesh, we ignored the colors array at--

at--at this time. Again, we can't support it.

DrawLines we don't have anti-aliasing,

although ICS will add support for that.

Haase: I--I hope so or else I've been wasting my life

for the last three weeks.

Guy: That was part of my plan.

Haase: [laughter] Ha ha ha ha.

Guy: SetDrawFilter, we completely ignore it,

and on the Paint, if you try to use--

to enable dithering or to disable filtering on bitmaps,

we ignore that.

We--those--those flags are here to lower the quality

of the rendering, which makes sense in software

because you want to save memory and you want to save CPU time.

On the GPU, we have enough horsepower

that we can always choose the high-quality version.

And finally, shadows work only on text at the moment.

And finally, there's this thing called a "ComposeShader."

How many of you have ever used a shader in--on Android?

Yeah, that's not many.

In a way, that's good, but the other--you know--

Haase: I counted four. Guy: Yeah.

Shame on you, the other guys.

[laughter]

A ComposeShader lets you combine two different shaders,

so if you want to draw a gradient that's a shader,

if you want to draw-- to texture something,

you use a shader and you can compose them

in one operation.

So the limitations we have

is that you can't recursively use ComposeShaders,

and the two shaders inside of ComposeShader

have to be of different types.

So you can't use two bitmap shaders at the same time

or two linear or gradient shaders.

And the workaround for all these limitations,

pretty easy if you have custom code

that uses any of these APIs,

or that requires any of those limited functionalities.

Just use software layers on your view.

And so we told you that you should not worry too much about,

you know, the compatibility of your app with the--

the--the rendering pipeline because obviously it works for--

for--for--for very complex applications that we ship,

but you have to be aware of-- of the--the new drawing model.

We--we had to make-- make some changes

to the rendering model that exposed bugs in applications.

Those bugs were-- were actual bugs,

but they did not show up as bugs onscreen,

so Chet will take you through the new--

the old rendering model, and the new rendering model,

and you will understand what you should do,

and what you should not do in your app.

Haase: So in the old rendering model,

if a view changed-- so let's say, somebody--

you changed the text on a button--

then internally we're going to invalidate that view and--

and tell the system that it needs to be redrawn.

If it's a custom view, you--some property changes

and you call invalidate on the view yourself.

Same model. So the view gets invalidated,

and it propagates that information up the tree.

It says, "Okay, parent, I need to be invalidated.

You need to invalidate yourself,"

and it walks all the way up the tree to the ViewRoot,

and the ViewRoot says, "Okay, things are invalid.

They need to be redrawn,

so I'm going to tell the children

that need to be redrawn to redraw themselves."

So we're going to tell the parent to redraw itself

and then the parent redraws its children,

and everything is very happy.

Now, the-- the thing that we did

to make some of the GPU stuff more efficient

was basically to not redraw anything that hasn't changed.

So we created this thing that we called "DisplayList,"

which is basically a cache of the rendering operations

that are going to happen for a particular view.

So there's no logic in this thing,

it's just the rendering operations themselves.

For a button, there's going to be, you know,

drawing a border or filling the background,

drawing the text, whatever it is. Right?

So for instance, we can take a look at the--the code.

This actually output from a-- a logcat

when we've got a debug mode

where we can actually see what the display lists are

to help chase some issues down.

So we're saving a context, and then we're drawing the patch

that the-- the button is using.

We're saving the context again because we're about to change

some transform information. We have a ClipRect.

We translate to get to the interior

because we're about to draw the--

the label itself of the button,

and then we restore the contexts that were saved.

So this is basically the way all of our DisplayLists look,

is--is this list of commands for any particular view,

and then if it's a parent,

it's going to be a list of its drawing commands

and then all of the stuff that references

to the DisplayList inside the parent of--of the children.

So the new rendering model looks like this.

You invalidate a view.

Something in the view changes, you invalidate it.

That marks its DisplayList "dirty."

Now, one of the things to note here

is the parallel nature of this thing,

so for every view, we have a matching DisplayList.

So every view knows how to draw itself with its--

its DisplayList.

So the view gets invalidated,

it marks its DisplayList "dirty,"

which means, "My DisplayList needs to be recreated."

It invalidates its parent.

Now, notice, the parent didn't actually get dimmed out here

because the parent does not need to be redrawn.

The parent simply needs to redraw its children

which have changed. Right?

So we're basically propagating the information up the tree

to say, "Something in the tree has been invalidated,

and it needs its DisplayList recreated."

So the ViewRoot gets this information and says,

"Okay, tree, recreate the DisplayLists

that need to be recreated."

That does not recreate the DisplayList

at the top for the ViewGroup,

it just recreates the DisplayList

for the child that got invalidated

and then it tells everybody to draw

and that basically issues the DisplayLists again,

which is a very quick operation

compared to actually redrawing everything.

So notice the ViewGroup didn't get redrawn here.

Instead, the ViewGroup just issued its DisplayList

and the DisplayLists of the children.

The only thing that got redrawn was the actual view

that got invalidated.

Guy: And to better understand why this is so important,

we're going to look at a simple example.

So here, this is the-- the settings application

running on Android 3.0, and we're going to just click

on the Wi-Fi setting list item, and when you click--

when we click on this item, it turns blue.

So now I'm going to show you a simplified version

of what happened before Android 3.0.

So just to turn that item blue, we had to redraw everything

that intersected with the list item.

That meant--that means that we had to draw the background,

so the background texture in the back of the window.

We had to redraw the panel; there--

there was a gray panel behind the list.

Then we had to redraw the-- the text selector.

We had to redraw the text on top of--of the list item,

and then we had to redraw the checkbox on the right.

And that is a lot of code to run.

If you look at the source code of the Android platform

and you go look at what's in the drawing methods of ListView,

of TextView, you will see there's a lot of logic.

It's not just simple, you know, like, Canvas or drawLine.

There's a lot of-- of information there.

With the new model, what you are able to do

is run only the drawing method for the selector

because only the selector turned blue

and then we can just callDraw on the DisplayLists themselves.

So we run a lot less code at the dyadic level

and then we can just run these very simplified native list

of drawing commands in-- in native code.

Very fast.

So this is very important.

With the old model, if you call invalidate on a view,

that view will redraw, but if you call invalidate

on a view that intersects with another view,

that other view would also redraw.

And I've seen applications-- we run into this issue

internally in some of our applications.

Our developers turned on the hardware acceleration

and suddenly they had views that would not redraw

because they were relying on the fact

that by redrawing another view, you know,

a second view would draw as well.

With the new system, we will only redraw views

on which you call invalidate,

and that was always the contract.

The framework should only redraw views

on which you call invalidate.

There was just this weird side effect

that made it work in-- in that special case here.

So if you turn on the hardware acceleration

on your application and you have views

that suddenly don't refresh,

make sure you call invalidate on--

on any view that requires it.

And most of the time, you don't have to worry about it.

If you change a view property like the background drawable,

the text, the framework will do it for you.

So most of the time, this issue arises in custom views.

Haase: So an important case of this is--

is worth understanding more.

It's not just views that intersect,

it's also the parents that-- that get in the way here.

It used to be the case that if you invalidated a parent,

your child would get redrawn.

So if your custom view changed, you could invalidate

the container it was in and everything would get redrawn

and-- and life would be good.

In the new GPU model, we do not redraw that view

because we haven't been told that it changed.

We will redraw the parent, and then the parent

will reissue the DisplayList for the children.

So we can visualize this by saying,

"Okay, we invalidate the ViewGroup."

That gets dimmed out because it's going to be recreated here.

That invalidates all the way up.

ViewRoot says, "Okay, redraw yourself,"

and the ViewGroup happily does that.

I--actually, this is the old model, right?

So it's actually going to redraw the children and--

you know, life-- life was good.

I wanted to redraw that view anyway, so--

so the correct thing happened.

However, in the new model, we now have these DisplayLists.

We invalidate the ViewGroup,

it marks its own DisplayList "dirty,"

and invalidates all the way up to the ViewRoot,

knows what to do next and it says, "Okay, redraw yourself,"

and the ViewGroup happily rebuilds its display list

and that's it. Right?

It did not actually rebuild the DisplayLists

of its children, which means if one of those children changed,

we haven't done anything to update the display of it

because we weren't told that it changed.

This is a very important element in getting the performance

that we needed for the platform

because we really don't want to do anything

that we absolutely don't have to do.

It's one of the ways that we get the performance that--

that we do on these devices. Right?

So you need to be aware of that

and construct your application accordingly.

If you're using standard views, they do the right thing.

They--they do invalidation internally.

You do not need to worry about that,

and in general, you're probably doing the correct thing anyway,

but if you notice that a custom view

is not actually getting redrawn correctly,

then you might examine this a little bit more closely.

Guy: So we mentioned before,

software layers to work around some of the limitations

of the hardware pipeline, but we have other type--

another type of layers.

So we want to talk about that

because they're extremely useful.

So there's a new API on View called "SetLayerType."

You can specify the type. We have three different types,

and you can also pass in optional Paint,

and we'll see what the-- the paint is used for,

but most of the time, you'll probably pass No.

So even with the new model--

So here we have the example of drawing a button.

So the button has a DisplayList

and the DisplayList contains a series of, you know,

drawing operations where we save the--

the context, we draw a patch, we draw the text, and so on.

But when we--we need to draw the DisplayList,

we still need to run a fair amount of code

at the OpenGL renderer level to convert those--

those--those calls into OpenGL code.

And that can be--that can be a little bit too much

when you are trying, for instance,

to animate very complex views, like ListView.

So with layers, you can think of layers as caches.

With layers, what we're going-- what we're going to do

is we're going to use that DisplayList,

we're going to render it inside this layer.

This will be, for instance, an OpenGL text here.

And then when we need to redraw the button onscreen,

if the button hasn't changed,

if you haven't called invalidate on the button,

and you're only moving it,

we're simply going to draw the layer itself.

So it's one operation, it's very fast to do on the GPU.

And here are the three types of layers.

We have the layer of "NONE." This is the default value.

That means the view does not have a layer.

We have the software layer that we talked about.

So when you-- when you use a software layer,

the view will be rendered into a bitmap.

So it will use the-- the old software rendering model

to render the view inside the bitmap.

The bitmap will then be converted

into an OpenGL texture,

and then will render that texture onscreen.

Haase: That's basically equivalent to what--

Guy: Yeah, they-- they use--used it--

Haase: --render cache. Guy: Yeah, there's a--

an API called "setDrawingCacheEnabled."

It does the exact same thing.

And now we--and we also have the hardware layer.

So the hardware layers are very interesting

because we can use the GPU to render the view inside an--

an OpenGL texture

and then we can render the OpenGL textures--

texture with the GPU. So it's extremely fast.

It costs GPU memory, but it's something that we can afford.

The reason why you should use these different types of layers,

we have several reasons.

The first one is performance.

So if you need performance, especially during animations,

you should use hardware layers.

And we'll see an example of how to--

on how to use them properly.

And to give you an idea of how useful the--

the hardware layers can be,

this is a small comparison of performance to draw a ListView.

So if we render a simple ListView on Android 3.0--

3.0 using software,

it's going to take about ten milliseconds,

which is way too much to get nice, smooth frame rate.

If you use--if we use DisplayList on the GPU,

it takes only two milliseconds, but that's still a fair bit--

a fair amount of time if you're moving, you know,

other complex views onscreen at the same time.

And if we turn that-- that ListView inside the--

into a hardware layer, it takes only a fraction

of a millisecond to do it.

So it's extremely efficient.

Haase: Like, it's like a factor of 1,000 speed-up

by using a hardware layer.

Guy: Tiny tiny fraction. Haase: That's pretty cool.

Guy: Software layers and hardware layers

can be used for visual effects.

I mentioned before the optional "Paint"

that you can pass to the LayerType API,

and it can be used to control how you blend,

how you composite the layer onscreen.

So you can use it to apply color filters,

for instance.

You can also apply transparency,

and you can change the blending mode.

So here I have a screenshot

of a modified version of the launcher.

So I modified the launcher to apply one of those

special color filters onto hardware layers.

When you go onto the All Apps list in the launcher,

each page of your All Apps is-- is one hardware layer.

So all I had to do was add a few lines of code

to turn the page into a grayscale page.

Haase: This is the launcher that came out in 1930s

before they had color. [laughter]

Guy: Yeah. Now we can afford colors.

So this is the code I added.

First, we--we created a simple color matrix.

We set the saturation to "zero," then we create a new "Paint."

We set that color matrix as a color filter,

and then, when we enabled the hardware layer,

we just pass that Paint, which means,

"When you draw that layer,

please set the saturation to zero."

And that's all you have to do.

Layers are also useful for compatibility reasons.

We mention-- we mentioned it before.

Just use software layers if you rely on stuff

that we do not support.

And animations--

if you use hardware layers and software layers,

you can get really good performance when animating,

and Chet will tell you a little bit more about that.

Haase: So one of the things that we did in 3.0

was to add a bunch of properties to View.

This was as part of the work that I was doing

on the animation framework.

The way that you used to change properties

like the translucency of a view was by using an AlphaAnimation,

which was nice if you wanted to actually fade-in or fade-out,

but if you actually wanted the view to just be translucent,

there was really no way to do that without a--

a really sort of, hacky workaround

of using an animation as a side effect.

So we've actually put properties on Views

where, frankly, they belong.

So there's some very simple properties

that effect the way that a view is--is displayed.

There is the Alpha property that affects the translucency,

there is some post-layout translation properties,

so you can move the view around within the layout

after the layout positions it.

You can scale it, you can rotate it,

including rotating it around the "y" and "z"--

"y" and "x" axis for 3D rotation,

and finally you can declare the pivot point

around which scaling and rotation happen.

So all of these properties share the common attribute

that they are actually set by the parent

prior to drawing the view.

Now, the--the nuance there that's important for animations

in the layers is that,

that means that the view itself is not changing,

only the display attributes of the view

that are handled by the parent change.

Which means that if you draw your view to a layer,

and then change the attribute,

well, that layer can simply be copied

after the parent sets the attribute.

So we can visualize this a little bit here.

So if you set the rotation "y" property, for instance,

you're going to rotate by 45 degrees around the "y-axis."

That invalidates the parent.

Note that it didn't invalidate the view itself.

The view knows that this is one of those properties,

those magical properties,

that are actually handled at the parent level.

So it says, "Hey, parent, you need to be invalidated

and you need to recreate your DisplayList and--

and reissue your childrens' DisplayLists.

So the ViewGroup marks its own DisplayList as "dirty"

and it gets recreated, and everything happens.

It's all good.

So we can see the code that actually happens

in the DisplayList level here.

The original DisplayList was the ViewGroup drawing itself,

and then drawing the-- the DisplayList for "A"

and then drawing "B's" layer.

So if you "setRotationY"

then it's going to reissue the DisplayList,

but before it does that, it's going to set

the rotation property and then reissue the layer.

And--and the key here

is that we didn't need to recreate the DisplayList.

Right? So this could be a really complicated view

that you have drawn to a layer,

and then whenever we need to redraw it,

all we need to do is copy the layer,

which is an incredibly fast operation on the GPU,

and that's how we got the, you know,

three orders of magnitude speed-up

on that ListView example that we saw.

So we could touch on the animation a little bit.

So this is--we're not really going to get

into the animation framework here.

There's articles and presentations

about that you can check out otherwise.

But, this is a little bit of code

that animates the "rotationY" property

using the new animation framework in 3.0.

So what you want to do, especially for complex views,

is set a hardware layer temporarily. Right?

Layers tend to be expensive

because they are large objects,

so only use them when you need them,

and in general, probably use them temporarily

during an animation.

Guy: They are also expensive

because if you would-- you might be tempted

to enable hardware layers all the time,

but if you have a ListView

and you enable the hardware layer

and the user starts calling the list

on every frame of this callAnimation,

we will redraw the list into the hardware layer

and then we will draw the hardware layer onscreen,

so we will--will be doing twice the amount of work.

Haase: Right.

So what you want to do is to set the layer,

enable the layer, and then run the animation.

But there's a little more to it because then,

you want to disable the layer when you're done.

So the way you do that in the new animation framework

is by setting the hardware layer,

declaring the animation.

So if you can't read the animation code here,

it's basically saying,

"I'm going to create an animation on the view "Object,"

I'm going to animate the "rotationY" property,

and I'm going to animate it to the value of 180 degrees."

We're going to add a listener to the animation

so that we know when the animation is done,

and then, inside that listener,

we're going to disable the layer.

We're going to set the layer type to "none"

when the animation finishes,

and then we start the animation when we're done.

Guy: And, so now we have a few tips and tricks

that you should be aware of if you plan on using

the new hardware accelerated pipeline.

They're general tips and tricks that you can also use

when you're doing software rendering,

so the first one is pretty obvious.

Don't use too many views.

The deeper your hierarchy is going to be,

the harder it will be for the system to render your--

your UI in a very efficient way.

You can have really wide view hierarchies,

as long as they are flat, that's okay.

But, if they are really deep,

then it's going to be very costly.

It's--again, it's also an issue

when you're doing software rendering.

It's even more of an issue when you're doing software rendering

because we have to redraw every single level

when we need to redraw, like, one of the views

at the bottom of the tree.

So I gave a talk, I think in 2009, at Google IO.

The presentation is available on YouTube.

I--I gave--I explained how you can, you know,

keep the number of views to a minimal amount.

So you should refer to that talk and make sure

that your application does not use too many views.

Be very careful of setAlpha.

So setAlpha is this new property

that we've introduced in Android 3.0.

It's really cool 'cause you can, you know,

change the translucency of your view,

you can animate your view, you can fade it in, fade it out.

The problem with setAlpha is,

if the view is not backed by a layer--

software layer or hardware layer--

we'll have to draw the view twice.

We're going to first render the view inside an--

an OpenGL texture, and then we're going to render

that texture with the Alpha ID that you gave us onscreen.

So that means that we're going to have to

process twice the amount of pixels.

We have really cool optimizations

to minimize the amount of blending that we're going to do,

but it will cost you twice the-- the fill-rate.

Haase: Reusing objects is a good idea in general anyway,

especially if you're inner loop,

and particularly if you're running an inner loop

in an animation.

The VM is really good at collecting garbage,

however, it does take some effort. Right?

So if you're creating a little bit of garbage

every time you're running it-- the animation frame

because your inner loop is creating a new object,

and then you're getting rid of it,

eventually you're going to get a GC

and if that GC happens in the middle of the animation,

it may be noticeable to the user.

It'll just cause a little hiccup there.

So that's just kind of a general tip anyway,

but there's another subtlety here

with GPU acceleration that these objects,

the paints and the bitmaps,

may be expensive for us to create

because we're doing stuff at the GPU level

of recreating these textures, re-uploading the textures,

that you don't really want us to do all the time.

So keep these objects around and reuse them

as opposed to creating them new every time.

A link to that is, don't modify the bitmaps very often.

Every time you modify a bitmap,

even if it's just a little bit of the bitmap,

we actually need to upload a texture.

That tends to be an expensive operation with GPUs.

So you may have not changed much on that bitmap,

but we basically need to send the entire thing over the GPU,

which is expensive.

So don't do it if you don't need to.

Guy: It's actually a problem we ran into,

I believe it was in the web browser

during the development of Android 3.0,

we noticed that the frame rate

was not what it was supposed to be and,

so we're, you know, looking at the code

and said, "Web viewing,"

we're wondering why is the web page so slow to render,

until we realized that the tabs at the bottom--

at the top of the screen

were modifying a bitmap on every single frame.

And we were spending, actually, most of our time

just rendering the tab at the top, not the page.

So that was an easy fix.

Haase: Actually, it's really cool

to have bugs like that during a release,

because then, toward the end of the release,

you can make a very simple fix,

and just up the performance tremendously.

I--there-- [laughter]

there was an awesome speed up that we had

in the animation system because early on

in the development of the animation framework,

I, you know, put in an arbitrary value

that basically said,

"Okay, we're going to get 30 frames a second here."

And then, a couple months later,

I'd forgotten that I'd put this thing in,

and we're trying to figure out,

"How come we can't get to 60 frames a second?

Everything has been optimized. We've tuned everything here."

And then, I saw my little value there,

and I changed it and I checked it in,

and we got 2x the animation performance.

[laughter] Pretty awesome.

Guy: And actually, I've--

I have one more little story like that.

The Maps application tried hardware acceleration

and they were disappointed by the--the results.

And it's back and forth on the bug reporting.

We're wondering what's going on because it didn't make sense.

I think it was taking, like, taking 300 milliseconds

to load one frame, until I finally used the Traceview,

which is an awesome tool in the SDK,

and I realized that on every frame when you scrolled the map,

they were reloading their XML, their menu from XML, 96 times.

[laughter]

So that took a little bit of time,

and it had nothing to do with drawing.

And that happens so many times.

Like, very often, like, the kind of bug,

performance bugs that we have, like, use Traceview

and look at how much time you're spending drawing

and how much time you're spending doing other stuff.

You'll be really surprised by what's going on in applications.

So you can't always blame the drawing.

Haase: Was it Traceview I used

to trace down the memory thing too?

The-- Was that--

Guy: LX? Haase: Allocation tracker.

Guy: Yeah. Haase: Also in the tools.

Very useful.

If the element I was mentioning about animations

creating a little bit of garbage and eventually getting a GC,

allocation tracker in the tools in DDMS,

is very useful for figuring out

what is actually being allocated on the fly,

'cause it may not be obvious from looking at your code,

but it is when you look at the output of the tool.

Guy: So similar to not modifying bitmaps too often,

if you're drawing your path

and you change the path all the time,

if you remember the example

of the path that we showed you

at the beginning of the presentation,

we have this texture mask,

and to generate that texture mask,

we have to upload a texture

and we have to render the path inside the texture.

So if you change the path on every frame,

we'll have to regenerate the texture on every frame,

and it will be the equivalent

of modifying the bitmap on every frame.

If your path is relatively small,

it's going to be just fine,

but be very careful if you have a really large path

that's pretty much, you know,

half the size of the screen or more.

Avoid overdraw.

This is extremely important with the GPU because,

I mean, we all think, that, you know,

that GPUs are this awesome things

that can run the 3-D games,

but games are really well optimized

to draw each pixel once if they can.

It's much harder to do that with a UI

where we have several layers, and very often,

we have anti-aliasing and translucency,

and so we have to draw each layer.

There's nothing we can skip.

So to give you an idea on the current generation GPUs,

you can render about 2.5 times the number of pixels onscreen

per frame and run at 60 frames per second.

So if you draw a background image

and you draw a second background image on top of it

and you can't see the one that was behind, like,

you have a background texture

and then you put a white rectangle on top of it,

that's already two times

the number of pixels onscreen, and you only--

and most of your frame rate is gone.

And actually, that's one of the optimizations

we did in some of our apps.

We have a default background texture in Android 3.0

and some of the apps were just covering it.

And they were losing a lot of frame rate

just because they were not removing

that background texture.

We have some optimizations to take care of it.

They cannot always be applied.

So make sure that you draw only what you have to draw.

And finally, we just told you a few stories.

If there's one thing you should take away

from this presentation is profile your application.

We have tools in the SDK. We have DDMS and Traceview.

Use them.

They will give you a lot of really cool information

about what's going on in your app,

and very often, you may be surprised

by what's going on.

It's not necessarily your fault.

Sometimes it's, you know, you're calling an API

and you don't realize that this very simple API,

just one line of code

is going to do something very expensive.

So if you have any doubts about the performance

of your application, run Traceview, run DDMS,

and go to the talk this afternoon about the tools.

That's about it for today. If you want more information,

there's an article that recaps all of this on

"d.android.com,"

the official development website for Android.

We also post technical articles from time to times

on our personal blogs, and we have times--

time for Q&A, so if you have questions,

please come to the mics.

Haase: And that's the end of the main talk, anyway.

[clapping] Thank you.

Yep? man: So quick question.

So in the hardware acceleration thing

you generate shaders.

Have you found any benefit compiling those shaders

for the individual GPUs?

Guy: Compiling them to--?

man: Compiling them, for example,

I know PowerVR offers a GL/SL compiler.

Have you found any benefit there,

or is it just good enough to hand it off to the driver?

Guy: I mean, the driver will do the compilation.

We thought about caching the results.

The problem is, caching the result

of a shader compilation is a proprietary extension.

Haase: Right.

Guy: And, for instance, the one that NVIDIA offers

has severe limitations, like it takes--

it contains the cron blend mode.

So we decided against it because it was too complicated

and it didn't offer any real benefit for us.

man: So are you talking about caching the result

from on the on device compilation

or actually compiling it?

Guy: From the on device compilation.

man: Okay.

Guy: We can't afford to do--

We're going to run on so many devices that we can't do the--

compilation ahead of time.

man: I--I just didn't know if you were--

Guy: Yeah, sure. No, no.

man: Yeah. And another question.

Romain, what's your score on Stack Overflow?

Guy: My score? Uhhhh, I don't know.

I can look if you want. Haase: He cheats anyway.

He totally votes up his own answers.

man: Well, I don't know how you get any work done.

I mean, every time I Google an Android problem,

I--it's all-- It's honestly like that's,

I might as well just Google "problem + Romain," so anyways.

Guy: I just don't sleep very much.

man: Okay. Fair enough.

man: Hi. Do you know, with 3.0,

is it possible to get hardware acceleration

of the HTML5 Canvas element in the browser

when you're drawing to it?

Guy: No, it's--it does--

it's one of the few things that it doesn't do,

but I know that the browser team has plans for that.

man: Okay.

Haase: And our Q&A is going to be cut short

because we were very late in starting,

so we will probably go a couple more minutes here

and we're going to be in and out of office hours,

which is go across the hall and upstairs.

man: So one thing I noticed while using fragments

is when you add fragments or if you're replacing fragments,

in Honeycomb, the animation is just terrible.

It stutters a lot and, you know,

it kind of tries to hide

one of the fragments and kind of readjusts the size

and starts to show the other fragment

and it's really slow. I mean, why is that?

And I'm not using any custom animations

on the fragments.

Guy: That sounds weird because, I mean,

we use Fragments in a similar application

and we don't have these issues.

So that's why you should profile.

So profile the application and try to see where

the time is spent,

and then you can probably do stuff like enable

hardware layers or try to reduce the number of views

you have to improve the performance.

I mean, it's great to run on the GPU,

but GPUs have their limitations.

If you give us too much work to do,

we'll happily send it to the GPU, but, you know,

it's going to choke on it.

Haase: It could be that the fragment

is doing something like, you know,

the animation that's happening is resizing the fragment

and if you have a really complex layout,

well, resizing it is going to cause a layout on every frame.

man: I don't have a very complex layout.

It's a simple list which, I mean,

even the list items itself is just, like, you know,

probably just text or probably just two lines--

two text labels.

Guy: Have you turned on the hardware acceleration?

man: Yes, of course. Guy: Okay.

Then profile the application.

See what's going on.

man: Okay.

Guy: You know, file bugs if it's awful.

man: And the other question I had is, I mean,

why do you need to turn on hardware acceleration?

Why can't the system be--

Guy: I mean, that's pretty obvious.

I mean, we showed the limitations.

Like, we can't guarantee 100% compatibility of the--

with the apps, so if you have an app on market

and we turn on the hardware acceleration for you

but it makes your app crash,

then your users won't be very happy with you,

and it's going to be our fault.

So we don't want that to happen.

man: Shouldn't it be transparent to the user

if a hardware acceleration is turned on automatically?

Haase: It can't be transparent to the user

if we don't support something that the application depends on,

and we don't know that the application depends on it.

Right? So we could either opt in--we could either

opt in everybody by default and end up really messing up

some applications that weren't doing anything wrong

to begin with. man: Right.

Haase: Or we could have you choose to opt-in

after you test your applications,

which is probably a more robust model.

Guy: But, our goal is somehow, someday to,

you know, make it a default.

Haase: Yeah.

man: Because, I've seen that there are lots of--

Haase: Sir, Sorry. If you can catch us later.

man: I'll catch you later.

Haase: We can't debug it anymore from there,

but, we'll take one more question

and then we should shut this down.

man: Yeah, sorry.

I apologize if this was covered, but was there any enhancements

that would benefit apps that still have to target

older platforms?

Like 1.6 and up, let's say.

Guy: So yeah.

If you target an older platform, you will have to do

some trickery to enable hardware acceleration,

but that's doable, and if you're--

we can't simply optimize the framework,

so if you don't use the hardware acceleration, you know,

we try to make the software pipeline faster.

I can't give you any number, but--

man: So I would be able to

add the line in the manifest and that still would work,

and then reflection?

Guy: Yeah. Yes, if your target--

If you change your target SDK to 11,

you can add the attributes,

or you can do it from code, you know,

you can have a special case,

like, if you're running on Honeycomb then...

man: Turn it on?

Guy: You can run that line of code.

man: Okay. Thanks.

Haase: And that's it. Thank you.

The Description of Google I/O 2011: Accelerated Android Rendering