Guy: Good morning, and welcome
to this Android Accelerated Rendering session.
Yes, Chet worked really hard
on that sound effect once again.
Guy: And I believe, yeah, if you want to tweet
because we're boring, you have hashtags
at the bottom of the-- of the slide
that you can use so we can,
you know, see what you are saying about us later.
If you have feedback,
there's this incredibly-hard-to-parse link,
so go there, tell us what you think about the presentation.
And, for those of you who don't know me,
I'm Romain Guy.
I've been working on the Android UI Toolkit team
for about four years now and, on Android 3.0, I was in charge
of designing and implementing the OpenGL pipeline
that we're going to talk about in this presentation.
And joining me today is Chet Haase,
who's been working on animation
and was helping me on the pipeline as well.
Haase: Oh, I thought I was going to do this in mime.
Okay, so contrary to popular belief,
we've actually been doing GPU stuff for a long time.
People keep asking us for GPU support for the platform.
We--we've actually got it all over the place.
It just may not be quite obvious to you.
For one thing, most of the games are being written using the GPU,
either using the NDK, OpenGL ES1 or OpenGL--
OpenGL ES2 are both supported for the NDK,
or they could be using the STK APIs,
which are basically wrappers around those same libraries.
We also use it for window compositing,
so we do software rasterization of everything,
all the graphics that we draw,
and that gets drawn into a buffer
that is then composited by the hardware,
so there's lots of GPU stuff happening already.
Oh, and including live wallpapers
which were written with an earlier version
of Render Script, which uses the GPU.
So there's all kinds of stuff that's actually using the GPU.
However, the basic UI is not using it.
So everything that we draw on the screen in the framework
when--when you say "Canvas.drawLine" or "Canvas."
you know, "drawRect," or, you know,
any of these shape operations,
all of the stuff that the standard views are using.
That goes through our Canvas APIs.
None of that stuff is accelerated. Right?
And, on top of that, the entire UI that you're using
in all the apps that--
that ship with the core platform are not accelerated,
and that's kind of what people mean,
is when is that stuff going to get faster?
But, do we really need it?
Because all the CPUs are getting faster,
the tablets are coming out with multi-cores.
I mean, there's lots of devices that are coming out
with faster CPUs, more, bigger, better,
lots of memory to use, multi-core,
so do we actually need this thing accelerated,
because maybe we can just depend upon the hardware
that we're getting to speed it up for us.
Guy: And one of the key differences
between tablets and phones is that now we have--
we have much more pixels onscreen,
and the difference is pretty staggering, actually,
between a--a Nexus S that has a high-density display
and the Motorola's tablet that you see here
or the Samsung tablet that-- that you got yesterday.
I think we have more than two times the amount of pixels
on the tablet and, you know, now it's 2011
and you're expecting richer experiences.
You want more animations everywhere,
you want smoother animations, and--
and just in general, you know, you want the UI
to be perfectly smooth and nice to use.
So here's the problem that we've run into.
This is a little graphics-- a little graphic
that shows you the expansion of the number of pixels
since the G1 that shipped in 2008
up until the Motorola Zoom that shipped a couple month ago,
and that's the red line.
So you can see that between the Nexus S and the Zoom,
we've seen a dramatic increase in the number of pixels,
and the yellow line shows you the available bandwidth
that we have to move pixels around
between the CPU and the memory,
and unfortunately, the bandwidth itself has always been, like,
on par with the number of pixels,
so it was good enough, but now we have more pixels
than we can afford to move on--on the bus.
So software rendering is not enough.
So with Honeycomb, we've introduced
a new rendering pipeline for the UI,
so all the standard drawing operations
that Chet mentioned on the Canvas,
for instance "drawLine," "drawBitmap," "drawText,"
are now hardware accelerated if you opt in,
and to give you an idea of the level of compatibility
that we have,
all the standard applications running on Android 3.0
are running with this new pipeline.
So the browser, which includes the web view,
Gmail, contacts, books,
they all run with hardware acceleration turned on.
Haase: They--the acronym that we have here
we're trying to popularize.
It's actually pronounced "guh-poo-ee."
Haase: So it's--it's user interface on the GPU.
That's the whole idea, so--
Guy: I'm sure it will have a lot of success.
Haase: --you can start using that today.
Haase: So it helps to understand
how the rendering model used to work,
and then how it works now under this wonderful
new world of GPU acceleration.
So in the releases prior to 3.0,
a view would be redrawn.
So let's say a button is being drawn and so
its onDraw method gets called,
and then it calls rendering operations,
it calls these drawing operations on the Canvas API.
It's going to draw a border, it's going to draw a background,
gradient fill, or a bitmap,
it's going to draw some text, whatever.
And then we go into the software rasterizer.
We happen to use a library internally called Skia,
and that basically turns these operations into pixels
in the frame buffer. So it, you know, does a little
Bresenham algorithm or whatever it does,
and actually lights up the pixels in the appropriate color
and then it gets copied onto the display.
So all of that's very straightforward.
In the new model, we have this alternative way
of doing it instead.
So we go through the same process
from the views perspective. Nothing changes there.
The button's onDraw method will get called,
it'll issue the rendering calls into the Canvas API,
but instead of going through the software rasterization route,
we go through an alternative path
that turns those into GPU operations,
essentially calls into OpenGL.
So instead of walking the line and--
and casting out pixels, we say,
"Hey, OpenGL, why don't you draw a line for us."
And this is basically the pattern of all the operations,
all the GPU acceleration stuff that we do,
and there's different ways of doing it
that Romain is about to talk about,
but basically everything becomes a series of operations
that we send down to the GPU via the OpenGL ES2 library.
Guy: So with OpenGL,
the only primitives we have available are points,
lines, triangles, and textures.
I mean, it's a simplification of OpenGL,
but that's pretty much what we have to work with.
So we wanted to show you-- we wanted to give you a peek
behind the scenes, like, what happens when we draw
some of the standard elements, UI elements on Android.
So this is a simple ListView running on Android 3.0,
and that list is turned into a combination of those points,
lines, triangles, and textures that I just mentioned.
So for instance, that ListView,
once it goes through the OpenGL pipeline, looks like this.
This is a screen shot I took using one of our debugging tools
that shows us what exactly is-- is used by the GPU.
So you can see the mesh that we're using to draw
all the icons, the pieces of text,
and at the bottom of--
at the top and bottom of the list,
you can see these really wide rectangles.
Those are for the fade regions that you see at the top
and bottom of the list when you scroll them.
Haase: I actually like this version better.
I think it's very geeky-cool.
You can't read the text very well, but it--
it looks really neat. [laughter]
Guy: It's-- it's a new style of UI.
Here's an example of a button.
So we will ignore the text in the--the--
the GPU transformation, but a button uses something
we call "nine patches."
So nine patches are pretty much textures
we stretch in--in--in-- in very specific ways,
and it's made very easy by OpenGL
because we can just map that texture onto a mesh
that looks like this, and OpenGL will do
the work for us of stretching the texture correctly.
The text itself is drawn in a very similar way to path.
So here we have this beautiful blue line with a--
with curves, and to render a path like that,
what we do is we generate a-- an alpha texture, so--
so it's a mask. And we apply this mask
on a very simple piece of geometry, or quads.
We have only two triangles.
And then with our fragmentators,
we colorize the path.
So this is what it looks like.
This is our mask.
This is the--the contents of the OpenGL texture,
and we just draw it on top of the simple piece of geometry.
And text is drawn exactly the same way.
Our--our text-- this is a--
a very common way of drawing text in GL.
You have this mask that contains your font
and you just apply it on very simple pieces of geometry.
For those of you geeks who love OpenGL
and have used OpenGL before, I just wanted to show you
a couple of the shaders that are generated by the renderer,
and if you don't know anything about OpenGL,
you probably don't care about those next two slides,
so we won't go into the details.
But, the--the-- one of the difficulties
of implementing the-- the OpenGL renderer was
we have so many different combinations, you know,
API, that we could not write all the shaders by hand,
so we have a--
a program that generates those shaders at run time.
And I think we have, like, several millions of combinations
of--of shaders possible. So this is an example.
If you draw a piece of text and you colorize
the inside of the text with a gradient,
this is the vertex shader that's generated by the--
the system. And the vertex shader,
for those of you who don't know that,
is applied to every single vertex inside a mesh.
So with a triangle, you have three vertices
and each vertex runs this code.
This is the fragment shader. Again, just an example.
You can see that we have two textures.
Let me show you. Two textures right here.
So that's the texture that contains the font,
and that's the texture that contains the gradient,
and the shader simply combines them
into something that looks awesome.
Haase: So obviously this is really complicated stuff
and we did a lot of work to make this work and--
and there's all these really gnarly meshes being sent down
with texture maps and-- and complicated shaders.
So then the question is, "How do you actually access it?"
What is the-- the complicated API
that you need to be aware of
to enable hardware acceleration in your applications?
So first of all-- no, actually that's it.
Basically, opt in.
The only trick is that you need to choose
to have hardware acceleration for your application,
because, as we'll get into a little bit later,
there may be some stuff that doesn't work quite--
quite correctly for you.
You may be using capabilities of the Canvas API
that we simply haven't enabled on the GPU at this time.
So you need to test your application,
make sure it works for you,
and then go ahead and opt in at the application level
by issuing this in your manifest file.
There's a little bit more detail to it than--than that,
which is that you can enable it at the application level
or at the activity level, or at the window level.
You can also disable it at either the activity
or the application level.
And then, finally, if your overall application
is running with hardware acceleration enabled,
you can disable it for particular views if you need to.
You can't enable it per view,
but once the application has it running,
then you can disable it per view,
and this turns out to be useful for particular situations
where you really want to use GPU acceleration
but your custom view clips against a path
which is not currently supported.
Well, what you can do is just have that single view
use software rasterization.
Everything else in your application
is happily using the GPU.
So here's a little bit more detail
about how to actually opt into this stuff.
So at the application level, as I said,
you can opt in, say "hardwareAccelerated = true."
Maybe there's a particular activity
that can't use it for some reason,
so you can disable it at the activity label--
at--at the activity level by setting the same flag
equal to "false."
You can actually see this in the STK API demos.
There are some API demos that used Canvas APIs
that were not accelerated at the time that they were written,
so the activities opt out by setting this flag to "false,"
even though the overall application
is hardware accelerated.
And so you can also opt in at run time on a window level
by setting this flag,
the "FLAG HARDWARE ACCELERATED" flag.
And, finally, you can disable it at the view level
by using a software layer.
We're going to learn more about layers in a few minutes.
There's more to it than simply being able
to disable hardware acceleration for a view.
But, basically, if you want to draw your view into a bitmap,
you can do this,
and that will use software rasterization.
Guy: And if you plan on using hardware acceleration,
you might run into several issues
where you are using an API
that is not currently supported by the pipeline,
so you might want to use one of those two methods
to check whether hardware acceleration
is currently enabled.
For instance, the team working on the launcher application
for Android 3.0, they wanted the code
to still be able to run on our phones
that don't have hardware acceleration at this time,
so their code contains some of these checks to--
to perform some operations in one situation and--and--
and some other operations in the other case.
So for instance, if the hardware acceleration
is on at the view level, you might want to display,
you know, fancier graphics or enable more animations,
but what's very important is to check the hardware acceleration
at the Canvas level,
because even if your application is hardware accelerated,
the Canvas that's patched to your view for drawing
may be a software canvas.
If the view for some reason is rendered into a bitmap,
then the view will be rendered in software.
So just remember those two methods.
You should not have to use them in many situations.
I haven't used them myself in-- in any place,
I think, so far, but it can be very useful.
And, like we mentioned, we did not--
we do not cover 100% of the existing 2-D API
with the hardware pipeline,
and there are various reasons for that.
We had, you know, time constraints,
there are some APIs that are very rarely used,
if at all, by applications,
and there are also some of the APIs
that we have on the Canvas are difficult to implement
efficiently on the GPU so we-- we had to--to make a decision,
and some of these APIs were left out.
Download that menu.
Haase: Yeah, it--it's important to point out that,
before you get too worried about all the stuff
that we don't support on the GPU,
everything you see on the tablets
that you got yesterday is hardware accelerated
in the home screen, the launcher,
the all apps view, everything in the basic UI,
also everything in the applications we ship,
so that's the level of support that we have
for graphics acceleration, you know.
Everything you're seeing in contacts and Gmail
and YouTube and, you know, all the API--all the--
all the UIs for all these applications
in the home screen itself,
that's all supportable by the GPU.
These are more like corner cases that you should be aware of,
but won't run into with-- with the standard views
and probably won't run into with your custom views either.
Guy: Yeah, if--if you use the standard views
and the standard drawables, you'll be fine.
So this is the complete list of APIs we do not support
in Android 3.0 and I believe 3.1,
so when you clip, you can only clip with the rectangles.
Clipping with "path" and "regions"
is not currently supported.
Drawing pictures is not supported,
this should be pretty easy to add in the future.
DrawPoints was added either in 3.1 or ICS,
I don't remember exactly, but it's coming.
The DrawPosText and TextOnPath are currently not supported,
but we probably will.
There's no technical reason why we wouldn't.
DrawVertices is the same thing; we can support it,
we just didn't have time to do it.
And on the paints, you can't set a MaskFilter or Rasterizer,
and I don't even remember what the Rasterizer does,
so it's probably not very useful.
Haase: Yeah, I think it rasterizes.
Guy: I'm sure it does. [laughter]
On top of that,
some of the supported APIs have limitations.
So if you clip with a rectangle
and you have a 3-D transformation on the canvas,
the 3-D transform will not be applied to the clipRect.
And there are some operations that you can't do
on the clipRect.
You can't do an "x" or a "difference"
or a "reverse difference."
Those are currently not supported,
and I've never seen an application use them.
DrawBitmapMesh, we ignored the colors array at--
at--at this time. Again, we can't support it.
DrawLines we don't have anti-aliasing,
although ICS will add support for that.
Haase: I--I hope so or else I've been wasting my life
for the last three weeks.
Guy: That was part of my plan.
Haase: [laughter] Ha ha ha ha.
Guy: SetDrawFilter, we completely ignore it,
and on the Paint, if you try to use--
to enable dithering or to disable filtering on bitmaps,
we ignore that.
We--those--those flags are here to lower the quality
of the rendering, which makes sense in software
because you want to save memory and you want to save CPU time.
On the GPU, we have enough horsepower
that we can always choose the high-quality version.
And finally, shadows work only on text at the moment.
And finally, there's this thing called a "ComposeShader."
How many of you have ever used a shader in--on Android?
Yeah, that's not many.
In a way, that's good, but the other--you know--
Haase: I counted four. Guy: Yeah.
Shame on you, the other guys.
A ComposeShader lets you combine two different shaders,
so if you want to draw a gradient that's a shader,
if you want to draw-- to texture something,
you use a shader and you can compose them
in one operation.
So the limitations we have
is that you can't recursively use ComposeShaders,
and the two shaders inside of ComposeShader
have to be of different types.
So you can't use two bitmap shaders at the same time
or two linear or gradient shaders.
And the workaround for all these limitations,
pretty easy if you have custom code
that uses any of these APIs,
or that requires any of those limited functionalities.
Just use software layers on your view.
And so we told you that you should not worry too much about,
you know, the compatibility of your app with the--
the--the rendering pipeline because obviously it works for--
for--for--for very complex applications that we ship,
but you have to be aware of-- of the--the new drawing model.
We--we had to make-- make some changes
to the rendering model that exposed bugs in applications.
Those bugs were-- were actual bugs,
but they did not show up as bugs onscreen,
so Chet will take you through the new--
the old rendering model, and the new rendering model,
and you will understand what you should do,
and what you should not do in your app.
Haase: So in the old rendering model,
if a view changed-- so let's say, somebody--
you changed the text on a button--
then internally we're going to invalidate that view and--
and tell the system that it needs to be redrawn.
If it's a custom view, you--some property changes
and you call invalidate on the view yourself.
Same model. So the view gets invalidated,
and it propagates that information up the tree.
It says, "Okay, parent, I need to be invalidated.
You need to invalidate yourself,"
and it walks all the way up the tree to the ViewRoot,
and the ViewRoot says, "Okay, things are invalid.
They need to be redrawn,
so I'm going to tell the children
that need to be redrawn to redraw themselves."
So we're going to tell the parent to redraw itself
and then the parent redraws its children,
and everything is very happy.
Now, the-- the thing that we did
to make some of the GPU stuff more efficient
was basically to not redraw anything that hasn't changed.
So we created this thing that we called "DisplayList,"
which is basically a cache of the rendering operations
that are going to happen for a particular view.
So there's no logic in this thing,
it's just the rendering operations themselves.
For a button, there's going to be, you know,
drawing a border or filling the background,
drawing the text, whatever it is. Right?
So for instance, we can take a look at the--the code.
This actually output from a-- a logcat
when we've got a debug mode
where we can actually see what the display lists are
to help chase some issues down.
So we're saving a context, and then we're drawing the patch
that the-- the button is using.
We're saving the context again because we're about to change
some transform information. We have a ClipRect.
We translate to get to the interior
because we're about to draw the--
the label itself of the button,
and then we restore the contexts that were saved.
So this is basically the way all of our DisplayLists look,
is--is this list of commands for any particular view,
and then if it's a parent,
it's going to be a list of its drawing commands
and then all of the stuff that references
to the DisplayList inside the parent of--of the children.
So the new rendering model looks like this.
You invalidate a view.
Something in the view changes, you invalidate it.
That marks its DisplayList "dirty."
Now, one of the things to note here
is the parallel nature of this thing,
so for every view, we have a matching DisplayList.
So every view knows how to draw itself with its--
So the view gets invalidated,
it marks its DisplayList "dirty,"
which means, "My DisplayList needs to be recreated."
It invalidates its parent.
Now, notice, the parent didn't actually get dimmed out here
because the parent does not need to be redrawn.
The parent simply needs to redraw its children
which have changed. Right?
So we're basically propagating the information up the tree
to say, "Something in the tree has been invalidated,
and it needs its DisplayList recreated."
So the ViewRoot gets this information and says,
"Okay, tree, recreate the DisplayLists
that need to be recreated."
That does not recreate the DisplayList
at the top for the ViewGroup,
it just recreates the DisplayList
for the child that got invalidated
and then it tells everybody to draw
and that basically issues the DisplayLists again,
which is a very quick operation
compared to actually redrawing everything.
So notice the ViewGroup didn't get redrawn here.
Instead, the ViewGroup just issued its DisplayList
and the DisplayLists of the children.
The only thing that got redrawn was the actual view
that got invalidated.
Guy: And to better understand why this is so important,
we're going to look at a simple example.
So here, this is the-- the settings application
running on Android 3.0, and we're going to just click
on the Wi-Fi setting list item, and when you click--
when we click on this item, it turns blue.
So now I'm going to show you a simplified version
of what happened before Android 3.0.
So just to turn that item blue, we had to redraw everything
that intersected with the list item.
That meant--that means that we had to draw the background,
so the background texture in the back of the window.
We had to redraw the panel; there--
there was a gray panel behind the list.
Then we had to redraw the-- the text selector.
We had to redraw the text on top of--of the list item,
and then we had to redraw the checkbox on the right.
And that is a lot of code to run.
If you look at the source code of the Android platform
and you go look at what's in the drawing methods of ListView,
of TextView, you will see there's a lot of logic.
It's not just simple, you know, like, Canvas or drawLine.
There's a lot of-- of information there.
With the new model, what you are able to do
is run only the drawing method for the selector
because only the selector turned blue
and then we can just callDraw on the DisplayLists themselves.
So we run a lot less code at the dyadic level
and then we can just run these very simplified native list
of drawing commands in-- in native code.
So this is very important.
With the old model, if you call invalidate on a view,
that view will redraw, but if you call invalidate
on a view that intersects with another view,
that other view would also redraw.
And I've seen applications-- we run into this issue
internally in some of our applications.
Our developers turned on the hardware acceleration
and suddenly they had views that would not redraw
because they were relying on the fact
that by redrawing another view, you know,
a second view would draw as well.
With the new system, we will only redraw views
on which you call invalidate,
and that was always the contract.
The framework should only redraw views
on which you call invalidate.
There was just this weird side effect
that made it work in-- in that special case here.
So if you turn on the hardware acceleration
on your application and you have views
that suddenly don't refresh,
make sure you call invalidate on--
on any view that requires it.
And most of the time, you don't have to worry about it.
If you change a view property like the background drawable,
the text, the framework will do it for you.
So most of the time, this issue arises in custom views.
Haase: So an important case of this is--
is worth understanding more.
It's not just views that intersect,
it's also the parents that-- that get in the way here.
It used to be the case that if you invalidated a parent,
your child would get redrawn.
So if your custom view changed, you could invalidate
the container it was in and everything would get redrawn
and-- and life would be good.
In the new GPU model, we do not redraw that view
because we haven't been told that it changed.
We will redraw the parent, and then the parent
will reissue the DisplayList for the children.
So we can visualize this by saying,
"Okay, we invalidate the ViewGroup."
That gets dimmed out because it's going to be recreated here.
That invalidates all the way up.
ViewRoot says, "Okay, redraw yourself,"
and the ViewGroup happily does that.
I--actually, this is the old model, right?
So it's actually going to redraw the children and--
you know, life-- life was good.
I wanted to redraw that view anyway, so--
so the correct thing happened.
However, in the new model, we now have these DisplayLists.
We invalidate the ViewGroup,
it marks its own DisplayList "dirty,"
and invalidates all the way up to the ViewRoot,
knows what to do next and it says, "Okay, redraw yourself,"
and the ViewGroup happily rebuilds its display list
and that's it. Right?
It did not actually rebuild the DisplayLists
of its children, which means if one of those children changed,
we haven't done anything to update the display of it
because we weren't told that it changed.
This is a very important element in getting the performance
that we needed for the platform
because we really don't want to do anything
that we absolutely don't have to do.
It's one of the ways that we get the performance that--
that we do on these devices. Right?
So you need to be aware of that
and construct your application accordingly.
If you're using standard views, they do the right thing.
They--they do invalidation internally.
You do not need to worry about that,
and in general, you're probably doing the correct thing anyway,
but if you notice that a custom view
is not actually getting redrawn correctly,
then you might examine this a little bit more closely.
Guy: So we mentioned before,
software layers to work around some of the limitations
of the hardware pipeline, but we have other type--
another type of layers.
So we want to talk about that
because they're extremely useful.
So there's a new API on View called "SetLayerType."
You can specify the type. We have three different types,
and you can also pass in optional Paint,
and we'll see what the-- the paint is used for,
but most of the time, you'll probably pass No.
So even with the new model--
So here we have the example of drawing a button.
So the button has a DisplayList
and the DisplayList contains a series of, you know,
drawing operations where we save the--
the context, we draw a patch, we draw the text, and so on.
But when we--we need to draw the DisplayList,
we still need to run a fair amount of code
at the OpenGL renderer level to convert those--
those--those calls into OpenGL code.
And that can be--that can be a little bit too much
when you are trying, for instance,
to animate very complex views, like ListView.
So with layers, you can think of layers as caches.
With layers, what we're going-- what we're going to do
is we're going to use that DisplayList,
we're going to render it inside this layer.
This will be, for instance, an OpenGL text here.
And then when we need to redraw the button onscreen,
if the button hasn't changed,
if you haven't called invalidate on the button,
and you're only moving it,
we're simply going to draw the layer itself.
So it's one operation, it's very fast to do on the GPU.
And here are the three types of layers.
We have the layer of "NONE." This is the default value.
That means the view does not have a layer.
We have the software layer that we talked about.
So when you-- when you use a software layer,
the view will be rendered into a bitmap.
So it will use the-- the old software rendering model
to render the view inside the bitmap.
The bitmap will then be converted
into an OpenGL texture,
and then will render that texture onscreen.
Haase: That's basically equivalent to what--
Guy: Yeah, they-- they use--used it--
Haase: --render cache. Guy: Yeah, there's a--
an API called "setDrawingCacheEnabled."
It does the exact same thing.
And now we--and we also have the hardware layer.
So the hardware layers are very interesting
because we can use the GPU to render the view inside an--
an OpenGL texture
and then we can render the OpenGL textures--
texture with the GPU. So it's extremely fast.
It costs GPU memory, but it's something that we can afford.
The reason why you should use these different types of layers,
we have several reasons.
The first one is performance.
So if you need performance, especially during animations,
you should use hardware layers.
And we'll see an example of how to--
on how to use them properly.
And to give you an idea of how useful the--
the hardware layers can be,
this is a small comparison of performance to draw a ListView.
So if we render a simple ListView on Android 3.0--
3.0 using software,
it's going to take about ten milliseconds,
which is way too much to get nice, smooth frame rate.
If you use--if we use DisplayList on the GPU,
it takes only two milliseconds, but that's still a fair bit--
a fair amount of time if you're moving, you know,
other complex views onscreen at the same time.
And if we turn that-- that ListView inside the--
into a hardware layer, it takes only a fraction
of a millisecond to do it.
So it's extremely efficient.
Haase: Like, it's like a factor of 1,000 speed-up
by using a hardware layer.
Guy: Tiny tiny fraction. Haase: That's pretty cool.
Guy: Software layers and hardware layers
can be used for visual effects.
I mentioned before the optional "Paint"
that you can pass to the LayerType API,
and it can be used to control how you blend,
how you composite the layer onscreen.
So you can use it to apply color filters,
You can also apply transparency,
and you can change the blending mode.
So here I have a screenshot
of a modified version of the launcher.
So I modified the launcher to apply one of those
special color filters onto hardware layers.
When you go onto the All Apps list in the launcher,
each page of your All Apps is-- is one hardware layer.
So all I had to do was add a few lines of code
to turn the page into a grayscale page.
Haase: This is the launcher that came out in 1930s
before they had color. [laughter]
Guy: Yeah. Now we can afford colors.
So this is the code I added.
First, we--we created a simple color matrix.
We set the saturation to "zero," then we create a new "Paint."
We set that color matrix as a color filter,
and then, when we enabled the hardware layer,
we just pass that Paint, which means,
"When you draw that layer,
please set the saturation to zero."
And that's all you have to do.
Layers are also useful for compatibility reasons.
We mention-- we mentioned it before.
Just use software layers if you rely on stuff
that we do not support.
if you use hardware layers and software layers,
you can get really good performance when animating,
and Chet will tell you a little bit more about that.
Haase: So one of the things that we did in 3.0
was to add a bunch of properties to View.
This was as part of the work that I was doing
on the animation framework.
The way that you used to change properties
like the translucency of a view was by using an AlphaAnimation,
which was nice if you wanted to actually fade-in or fade-out,
but if you actually wanted the view to just be translucent,
there was really no way to do that without a--
a really sort of, hacky workaround
of using an animation as a side effect.
So we've actually put properties on Views
where, frankly, they belong.
So there's some very simple properties
that effect the way that a view is--is displayed.
There is the Alpha property that affects the translucency,
there is some post-layout translation properties,
so you can move the view around within the layout
after the layout positions it.
You can scale it, you can rotate it,
including rotating it around the "y" and "z"--
"y" and "x" axis for 3D rotation,
and finally you can declare the pivot point
around which scaling and rotation happen.
So all of these properties share the common attribute
that they are actually set by the parent
prior to drawing the view.
Now, the--the nuance there that's important for animations
in the layers is that,
that means that the view itself is not changing,
only the display attributes of the view
that are handled by the parent change.
Which means that if you draw your view to a layer,
and then change the attribute,
well, that layer can simply be copied
after the parent sets the attribute.
So we can visualize this a little bit here.
So if you set the rotation "y" property, for instance,
you're going to rotate by 45 degrees around the "y-axis."
That invalidates the parent.
Note that it didn't invalidate the view itself.
The view knows that this is one of those properties,
those magical properties,
that are actually handled at the parent level.
So it says, "Hey, parent, you need to be invalidated
and you need to recreate your DisplayList and--
and reissue your childrens' DisplayLists.
So the ViewGroup marks its own DisplayList as "dirty"
and it gets recreated, and everything happens.
It's all good.
So we can see the code that actually happens
in the DisplayList level here.
The original DisplayList was the ViewGroup drawing itself,
and then drawing the-- the DisplayList for "A"
and then drawing "B's" layer.
So if you "setRotationY"
then it's going to reissue the DisplayList,
but before it does that, it's going to set
the rotation property and then reissue the layer.
And--and the key here
is that we didn't need to recreate the DisplayList.
Right? So this could be a really complicated view
that you have drawn to a layer,
and then whenever we need to redraw it,
all we need to do is copy the layer,
which is an incredibly fast operation on the GPU,
and that's how we got the, you know,
three orders of magnitude speed-up
on that ListView example that we saw.
So we could touch on the animation a little bit.
So this is--we're not really going to get
into the animation framework here.
There's articles and presentations
about that you can check out otherwise.
But, this is a little bit of code
that animates the "rotationY" property
using the new animation framework in 3.0.
So what you want to do, especially for complex views,
is set a hardware layer temporarily. Right?
Layers tend to be expensive
because they are large objects,
so only use them when you need them,
and in general, probably use them temporarily
during an animation.
Guy: They are also expensive
because if you would-- you might be tempted
to enable hardware layers all the time,
but if you have a ListView
and you enable the hardware layer
and the user starts calling the list
on every frame of this callAnimation,
we will redraw the list into the hardware layer
and then we will draw the hardware layer onscreen,
so we will--will be doing twice the amount of work.
So what you want to do is to set the layer,
enable the layer, and then run the animation.
But there's a little more to it because then,
you want to disable the layer when you're done.
So the way you do that in the new animation framework
is by setting the hardware layer,
declaring the animation.
So if you can't read the animation code here,
it's basically saying,
"I'm going to create an animation on the view "Object,"
I'm going to animate the "rotationY" property,
and I'm going to animate it to the value of 180 degrees."
We're going to add a listener to the animation
so that we know when the animation is done,
and then, inside that listener,
we're going to disable the layer.
We're going to set the layer type to "none"
when the animation finishes,
and then we start the animation when we're done.
Guy: And, so now we have a few tips and tricks
that you should be aware of if you plan on using
the new hardware accelerated pipeline.
They're general tips and tricks that you can also use
when you're doing software rendering,
so the first one is pretty obvious.
Don't use too many views.
The deeper your hierarchy is going to be,
the harder it will be for the system to render your--
your UI in a very efficient way.
You can have really wide view hierarchies,
as long as they are flat, that's okay.
But, if they are really deep,
then it's going to be very costly.
It's--again, it's also an issue
when you're doing software rendering.
It's even more of an issue when you're doing software rendering
because we have to redraw every single level
when we need to redraw, like, one of the views
at the bottom of the tree.
So I gave a talk, I think in 2009, at Google IO.
The presentation is available on YouTube.
I--I gave--I explained how you can, you know,
keep the number of views to a minimal amount.
So you should refer to that talk and make sure
that your application does not use too many views.
Be very careful of setAlpha.
So setAlpha is this new property
that we've introduced in Android 3.0.
It's really cool 'cause you can, you know,
change the translucency of your view,
you can animate your view, you can fade it in, fade it out.
The problem with setAlpha is,
if the view is not backed by a layer--
software layer or hardware layer--
we'll have to draw the view twice.
We're going to first render the view inside an--
an OpenGL texture, and then we're going to render
that texture with the Alpha ID that you gave us onscreen.
So that means that we're going to have to
process twice the amount of pixels.
We have really cool optimizations
to minimize the amount of blending that we're going to do,
but it will cost you twice the-- the fill-rate.
Haase: Reusing objects is a good idea in general anyway,
especially if you're inner loop,
and particularly if you're running an inner loop
in an animation.
The VM is really good at collecting garbage,
however, it does take some effort. Right?
So if you're creating a little bit of garbage
every time you're running it-- the animation frame
because your inner loop is creating a new object,
and then you're getting rid of it,
eventually you're going to get a GC
and if that GC happens in the middle of the animation,
it may be noticeable to the user.
It'll just cause a little hiccup there.
So that's just kind of a general tip anyway,
but there's another subtlety here
with GPU acceleration that these objects,
the paints and the bitmaps,
may be expensive for us to create
because we're doing stuff at the GPU level
of recreating these textures, re-uploading the textures,
that you don't really want us to do all the time.
So keep these objects around and reuse them
as opposed to creating them new every time.
A link to that is, don't modify the bitmaps very often.
Every time you modify a bitmap,
even if it's just a little bit of the bitmap,
we actually need to upload a texture.
That tends to be an expensive operation with GPUs.
So you may have not changed much on that bitmap,
but we basically need to send the entire thing over the GPU,
which is expensive.
So don't do it if you don't need to.
Guy: It's actually a problem we ran into,
I believe it was in the web browser
during the development of Android 3.0,
we noticed that the frame rate
was not what it was supposed to be and,
so we're, you know, looking at the code
and said, "Web viewing,"
we're wondering why is the web page so slow to render,
until we realized that the tabs at the bottom--
at the top of the screen
were modifying a bitmap on every single frame.
And we were spending, actually, most of our time
just rendering the tab at the top, not the page.
So that was an easy fix.
Haase: Actually, it's really cool
to have bugs like that during a release,
because then, toward the end of the release,
you can make a very simple fix,
and just up the performance tremendously.
there was an awesome speed up that we had
in the animation system because early on
in the development of the animation framework,
I, you know, put in an arbitrary value
that basically said,
"Okay, we're going to get 30 frames a second here."
And then, a couple months later,
I'd forgotten that I'd put this thing in,
and we're trying to figure out,
"How come we can't get to 60 frames a second?
Everything has been optimized. We've tuned everything here."
And then, I saw my little value there,
and I changed it and I checked it in,
and we got 2x the animation performance.
[laughter] Pretty awesome.
Guy: And actually, I've--
I have one more little story like that.
The Maps application tried hardware acceleration
and they were disappointed by the--the results.
And it's back and forth on the bug reporting.
We're wondering what's going on because it didn't make sense.
I think it was taking, like, taking 300 milliseconds
to load one frame, until I finally used the Traceview,
which is an awesome tool in the SDK,
and I realized that on every frame when you scrolled the map,
they were reloading their XML, their menu from XML, 96 times.
So that took a little bit of time,
and it had nothing to do with drawing.
And that happens so many times.
Like, very often, like, the kind of bug,
performance bugs that we have, like, use Traceview
and look at how much time you're spending drawing
and how much time you're spending doing other stuff.
You'll be really surprised by what's going on in applications.
So you can't always blame the drawing.
Haase: Was it Traceview I used
to trace down the memory thing too?
The-- Was that--
Guy: LX? Haase: Allocation tracker.
Guy: Yeah. Haase: Also in the tools.
If the element I was mentioning about animations
creating a little bit of garbage and eventually getting a GC,
allocation tracker in the tools in DDMS,
is very useful for figuring out
what is actually being allocated on the fly,
'cause it may not be obvious from looking at your code,
but it is when you look at the output of the tool.
Guy: So similar to not modifying bitmaps too often,
if you're drawing your path
and you change the path all the time,
if you remember the example
of the path that we showed you
at the beginning of the presentation,
we have this texture mask,
and to generate that texture mask,
we have to upload a texture
and we have to render the path inside the texture.
So if you change the path on every frame,
we'll have to regenerate the texture on every frame,
and it will be the equivalent
of modifying the bitmap on every frame.
If your path is relatively small,
it's going to be just fine,
but be very careful if you have a really large path
that's pretty much, you know,
half the size of the screen or more.
This is extremely important with the GPU because,
I mean, we all think, that, you know,
that GPUs are this awesome things
that can run the 3-D games,
but games are really well optimized
to draw each pixel once if they can.
It's much harder to do that with a UI
where we have several layers, and very often,
we have anti-aliasing and translucency,
and so we have to draw each layer.
There's nothing we can skip.
So to give you an idea on the current generation GPUs,
you can render about 2.5 times the number of pixels onscreen
per frame and run at 60 frames per second.
So if you draw a background image
and you draw a second background image on top of it
and you can't see the one that was behind, like,
you have a background texture
and then you put a white rectangle on top of it,
that's already two times
the number of pixels onscreen, and you only--
and most of your frame rate is gone.
And actually, that's one of the optimizations
we did in some of our apps.
We have a default background texture in Android 3.0
and some of the apps were just covering it.
And they were losing a lot of frame rate
just because they were not removing
that background texture.
We have some optimizations to take care of it.
They cannot always be applied.
So make sure that you draw only what you have to draw.
And finally, we just told you a few stories.
If there's one thing you should take away
from this presentation is profile your application.
We have tools in the SDK. We have DDMS and Traceview.
They will give you a lot of really cool information
about what's going on in your app,
and very often, you may be surprised
by what's going on.
It's not necessarily your fault.
Sometimes it's, you know, you're calling an API
and you don't realize that this very simple API,
just one line of code
is going to do something very expensive.
So if you have any doubts about the performance
of your application, run Traceview, run DDMS,
and go to the talk this afternoon about the tools.
That's about it for today. If you want more information,
there's an article that recaps all of this on
the official development website for Android.
We also post technical articles from time to times
on our personal blogs, and we have times--
time for Q&A, so if you have questions,
please come to the mics.
Haase: And that's the end of the main talk, anyway.
[clapping] Thank you.
Yep? man: So quick question.
So in the hardware acceleration thing
you generate shaders.
Have you found any benefit compiling those shaders
for the individual GPUs?
Guy: Compiling them to--?
man: Compiling them, for example,
I know PowerVR offers a GL/SL compiler.
Have you found any benefit there,
or is it just good enough to hand it off to the driver?
Guy: I mean, the driver will do the compilation.
We thought about caching the results.
The problem is, caching the result
of a shader compilation is a proprietary extension.
Guy: And, for instance, the one that NVIDIA offers
has severe limitations, like it takes--
it contains the cron blend mode.
So we decided against it because it was too complicated
and it didn't offer any real benefit for us.
man: So are you talking about caching the result
from on the on device compilation
or actually compiling it?
Guy: From the on device compilation.
Guy: We can't afford to do--
We're going to run on so many devices that we can't do the--
compilation ahead of time.
man: I--I just didn't know if you were--
Guy: Yeah, sure. No, no.
man: Yeah. And another question.
Romain, what's your score on Stack Overflow?
Guy: My score? Uhhhh, I don't know.
I can look if you want. Haase: He cheats anyway.
He totally votes up his own answers.
man: Well, I don't know how you get any work done.
I mean, every time I Google an Android problem,
I--it's all-- It's honestly like that's,
I might as well just Google "problem + Romain," so anyways.
Guy: I just don't sleep very much.
man: Okay. Fair enough.
man: Hi. Do you know, with 3.0,
is it possible to get hardware acceleration
of the HTML5 Canvas element in the browser
when you're drawing to it?
Guy: No, it's--it does--
it's one of the few things that it doesn't do,
but I know that the browser team has plans for that.
Haase: And our Q&A is going to be cut short
because we were very late in starting,
so we will probably go a couple more minutes here
and we're going to be in and out of office hours,
which is go across the hall and upstairs.
man: So one thing I noticed while using fragments
is when you add fragments or if you're replacing fragments,
in Honeycomb, the animation is just terrible.
It stutters a lot and, you know,
it kind of tries to hide
one of the fragments and kind of readjusts the size
and starts to show the other fragment
and it's really slow. I mean, why is that?
And I'm not using any custom animations
on the fragments.
Guy: That sounds weird because, I mean,
we use Fragments in a similar application
and we don't have these issues.
So that's why you should profile.
So profile the application and try to see where
the time is spent,
and then you can probably do stuff like enable
hardware layers or try to reduce the number of views
you have to improve the performance.
I mean, it's great to run on the GPU,
but GPUs have their limitations.
If you give us too much work to do,
we'll happily send it to the GPU, but, you know,
it's going to choke on it.
Haase: It could be that the fragment
is doing something like, you know,
the animation that's happening is resizing the fragment
and if you have a really complex layout,
well, resizing it is going to cause a layout on every frame.
man: I don't have a very complex layout.
It's a simple list which, I mean,
even the list items itself is just, like, you know,
probably just text or probably just two lines--
two text labels.
Guy: Have you turned on the hardware acceleration?
man: Yes, of course. Guy: Okay.
Then profile the application.
See what's going on.
Guy: You know, file bugs if it's awful.
man: And the other question I had is, I mean,
why do you need to turn on hardware acceleration?
Why can't the system be--
Guy: I mean, that's pretty obvious.
I mean, we showed the limitations.
Like, we can't guarantee 100% compatibility of the--
with the apps, so if you have an app on market
and we turn on the hardware acceleration for you
but it makes your app crash,
then your users won't be very happy with you,
and it's going to be our fault.
So we don't want that to happen.
man: Shouldn't it be transparent to the user
if a hardware acceleration is turned on automatically?
Haase: It can't be transparent to the user
if we don't support something that the application depends on,
and we don't know that the application depends on it.
Right? So we could either opt in--we could either
opt in everybody by default and end up really messing up
some applications that weren't doing anything wrong
to begin with. man: Right.
Haase: Or we could have you choose to opt-in
after you test your applications,
which is probably a more robust model.
Guy: But, our goal is somehow, someday to,
you know, make it a default.
man: Because, I've seen that there are lots of--
Haase: Sir, Sorry. If you can catch us later.
man: I'll catch you later.
Haase: We can't debug it anymore from there,
but, we'll take one more question
and then we should shut this down.
man: Yeah, sorry.
I apologize if this was covered, but was there any enhancements
that would benefit apps that still have to target
Like 1.6 and up, let's say.
Guy: So yeah.
If you target an older platform, you will have to do
some trickery to enable hardware acceleration,
but that's doable, and if you're--
we can't simply optimize the framework,
so if you don't use the hardware acceleration, you know,
we try to make the software pipeline faster.
I can't give you any number, but--
man: So I would be able to
add the line in the manifest and that still would work,
and then reflection?
Guy: Yeah. Yes, if your target--
If you change your target SDK to 11,
you can add the attributes,
or you can do it from code, you know,
you can have a special case,
like, if you're running on Honeycomb then...
man: Turn it on?
Guy: You can run that line of code.
man: Okay. Thanks.
Haase: And that's it. Thank you.