Practice English Speaking&Listening with: Google Container Engine and the path to cloud-native operations (Google Cloud Next '17)

FABIO YEON: All right.

Let's get started, then.

Welcome.

This is IO-314.

My name is Fabio Yeon.

I'm one of the engineers on the Global Container Engine Team

and tech lead on a team as well.

"Google Container Engine and the Path

to Cloud-Native Operations," or as I like to call it,

"the pi talk."

Wow, no math person in the room.

[CHUCKLES]

All right, so start with the basics--

containers.

How many people here have used or are deploying

containers in production?

Wow, that's a pretty good number.

How many of you have played around with containers,

in general?

All right, how many people have heard about containers?

There, OK.

So let's start with, then, some simple intro.

What are containers?

The most succinct explanation I've heard of a container

is that a virtual machine virtualizes hardware.

A container virtualizes a kernel.

So what happens, then, is if you have a bunch of containers

on the same machine, they're all sharing the same kernel.

But the container runtime provides enough abstraction

so each one of them thinks they have

the machine for themselves.

So, why are they useful?

Well, in this little box called a container,

you can put in just your application, its dependencies,

and you're done.

Compared to what you had to do to set up

a deployment on a VM or a bare metal,

you don't have to install an OS.

You don't have to configure your networking.

You don't have to configure and set up the dependencies.

It just shows you, your application, your binaries,

and a little container image, and you're done.

In many ways, this is what I call getting serenity

in a sea of chaos.

All right-- whoops.

Clicker.

So once you start playing around with containers,

you start running into little more complicated,

complex scenarios.

And this is where Kubernetes comes in and helps you out.

Your little box-- if you have one or two of them,

sure, it's easy to get them up and running and deployed.

Once you start getting more of those,

then you start getting a little bit more help.

Kubernetes helps you ensure that there's

a place for your container to run,

makes sure that the proper number of them are running,

makes sure that the network packets intended

for your containers gets routed properly through the cluster

and arrives at the appropriate place.

And, of course, if you use Kubernetes,

Google Container Engine provides to you

the best and purest Kubernetes experience you can get.

Beyond just allowing to set up a Kubernetes cluster easily,

Google Container Engine strives to provide to you

management capabilities that makes it easy

for you to maintain and keep your Kubernetes cluster

running.

So the short little intro, let's dive

in a little bit into what I'm going

to call cloud-native management capabilities

and show you some of the things that we do in Google Container

Engine to help you be more productive

and to care less about the machinations of your clusters

and go back to what you want to do more, which

is maintain your applications and your services

on Kubernetes.

"Change is the only constant in life."

I like this quote for many different reasons.

But that's because in the little container box,

you have been able to provide a level of stability

on your thing, on your applications and your service.

Remember that around you, the clusters,

things may be changing continually,

and you have to have a little strategy

to be able to manage some of those changes.

So just a quick overview of the agenda.

Gonna talk a little bit about Kubernetes, the Kubernetes

versions and why the version management

is kind of important for you.

Talk a little bit about node health--

what it takes to maintain your node health

and how to evaluate that.

There's a little bit about operations logs and then

talk about some strategies that we

have come up with for successful operations of a cluster.

So keeping up with Kubernetes.

So remember, Kubernetes is an open source project.

As such, it has its own ecosystem,

it has its own community, and it also

has its own release cadence.

Kubernetes strives to get one minor version

release a quarter.

Now [CHUCKLES] this is where a minor version is like a 1.4,

a 1.5, or sometime later this month,

I think a 1.6 is slated to come out.

It seems-- seemingly like a small thing.

So it's a small number change.

But because Kubernetes is still a fast-moving, fast-developing

platform, every quarterly release

tends to be a fairly large release, a fairly big release.

New features gets launched, some existing features

get upgraded from alpha to beta, beta to ga, and so forth.

So every minor release of Kubernetes

comes in as a big package.

If you look at the release notes or change

logs for every minor release, it is very comprehensive.

It is very extensive.

Beyond that, every couple of weeks patch releases come out.

These may be bug fixes, security fixes, small improvements

that were missed, and so on.

In Google Container Engine, our goal

is to provide to you the latest version of Kubernetes

as quickly as possible.

Order days, if possible, but if not, sooner.

And we also provide with you a collection of versions,

because we understand that not everything is perfect.

And therefore, sometimes you want to hold back a little bit.

So we provide to you the latest version of Kubernetes--

one older version, plus the latest version

of every previous minor version that's been released.

Now, I know that's a little hard to parse, which

is why I put an example here.

Currently, the latest version of Kubernetes available is 1.53,

and that is available.

The previous version of that is 1.52,

which is the previous patch release of 1.5.

And then we try to provide the latest version of the 1.4

branch, which is 1.49, and the latest version from the 1.3

branch, which is 1.310.

So why does collection of versions,

why this seemingly subset of minor versions and not others?

In Kubernetes, the master and the nodes

can be running different versions

of Kubernetes binaries.

And the official support states that--

by the community and so forth, is that the master and the node

can differ by, at most, two minor versions.

So Kubernetes Master is running at 1.5.

The oldest version of Kubernetes that can be running on a node

and still be considered supported

is something in the 1.3--

1.4, 1.2, 1.5, 1.3.

When 1.6 comes out, that will go up to 1.4.

Anything beyond that, meaning that any version skewed

beyond those two minor releases is considered out of support

by both Kubernetes and Google Container Engine.

It goes into what is called best effort.

Your node may still be running, it may still work.

But the likelihood is that some functionality may or may not

work and may break in the future.

So Container Engine, remember, will automatically

upgrade your cluster masters.

This is one of the things that we offer to you

when you create a Kubernetes cluster on Google Container

Engine.

Among the things that we do in managing

a master is to upgrade it to ensure that it is at the latest

version of Kubernetes.

So, as an example, if six months ago you created a Kubernetes

cluster on Container Engine at version 1.3, at some point,

your master was upgraded to 1.4.

And if you didn't do anything [INAUDIBLE] nodes at 1.3,

you were still OK.

It's one version difference.

At some point in the near past, it

was probably upgraded to 1.5.

So now your master is at 1.5 and your nodes remained at 1.3.

And 1.16 comes out, and at some point

in the future, when Google Container Engine upgrades

your masters to 1.6--

uh-oh.

If you have upgraded or nodes, you're

not officially out of support.

So if [INAUDIBLE] to do anything, because

of the management of the master and the auto upgrade that

exists, you have to upgrade your nodes

at least once every six to nine months to at least stay

with the support band.

Of course, we highly recommend that you upgrade more often

to pick up some bug fixes and security

fixes that may have [INAUDIBLE] show up in Kubernetes.

So we come now to the very first feature

that we have on Google Container Engine

to kind of help you manage some of this change.

Node auto upgrade.

Just like the cluster master, where we manage it for you,

node auto upgrade will monitor the nodes

that you have in the clusters.

And if we detect it, it is, shall we say, older

than the master, we will automatically trigger

an upgrade of the nodes.

It is the same exact logic that is applied if you

upgrade your nodes manually.

It can be enabled and disabled at any time.

And just briefly, you can actually do this--

whether the auto upgrade is enabled or not,

you can always upgrade your nodes manually first.

If you want to try out, test out things,

or try to get the latest version into your cluster sooner.

Now, we kind of try to do this also in a very good judicious

manner.

Patch upgrades, because they're typically bug fixes or security

fixes, tend to be applied much more quickly.

So if my nodes are at 1.50 or 1.51, for example,

and I enable this feature, I will probably be upgraded to 1.

153 fairly quickly.

Because we understand the minor version, changes on Kubernetes

can be more disruptive.

And even though Kubernetes and the community

tries very hard to make sure that it is not it is not

very disruptive and lack of compatibility

and all those things are taken care of--

but because we understand that sometimes they

can be a little bit more disruptive,

minor version updates happens at a slightly slower pace.

So, for example, the 1.4 to 1.5 upgrades

will happen either later cadence than the minor patch releases.

So how to enable this thing?

You can enable in the gcloud.

When you create a cluster, it is on the gcloud beta channel

set of commands.

The flag is a enabled dash auto-upgrade.

You can also do it when you create a node pool,

and you can also do it in the UI at cluster creation time.

Like I mention, you can also enable or disable

disk of ability on node pool at any given time.

One thing to keep in mind, if you disable this feature

while an auto upgrade is running,

it will run to completion.

It just means that next time you do a scan of your clusters,

it will not auto upgrade it.

Now, let's pick a little bit behind the scenes

as to what Google Container Engine is doing for you when

this thing happens and go over the set of scenarios in case

you want to opt out of this and do it yourself.

So first thing, as I mentioned before, you can always

trigger upgrades manually-- does not affect the settings

for auto upgrades.

The first thing is that every week, Google Container Engine

does a push, and we do release nodes.

Among the things we publish in this released node,

along with bug fixes and teacher announcements,

is what versions of Kubernetes, if any,

are made available to you as a user.

We also provide the same functionality or notification,

both in the CLI, via gcloud whenever you list it,

or in the UI.

Notice that there are two versions that

are listed in this particular output

have two different types of asterisks.

The more asterisks there are, typically the worse, the more

dangerous, the worse it is.

In this case, it is telling you that while my cluster is

one version behind and the second one is

two versions behind, I, unfortunately,

did not have one that was three versions behind.

But you can imagine that would be--

the mourning would be even more severe in this one.

And in the UI, of course, I think

there the upgrade available notification

is available, as well, whenever an upgrade is

available for your nodes.

If you want something a little more programmatic,

gcloud container get server config will give you

all the current information about

the potential configurations of your Google Container Engine

for a particular zone.

And its particular output, you can

see that the default cluster version is 1.53,

but the valid masters are now 1.53 and 1.49,

which is the latest version from 1.4, which we still

are currently supporting at this time.

And then your nodes can be any number of these versions.

Notice that even though 1.27 is still available, it is only

supported if you actually have your master at the 1.4 branch.

If you have a master in the one 1.5 of Kubernetes,

the oldest version of Kubernetes that

could be supported on the node is in the 1.3 branch.

So thinking a little bit back.

I mentioned that Google Container Engine managers

your masters.

One of the things that we do is upgrade them.

You yourself can trigger an early upgrade of the master

before we get to it, if you still choose to do it.

And the option is from the command line.

It's a Container Engine upgrade, and then you

add the dash, dash master option.

This tells the system that you want to upgrade the master.

And then you can go ahead and trigger a manual upgrade

of your nodes with a send command, except this time,

you just take out the dash, dash master.

Why do we upgrade the master first?

It's because the node can, at most,

be at the same version as the master.

You cannot have a node that is a version higher,

even a patch version.

So in the hypothetical world, the 1.54 were to come out today

and made available to Container Engine customers.

If you want to try out 1.54 before we upgrade your cluster,

you would have to upgrade your masters first to 1.54

and then upgrade your nodes in your node pools

to 1.54 manually.

And this is regardless whether you

have enabled auto upgrade for your nodes or not.

You can just do it yourself.

Ah, so we've started doing an upgrade

or we trigger an upgrade of the nodes,

and you detect that something is going awry,

something has gone wrong, and you need to stop.

We have two sets of commands that are currently

in the alpha channel of gcloud.

The first one-- actually, the second one this list,

but I think the more important one is you just

want the upgrade to stop.

So you call gcloud, alpha container operations,

cancel and give you the operation ID, which

you can get it from listing the operations currently running.

The operation list is probably going

to say something like auto upgrade

or a manual upgrade, cluster upgrade.

Node upgrade, I think.

You can cancel the operation.

Cancelling an operation does not stop

an ongoing upgrade of a node.

So if a cluster has five nodes and you detected that

after the first one, something has gone south

and you want to stop.

If the second node upgrade has already started,

the cancelling of the operation will--

that upgrade will finish, but the subsequent upgrades

of the nodes in the node pool will not start.

The upgrade will be canceled at that point.

Now, your cluster is going to be in this kind

of a mixed-mode version between couple

of nodes are in the new version, a couple of nodes

in the old version.

So then you can do what's called a rollback.

If you call gcloud alpha container node pools rollback,

we will then take those nodes that were at the newer version

and roll them back to the previous version.

Couple things to keep in mind, as well, as you're doing this.

This is a recreation of a VM.

So if you happen to have any local configuration

or local data stored on those VMs,

fortunately, they don't not survive an upgrade operation

at this time.

Healthy nodes, healthy clusters.

Keeping your nodes healthy can be a little tricky.

The nodes can go unhealthy for a number of reasons.

You can run out of resources.

Maybe you over provisioned stuff and you used

too much memory, too much CPU.

It can run on a local disk.

Maybe you have a configuration problem or a bug in Kubelet

that is causing it to crash.

Maybe your workload is triggering a kernel bug,

or, of course, network segmentation network drivers

crash, so on and on.

Some of these are auto recoverable by Kubernetes,

specifically the first two, where you run out

of memory or resource.

Some signals provided to Kubernetes

to the cluster master and the rescheduler controller

on the master, you're supposed to observe

these things happening and say, oh, this node is overloaded.

Let me just take away some pods from it

and scatter them elsewhere.

So they're self-healing for the most part.

As long as you have enough resources in your clusters,

then there's nothing you need to do.

But if your click Kubelet starts crashing on you,

if your current node became deadlocked

because you happened to trigger a kernel bug, what then, then?

This is where it starts getting a little more interesting.

Second thing to remember about node health

is that it is the master's evaluation of your node's

health that is important.

Why?

Because it is the cluster master that determine what pods

are scheduled where.

So if your master thinks that your node is unhealthy or vice

versa, you may not schedule anything on it

if it's considered-- if it thinks it's unhealthy,

or if it's giving a false signal that is healthy but it is not,

and we try to schedule work into an unhealthy node.

So that's something to keep in mind.

It's kind of important, is that the self-evaluation

of the health of the node is less important here.

It is what the master thinks of your node's health

at that time that is important.

And, of course, repairs like upgrades

are typically limited to recreations of a node.

Oops, wrong button.

So this comes to a second [INAUDIBLE] container

on Google Container Engine to help you manage your cluster.

It is node auto-repair.

It is the semantic equivalent of the master repair

that we already do for your cluster today.

But this-- this is where we are observing and monitoring

your nodes.

We are taking into account the master's view

of your node health state.

We are also evaluating other signals around it.

We take a look at the signals from the managed instance

group backing your node pool.

We also sometimes take a look at the signals

provided by the node itself.

With those signals, the of it all, be coming to an evaluation

whether your node is healthy or not.

Too many unhealthy signals will trigger repair--

in this case, a recreation of the node, in

attempt to get it back into a good state

and having rejoined the cluster.

We also try to make sure that we rate limit this so that we're

not doing this too often.

Now, why rate limit?

It's because sometimes the node going on health

and may be triggered by your workload.

And if you start triggering this wait too often,

we may be in a situation where the node never

is alive long enough to do any useful work.

So we have somebody do some of the rate limiting as well.

Currently in beta-- as a matter of fact,

I think it's rolling out as we speak,

and so fairly soon you should be able to have

this capability show up in the UI and in the CLI as well.

How do I enable it?

Just like auto-repair, the flag is

dash dash enable auto-repair.

You can do this at cluster creation

time, at node for creation time.

And, as a matter of fact, I would highly

recommend that you enable both of them--

auto-repair and auto-repair, to make your life a little easier

when you run Kubernetes on a Google Container Engine.

One more thing before I go into the next slide,

which I've [INAUDIBLE] here.

You can also enable and disable this at any given time.

It is just like for auto upgrades.

You do a gcloud beta container node pools update,

and you can enable or disable auto-repair on your cluster--

on your-- sorry-- on your node pool

at anytime you that you want.

So let's now peek into behind the scenes

and see a little bit of what Google Container

Engine is doing on your behalf.

And kind of give you an idea if you wanted to do this yourself,

what kind of things you have to do.

So I mentioned that it is the master's view

of your node health that matters.

So how does a master know what's going on in a node?

The Kubelet publishes to the master periodically

a bunch of information.

And one of them is its evaluation

of its node health plus what's going on in its internals.

All this information is available from the node object

from your Kubernetes master.

So from the command line, you yourself

can get the same data by calling [INAUDIBLE] control, get nodes.

Of course, in this output, it gives

a little more formatted-shortened version

of it.

But if you see the JSAR or YAML output,

you can actually get at the raw data.

And in a little bit, in the demo,

kind of show you how I can look for this.

So you're listening your nodes, and you're determined

and one of them has gone unhealthy, not ready.

So what are the steps now for you

to try to get back to a good state?

Well, the first thing you should do is cordon off your nodes.

What this thing does is tell the scheduler,

hey, regardless of what's going on in this particular node,

don't schedule any further work on it.

Next thing we do, we attempt to do a drain,

try to do a graceful drain of all the pods,

make sure they all go away some place

so that the pod becomes empty.

This may or may not succeed depending

upon the state of Kubelet and other things on your node.

If your node had a hard deadlock because of a kernel issue,

this may not work.

But at least we try.

The next thing is we try to list your manage instance group back

in your node pool.

And from that manage instance group,

we have to identify the specific node from that

and then try to tell the manage instance group, hey, please

recreate that node.

Because we have set up--

on Google Container Engine we have set up your cluster

using instance group templates, a simple recreation

of it using the existing template

gets your back a node into your cluster.

Operations log-- [CHUCKLING] as we

start getting more and more capabilities on Google

Container Engine to help you manage your cluster more

easily, a question that keeps coming up is, like, hey,

how do I find out when you guys have done something

on my cluster?

So the first that we have done right now

is that we are augmenting the operations log for a Container

Engine to contain more information about when

we do automated operations on your clusters on your behalf.

So before we had repair clusters and upgrade masters available,

and we're adding auto-repair nodes and auto-upgrade nodes

as additional operation log types

so that you can find out whenever we have done

these things on your behalf.

How do you get to the operational log?

Well, gcloud container operations list

will give you a order list of all the operations that

have been done on your cluster.

Actually, I think it's beyond just

your cluster-- is in your project across your clusters.

And then if you do a container operations describe

on an operation, it gives a little more detail

as to what we did and what the status was.

Were we successful?

Did we fail?

So on and on.

Node pools and resource management--

another favorite topic of mine.

How many people here have heard about node pools

or have used node pools?

Handful.

OK, good.

So what are node pulls?

Node pulls-- the shortest definition is just a collection

of nodes managed together.

From a Google Container Engine perspective,

what it means is that you can have multiple node

pools in your cluster, each are defining

a particular configuration.

And When I say configuration, I mean machine type,

machine resources, and Kubernetes versions,

scopes, and on and on.

Within a node pool, all the machines

have to have the same exact configuration.

They have to be the same exact machine type,

they have to have the same exact machine--

Kubernetes version.

They must have the same exact machine resources.

So if I ask for an n1-standard-4, with, let's say,

two local SSDs, every single machine in that node pool

will have the same configuration.

But, of course, I can have multiple of these in a cluster,

so this gives me an additional level

of functionality-- flexibility, as well, where

I can have different machine types and different--

and mix and match these to meet my workload

and my resource needs more carefully.

And this is also a foundational unit for many management stuff

that I talked about, so when, before, I was talking

about cluster creation, enabling it

at cluster creation or node pool creation,

when you create a cluster on Google Container Engine, what

is actually happening is that we're

creating the master and so forth,

and we are also creating a default node

pool for your cluster.

So all the options available for creating a node pool

are available on your cluster creation as well.

So let's take a look and some things

that you can do with node pools might be kind of interesting.

So can we switch to the demo machine, please?

Ah, OK.

So you go container clusters list.

And here I have my cluster that I created for this demo.

It is currently running on master 1.53, and oh, wow--

my master was upgraded since I ran this demo,

so my nodes, right now, are version 1.52.

So now I have a notification that a node is--

upgrade is available.

It also tells me that my current cluster is running

and, of course, my master IP address.

Let's take a peek underneath, then,

at my node pools for this particular cluster.

Of course, pedantically-- has to be plural.

I have three node pools configured for this cluster.

I have a main focus comprised of [INAUDIBLE].

I have something that I call a pre-pool of n1-standard-2s,

and something that I called a lowmeme pool, which

is running the custom machine.

A custom 2x2048, which signals to me

that this is a two-core machine with only two gigs of RAM.

Let's then describe the main pool--

oops, let me pipe that.

OK.

n1 [INAUDIBLE] 4, identification scope's defined.

The number of initial counts enabled.

And, of course, in our management,

I can see that I have enabled both auto repair and auto

upgrade for this particular pool.

OK, I think-- interesting on this one.

But if I go to my pre-pool, among the things that you'll

notice on this particular listing

is that this thing is comprised of preemptible VMs.

Preemptible VMs, of course, are options provided to you

from DCP, where the costs for these are lower.

But in return, the VM may be preempted underneath of you

with a moment's notice.

But if you have a workload that are preemptible-friendly or can

be preemptible easily, this is an easy way for you

to instantiate a couple node pools in your cluster to have

this and leverage it--

the lower of cost that they provide.

And then, of course, the final one, which is my lowmeme pool.

It is just like my main pool, except the difference here

is that I'm running custom 2x2048 machines.

So, like I said, very low memory,

high, superior kind of workloads would benefit

from this kind of node pool.

So previously, I have already pre-deployed a number of pods

on this particular cluster.

So let's take a look at all of them,

make sure they're all running.

So I have deployments that are very high CPU.

Some deployments that are just regular, main workloads.

And I have something that I'm calling

preemptible workloads are workloads

that are eminently preemptible.

With a little bit of trickery--

and this is where I will cut and paste the command,

because I can never remember from memory.

I called [INAUDIBLE] control pause

provided a JsonPath path that says,

hey, fish out this little fields from your particular listing

and then combine them to write this output.

So in this view, you can see that all my pods are listed

there like before, but now I'm also listing

which node they are running on.

And you can see this thing scattered all over the place.

My high-CPU workload's running in a variety

different machines.

My main machines-- I have main workloads running

on a variety of other machines.

But more importantly-- they've been not a good idea--

I have some workloads that are running on my preemptible VMs.

Should be OK, maybe, but certainly not ideal.

Because maybe they're not workloads

that you want to run on a preemptible VM.

So, how do we then take care of that?

Let's take a look, then, in a node

and see what I can find out.

So if I list the raw Json object for a node,

you will notice a couple of things here among the metadata

so far.

It is that there's a whole section for labels.

Some of these labels are applied on each node

by the Kubernetes runtime, and they are identified easily

by the kubernetes.io prefix.

So thing like machine architecture, in the 64,

machine OS--

Linux.

Down here-- host name, host name on machine.

You'll also notice a few of these

that are actually set by Google Container Engine itself.

They are prefixed with cloud.google.com.

In this case by default, every single node

that we create on Google Container Engine

has, automatically, as part of it,

the node pool name associated with it.

And, of course, the other thing there is this other label.

On Google Container Engine, when you create a node pool,

you are allowed to provide to it a comma-delimited list

of key value pairs that become part of your node labels.

So you can, then, use these on your pods spec in Kubernetes

so that it ensure that those workloads, those pods,

ends up at a node that has this label in them.

One more thing I want to show up along here.

This is another node.

In this case, this happens to be a VM that is a preemptible VM.

When you create a preemptible VM--

a node pool, sorry-- in Google Container Engine,

we will automatically add apply this label as well,

so you don't have to do it.

You can certainly add more, like I had done below,

where I've added the pre label as well, where you can actually

add the Google Cloud Comp.

You can use this label as part of a node selector on your pod

spec as well.

Oh, before I go off--

I mentioned earlier about node condition.

If you want to explore a little bit more

about doing the node health evaluation yourself,

every single node contains this array of node conditions

under the conditions list.

And there are a variety of these.

Some of them talk about [INAUDIBLE]

health, disc, memory, pressure.

And, of course, the bottom one is whether the Kubelet is ready

or not.

This is part of the signal that is pushed from the node

to the masters periodically to let the master know

what the status of the node is.

And this is the signal that both the master and Google Container

Engine uses--

well, one of the signals--

to determine whether the node is healthy or not.

So back to this demo.

I have, then, deployment YAML for my high-CPU workload.

Of course, I have commented, for demo purposes,

the node selector.

So let's go ahead and delete the comment tags

and make sure that the node selector's there.

I'm gonna go to my other workloads as well.

And remove the node selector comments.

And notice that on this one, at least, while on the other ones

I've used my custom labels that I've set up on each node pool,

on this one, I just decided to use

the one that Google Container Engine sets

for preemptible VMs.

Might be kind of nice, because at this point,

I'm not relying upon a custom label that I've set

or the pool name or anything like that.

I'm just using an automatically set label that Google Container

Engine has put on it.

So it's available across any VM that is printable.

Let me go ahead and replace the deployment on Kubernetes.

And preempt.

OK?

Give control, get pods.

Bunch terminating, bunch running,

bunch container running.

We'll let this run for a little bit,

let everything kind of settled down again.

Mhm.

Wow, taking a little bit longer.

OK, things seem to be settling in, settling down.

Things are getting scheduled.

Excellent.

Ah, everything is running again.

So if we go back to that somewhat complicated Kubelet

control that I had before, what you'll find here now

is all of my high CPU deployment--

it is now running on my low-memory pool.

My main workload is my main pool.

But more importantly than anything else, my preemptible,

friendly workload is now all scheduled

and running on preemptible VMs.

So with this, I was able to, shall we say,

make sure that my high-CPU workloads end up on machines

that have very high CPU of low amounts of memory--

save me some money, ensured that the proper VMs

are assigned for those.

My main workload can be whatever.

My generic set of workloads can go on the main workload.

But more importantly, my preemptible

workloads that low priority, low cost are all set up

and now are all running on my preemptible VMs,

exactly as I desired to be.

Couple more things to keep in mind--

there are more node selector options

becoming available to you as part of the Kubernetes 1.6

release.

So, for example, node selector right now is a match.

One of the things that is coming up

that might be kind of interesting, especially

in this kind of scenario, is the [INAUDIBLE].

You may have workloads that I don't really

care where they run, I just don't want them

to run on this type of VM--

preemptive VM, for example.

So that's one of the options that will come online--

available in the 1.6 timeframe, I

believe, where you can just state that in your pod spec

and will be available when you deploy them.

So can you go back to the slides, please?

All right.

So a couple of slides I put it in here for you to reference.

Feel free to take a look at it, how to list some node pools,

how to create the node pools that I've-- whoops, sorry--

that I've used for this demo.

Notice the machine types, notice the dash,

dash printable option as an option, and the node labels,

how to specify these on your node pools.

And, of course, in assigning pods to node pulls,

it's the node selector option in your YAML file.

OK, some final notes.

Upgrades can be disruptive.

So to help mitigate them, you can use--

the upgrades are for node pool.

So you kind of use them to segment

the upgrades of your cluster so that you can have either

a canary test kind of pool, you can try out newer versions.

So you have an existing workflow that is running at version 1.51

and you want to try a 1.53, you can

spin up another node pool at 1.53,

schedule some workload on that particular node pool,

make sure everything works fine before you

let the rest of your cluster be upgraded.

Consider having what I call the one-node slack

to ensure proper capacity.

When we do upgrades, we take down one node, do the upgrade,

bring it back up.

Go on to the next node, and so on.

While the one node is down and being upgraded,

it would be nice for you to have enough slack on your cluster

so that whatever pod was running there can find a new home

while the upgrade is occurring.

Google Container Engine limits you

to having one cluster operation running at any given time.

Now just to differentiate between the Container Engine

operation versus the Kubernetes operation, for example,

these are things like adding or deleting a node pool,

updating a cluster-wide configuration,

doing an upgrade, a repair, any kind of events.

All those are serial on Google Container Engine

for various logistic reasons.

So for the most part, this restriction

is not a big deal except if you're

upgrading a fairly large cluster or node pool.

And you can imagine that because we're

doing the upgrade sequentially, it

may take a little while for it to complete.

In the middle of it, if you have to do something

on your cluster, go ahead and cancel the upgrade.

It will stop at a good point.

You can do whatever operation you

have to do on your clusters, and then

re-issue the upgrade command.

It will just kind of pick up from where it was left,

scanning your nodes there we have,

skipping those that have already been upgraded and just

keep on going the rest of them.

And, of course, we're truly trying

to make progress and try to minimize some disruptions.

And we have more capabilities that we're

going to be adding in a future for upgrades and repairs.

Some of the best practices that we have found--

pod disruptions are a reality.

Try to prepare your workload for them.

There are a number of capabilities

being added to Kubernetes that are gonna help you with that.

So please do take advantage of them.

And we'll keep integrating them into Google Container Engine.

Do use deployments of replica sets for your pods.

Make sure that you have enough sufficient redundancy.

The second point is actually kind of important.

In the cloud-native world, please

to not write set once and forget configuration scripts.

The mindset you need to have is reconciler.

You need to write reconcilers that read the current state.

If it's not the desired state that you want,

do some action to try to get there.

And then once it's completed, go to sleep and then

start over from step 1 again.

This way, whether it's a repair event, an upgrade event,

or some other things that happening outside

of the container, if you have some configuration

that you absolutely need, you know

that at some point, if it goes out of alignment for whatever

reason, it is a temporary thing that hopefully

should autocorrect itself.

And, of course, please use node pools.

Organize your resource more efficiently in your cluster.

Allow you to have [INAUDIBLE] configurations.

Leverage different types of machines, different types

of resources, and so on.

A quick recap-- change is constant.

Please be prepared.

Do enable auto-upgrade, and then try auto-repair.

Operations logs-- please look into them

to see if-- whenever we do something,

we'll try to make sure that they all end up in your operations

log for your clusters so nothing is hidden.

Use your node pools to organize your resources.

And, of course, think reconciler, not set and forget.

For those that may not be familiar,

here are a couple links for the Container Engine website,

the Google Groups where the release notes come out

whenever we do a push on Container Engine.

A lot of members of our engineering team

are on Stack Overflow, so if you have a question that

is container-engine specific, please use

the tag, the Google Container Engine tag in Stack Overflow.

And, of course, as always, the Kubernetes

GitHub depositary of highly technical, kind of complicated.

But always looking for contributors for the Kubernetes

project.

And while you're here at GCP Next,

I highly recommend going to other container talks

if you want to find out more about us.

IO205, I think is this afternoon and it's

a good intro to other capabilities and features

that are being added to Google Container Engine.

And, of course, I also want to highlight IO307,

even though the title says ABC is a--

Google Container Engine tips and best practices

is actually a talk mostly about monitoring with your Google

Container Engine cluster.

And with that, I want to say thank you for coming.