Practice English Speaking&Listening with: Cloud AI Platform Pipelines overview

Difficulty: 0

STEPHANIE WONG: The flexibility of Kubeflow

lets you run your ML jobs anywhere you can run Kubernetes

and gives you an entire platform to work with.

Sometimes you don't need the entire Kubeflow platform,

but you still want to use pipelines to orchestrate

machine learning workflows.

Let's look at Cloud AI Platform Pipelines, a Google Cloud

service that lets you focus even more on your ML

and less on your infrastructure.


If you want to read more about how

to run Pipelines on Google Cloud,

check out the documentation link below.

Previously, I talked about how to work with Pipelines as part

of a Kubeflow deployment, but we can use Cloud AI Platform

Pipelines to get the same user interface

and back-end functionality.

If you don't need the rest of Kubeflow

and are just looking for the ability to work with pipelines,

Cloud AI Platform gives you a quick way

to set up an installation.

On top of that, authentication with other Google Cloud

services is built in.

This lets you focus on building your pipeline

and integrating with other Cloud services

without needing to manage a complex deployment.

Let's start by getting a Pipelines installation set up.

To try it out, head to the Google Cloud console

and use the left navigation to find Pipelines

under AI Platform.

Then click the New Instance button.

Google Cloud will set up a Kubernetes cluster

and then deploy the Pipeline installation on top of it,

but all the details are handled for you.

Once it's set up and running, you'll

have the full Pipelines UI and SDK to start working with.

We'll go back to the Pipelines page

and click Open Pipelines Dashboard to get

to the same user interface.

We can see experiments and runs here, as well as look

at our metadata and artifacts.

Don't forget to check out the previous metadata management

episode, which goes into more detail about it.

So it's still possible to upload a compiled pipeline

through the same user interface, but what if you're using

a notebook for your model?

Here's where we can use another service to help us out--

Cloud AI Platform Notebooks.

In our previous videos, we've talked

about the advantages of running a notebook in the cloud

rather than on your local machine.

Cloud AI Platform Notebooks gives you

a simple way to do just that.

You can create a new notebook quickly and jump

into a JupyterLab interface.

Back in the Google Cloud console,

just click New Instance, and choose an environment.

Since we're working with a TensorFlow 1 example,

we'll choose TensorFlow 1.1.5 without any GPUs

and hit Create.

Once again, all of the infrastructure

is taken care of behind the scenes,

and we're able to just click Open JupyterLab to get

to our interface, where we can create notebooks and execute


Since we're using the same notebook from the last video,

we've already cloned the same repository here.

Code for Pipelines is made to be reusable

so we don't need to change anything

to create an execution.

We will need to specify the same properties-- our Google Cloud

project and the Cloud Storage bucket where

our model will be stored.

There's one more change we need to make.

We have to tell the pipeline's SDK where our pipeline's

endpoint is located.

This is as easy as copying the URL of our hosted pipeline,

which we can get by clicking Settings next to a pipeline

and storing it as a variable in our notebook.

Since the same user created both the notebook and the pipeline's

installation, the authentication is taken care of for us.

We'll then modify the line where we execute the pipeline

and point it to the server and then run the notebook.

Other than that, this is the same pipeline,

so we'll still be using the Cloud AI

platform to do the training and serving of the model.

We're just doing it from a different environment.

Cloud AI platforms can be an easy way

to get a pipelines environment without having to set up

an entire Kubeflow deployment.

In addition, it's easy to integrate with other Google

Cloud services to help extend your ML projects,

like hosting notebooks, model training, and serving.

Try it out for yourself.


The Description of Cloud AI Platform Pipelines overview