Practice English Speaking&Listening with: Matan Berkowitz: "Music is the Instrument" | Talks at Google

[MUSICAL FLOURISH]

[APPLAUSE]

MATAN BERKOWITZ: Hi, everybody.

So we're going to go on a little journey from music--

[DIGITAL MUSICAL NOTES ACCENTUATE HIS SPEECH]

--to tech and impact.

I'm going to go for different projects

and explore the evolution and ideas behind them.

But first, I'd like to start from the end.

Because for the past year and a half

or so, my team and I have been working on a device

and not really showing it to anybody.

And I'm super excited to be able to unveil it here

for the first time and share it with all of you.

So basically, I want to start with a question-- what

is the craziest thing that you can do with your tongue?

[LAUGHTER]

We're all thinking about the same thing, right--

controlling a wheelchair.

[LAUGHTER]

So-- [LAUGHS] What we did was basically to create this

device, Smart Braces, that fit in the mouth and enable you

to control digital devices around you--

like a phone, a computer, a smart home, and even

a wheelchair--

like this Medical Sunrise wheelchair right next to me.

[MURMURING FROM CROWD]

[APPLAUSE]

So how did we even get to the point

of trying to control wheelchairs with our tongue--

successfully or not?

[LAUGHTER]

It all starts for me with music.

I was performing live ever since I was 14 years old.

And after over a decade of playing shows and being

in the studio, I started getting really

frustrated with this notion of dudes in t-shirts

playing guitars.

And I felt like there has to be more,

both as an audience member and as a performer.

I wanted to expand the spectrum of what it means to play music.

Also, I feel like on some level, spending so long pursuing

your childhood dreams without really

trying to affect anybody else's lives in a really

specific or meaningful way starts

to become kind of a burden.

There's this emptiness to doing something only for yourself

for so long.

And these thoughts for me basically led me to experiment

with technology.

And specifically I was very drawn to sensors

and the idea of using the body to make music.

And this kind of became my main question at the time--

how can we translate the human body

into music in new and interesting and innovative

ways?

And this led me to do these conceptual tech infused shows,

like this one that was actually for Google

during the anniversary event in Israel,

where everything on stage was translating physical signals

into music.

So this woman, for example, was running,

and her heartbeat was creating the rhythm.

This guy was meditating, and his brainwaves

were generating the melodies.

My movements were tracked by a camera

so the motion would orchestrate the entire show.

And none of this was using any traditional instruments.

It was all the biological symphony of the human body.

And when we're thinking about the biological symphony

of the human body, I like to arbitrarily think about it

as two different kinds of signals.

The first one would be the physical, voluntary signals

of the body--

things like your movements and your voice.

These are the things you can actually control.

These are the things that you can

use to express yourselves like you would

with any musical instrument.

But then on the other hand, you have other kinds

of signals which are more like biological involuntary

signals--

your heartbeat, your brain activity.

So just like in the video before, you could influence

these kinds of signals.

By running, you could accelerate your heartbeat.

By meditating, you might be able to calm down

your brain activity.

But you can't really control them.

You can't decide what you want to play and just play it.

And therefore, the first one has more

to do with you playing the music,

while the second one has more to do with the music playing you--

like musical biofeedback.

The first time I ever saw this graph

by the Israeli start-up Neurosteer,

I was really inspired.

This is basically a guy who has been meditating

for many, many years connected to EEG, which

is a technology that measures and tracks

the electrical activity in the brain.

And using this kind of technology,

Neurosteer was able to show how at a certain point, when

he went deep into trance, his brain signal completely

changed.

Because you could see-- it's basically alpha brainwaves

showing you how focused or how tired

or how engaged someone is, and at a certain point,

when he became deep in trance, it completely

changed that graph.

And when I saw this, I thought about this as musical notation

almost.

It looked to me as if this was playing music back at me.

And I started to think about, how

can we use this to make music?

And so I collaborated with this company

and with a really inspiring individual named

Sefi Udi, who is a paraplegic.

He's paralyzed from the neck down, but before his accident,

he used to play the guitar.

And he wanted to play again and didn't know how.

And so what we did was we created a musical interface

using the EEG device.

And we enabled him to start controlling the music.

And this is kind of an important differentiator between the idea

of, I hear a melody in my head and my thoughts

are being played back--

like, [HUMS],, and the music just plays--

that doesn't really work.

What does work is by influencing your alpha brainwaves--

by calming yourself down or getting yourself worked up--

you could trigger different things in the music.

And Sefi was actually probably the most proficient

at this that I've ever seen.

After doing this with him, I actually

used this technology myself on stage,

and I gave it to other people.

But nobody has been able to replicate his level of control

over the music, which I think has a lot

to do with the sharpness of his mind

and the condition of his body.

And this person would actually come back and spiral back

in this talk, so try and remember him.

I then started playing around with this idea

of wearable devices.

The first thing I've ever created in this realm

was actually using the Google Glass--

the first edition back in the day.

And we used the head tracking feature of Google Glass

to enable you to play music using your head movements,

which led to some really bizarre shows.

And I then started thinking about,

can we take the two signals and combine them

into one creating this musical hat?

So the hat had this idea of head tracking,

where you can use those motions to consciously control

the music, and you could use these buttons to influence

the music at will.

But in the inside of the hat, there

was also the EEG device, which measured

the electrical activity in the brain.

And so the show was this weird combination

between you controlling the music and the music

playing back your biofeedback.

It then started becoming more and more

grandiose and ambitious in terms of shows.

This show for Microsoft was just pouring everything into it.

We had actual instruments-- like the aboriginal ancient

instrument of the didgeridoo, a drum kit, a guitar--

but we also had the EEG device and the heartbeat sensor

all playing together.

The different signals were being sent from the instruments

into these plates of sand and water

that would create these geometric shapes

and form according to the music.

And this is a really interesting phenomenon called [INAUDIBLE]..

So this idea of [INAUDIBLE] is really just the forms

and shapes that different frequencies and vibrations

get when they run through physical matter.

You normally have this kind of plate set up

with water sand in it, and as you

send the vibration or frequency into the plate,

it changes the form.

And it forms a very specific shape every time.

And if you do this right, it will form the same shape

every time, which is fascinating.

So for example, in this video of a demonstration

we did of this, we were sending 440 Hertz, which is basically

the note A. So the note A had this shape.

Interestingly enough, this wasn't really

a complete shape in my mind.

I felt like it was somewhat incomplete.

And so we tried other shapes.

We sent 563 Hertz into the same plate of sand,

and it became this shape.

And then, we realized we might try and send the golden ratio--

963-- which is of great significance in math

and in nature--

and see what this becomes.

And surely enough, this was a really interesting result.

AUDIENCE: Is the whole plate vibrating,

or it just at a point?

MATAN BERKOWITZ: The whole plate.

It has a speaker beneath it, and it

receives the entire vibration.

It then started occurring to me that I

have to literally fly to all of these places and be on stage

and do this myself, where it could also

be just objects or sculptures or interactive art

installations doing the same for many more people.

So we created different types of installations.

This one, for example, is a sculpture of a heart.

And when you touch your hand at the center of the heart, when

you put your palm at the center of it,

it plays back your pulse.

[MUSIC PLAYING]

This next installation is called Rirrom,

which is the word mirror spelled backwards,

and it mirrors your movements with lights and with sounds.

But it gets really interesting when two people use it

at the same time, because then it

starts reflecting or mirroring how in tune

and how harmonious they are with one another.

Only when you move at the exact same way

and the exact same time does the piece really come to life.

And a third example of these interactive installations

is a thing I created with my friends

near [INAUDIBLE] called Portals, which are different trees that

are interconnected and are communicating with one another

when you touch them.

So when you touch the first tree, you start charging it up.

[MUSICAL NOTE SUSTAINING]

When it's fully charged, it activates the second tree.

[MUSIC PLAYING]

Then someone at the other side could answer your call.

[MUSIC PLAYING]

We created all the different versions of this interaction.

This one was in Sweden and had two trees

right next to each other.

But we also created one where the first tree was

Tel Aviv in Israel and the second tree

was in Stockholm in Sweden, and people

could communicate not only with the trees

but also with these screens and cameras connecting the two

countries-- which was basically a very elaborate way

to get Israeli men to hit on Swedish women.

They were just, like, touching the tree, you know, going like,

is it cold there?

What's your name?

[LAUGHTER]

So doing these kinds of things was

kind of starting to answer the first question asked

in the beginning, which was how to, I guess,

make it more interesting and more innovative

as far as art goes.

But the second question I was asking myself

about the impact or meaning of this work was still unanswered.

I felt like these things were cool and they were interesting,

but they weren't really creating real impact in the world.

And so the next question became how can we

create real impact using this kind of work?

And I started understanding the beginning of my answer

when working with Raheli, who was at the time 9 years old

and living in Jerusalem.

And she was both physically and cognitively impaired

to the point where she couldn't really do anything.

She couldn't speak, and she couldn't really

move other than moving her right hand

in a very kind of general way.

But she had this amazing smile and kind of infectious energy,

and we really wanted to do something for her.

My friend Araz, who knew her, asked me,

how can we enable this person to play music?

And this was quite a riddle.

I framed it as one button machine,

because you only have one button--

in this case, this one movement of the hand--

and you have to use this one button to create music.

And so what we did was we created this glove

and gave it three different modes.

And the first mode was drum mode,

so every time Raheli would do this motion, one time would go,

[THUD].

The second time would go, [CLICK]..

And so when she'd go--

[USES THE TWO TO IMITATE A DRUM] She could start playing music.

And she could listen and kind of feel that sense of rhythm

in her arm.

The second mode was DJ mode, where

we took her favorite song, which is

a super happy song about the Messiah,

and we broke it into pieces.

And basically, every time she would trigger the next part

with her hand, she had to really listen.

Because it could be too early or too late in the song--

basically, really making her musical hearing develop.

And then the third mode wasn't even musical.

We invited Raheli's mother to my studio,

and we recorded her reading Raheli's favorite children's

book.

And then I cut that book into different sentences

and again gave her that active experience

of controlling her own storytelling experience.

So instead of her just being a passive listener,

she could actually trigger the next sentence

whenever she wanted to.

Doing this kind of work with Raheli and with Sefi--

the person I mentioned earlier--

kind of framed the next question that

was forming in my mind, which was how

can we make music accessible?

Maybe that is the path to the kind of impact

I was trying to find.

And so, this theory called the music accessibility

pyramid was forming in my mind.

At the top of this pyramid, there

are professional musicians.

These are the people who spend their time, energy, and money

on music and then make money back and are basically

in this position to do whatever they want to, if it goes well.

Then you have the hobbyists, who might spend all their energy,

time, and money on music, and they're not looking to receive

anything but their own satisfaction--

their own interest--

but they also could do whatever they

wanted to express themselves.

Then you have nonmusicians, who often

think, oh, I should have started playing

when I was much younger.

Now it's too late.

And that's obviously wrong.

Most of us can right now pick up an instrument secondhand,

borrow from a friend, we can go online, on YouTube,

or we can just get a teacher, and we

can learn to play in no time.

But there is a whole other layer of people with special needs

that might have all the time and energy and ambition and even

talent to make amazing music, but they're

hindered by their physical or cognitive condition.

And I know that last layer, by the way,

is not the biggest layer, but bear with me.

When we connect the two parts of the pyramid,

these two ends combined could create

some kind of transformation.

And trying to make that happen, I co-created an event called

Dis Co Tech--

which stands for Disability, Community, Technology.

And the idea behind it was creating the first framework

that was dedicated only to create music technology

for people with special needs--

creating the first framework for music technology

for people with special needs.

And so what we did was we had four different individuals

with four different challenges.

And around each of them, we had a team of makers, developers,

musicians, and therapists that were developing

a solution for that one person.

[PIANO MUSIC PLAYING]

[INAUDIBLE] suffers a neurological disorder,

and he used to play and compose and produce music.

But now he couldn't do that anymore,

so his team created an eye tracking

system that enabled him to compose music using his eyes.

[INAUDIBLE] was born autistic and blind,

and he could play really well with one hand--

but the other hand, he could only play with one finger.

So his team created a pedalboard that

enabled him to make up for the chords he

couldn't play with his left finger using his feet.

[PIANO MUSIC PLAYING]

[INAUDIBLE] was a professional musician,

and he was injured in a snowboarding accident

and ended up in a wheelchair.

So his team turned the wheelchair into a setup,

enabling him to plug in all of his instruments

in looping cells-- basically turning it into a one man band.

And last but not least, [INAUDIBLE]

is a professional singer that was born with one hand.

And so she could never really accompany her vocals

until her team designed and 3D printed a prosthetic arm that

enabled her to play the guitar for the first time in her life.

[MUSIC - "THE SCIENTIST"]

[APPLAUSE]

[PIANO MUSIC]

MATAN BERKOWITZ: So there are two main insights or lessons

learned from this experience.

The first one had to do with the pyramid.

So I realized while working on this that what we were actually

doing in a way was flipping the pyramid on its head.

Because in a way, when you're focusing

on the person that has the biggest challenge,

you actually kind of trickling down and helping everybody else

that is part of your pyramid.

In this example, we can take the eye tracking system.

So a person who cannot play music or cannot compose music

without the eye tracking system obviously needs it most.

But once you have this device that solves his condition,

you actually can help the professional musician explore

new ways to create, the hobbyist to have a new toy to play with,

and the nonmusician to have a new and intuitive way to try

and learn music and play music.

The second thing that I've learned

from doing this was that none of the things we created

was actually scalable.

All of these solutions were meant for a specific individual

with a specific condition and required

some kind of specific setup.

So in a way, it wasn't really going

to affect the lives of millions of people

like we had hoped for.

So this got me thinking about, how

can we really make that difference?

And I realized we are actually talking

about two different challenges that would require

two different solutions.

The first challenge is about music.

How can we make music accessible to as many people as possible,

regardless of their condition?

Regardless if they are healthier or disabled, if they're

young or old, if they're a professional musician or just

getting started, how can we make music

as accessible and intuitive as it could be?

And the answer to that question for me

was the Airstrument, which is the device on my hand.

The idea behind this wearable device

is that it takes that notion of voluntary signals of the thing

you can really control--

which is hand movements-- and turns it

into a super intuitive instrument.

So playing drums, for example, would look like this.

[DRUM SOUNDS]

Or you could just grab a frequency.

[FREQUENCY SOUNDS BEING MANIPULATED]

Or you can try and make a whole song.

[MUSIC PLAYING]

I'm going to loop myself and add layers to this.

[MUSIC DEVELOPS AND CONTINUES]

Now we could use a little bit of a chorus vocal type thing.

[VOCALS]

[DELICATE MELODY PLAYING]

VOCALS: Love will always win.

MATAN BERKOWITZ: [CHUCKLES]

[APPLAUSE]

Thank you.

The second question-- if the first question was really

about how to focus on music regardless of disabilities,

the second question became how to focus on disabilities

regardless of music.

How can we really make the world accessible for those

who need it--

for those who need this kind of accessibility--

regardless or beyond just playing music?

And the answer to that question became

this device called Tongo.

And the way it started was with a biomedical engineer called

[INAUDIBLE],, who came to meet me because she

wanted to create a musical instrument for people with ALS.

And the idea was to enable them to play music

using their tongue.

And I thought it was an amazing idea,

but being disillusioned with creating music technology

for people with special needs and nothing more,

I immediately told her, let's use the tongue

to enable them to do a lot of different things way

beyond music.

And this is what we've been doing ever since.

We've been working on this for about a year and a half.

Now I want to tell you a few things about the tongue

so you will get the idea.

First, I'd love you to just take the tongue right now

and touch--

from the inside of your teeth, just touch one tooth.

Now touch another one.

Now slide between them.

We all have this really incredible orientation

within our mouth using our tongue.

It's almost like we already have a keyboard in our mouth

that we're not using.

Secondly, the tongue is a very strong muscle.

It can keep going for hours without getting exhausted.

Unlike eye tracking technology or other things

that end up becoming very exhausting over time,

the tongue can just keep going.

And thirdly, it survives really severe conditions

like neurological disorders and spinal cord injuries.

And so we created these Smart Braces.

This is the prototype of these Smart Braces

that are already patent pending, and they're wirelessly

communicating with different devices

like smartphones, computers, wheelchairs, smart homes--

and they're very intuitive and accessible already,

making it kind of easy to use and learn for almost anybody.

We created different apps for this,

and I'm going to show you one of them very soon.

The thing about this project is that even though we

were doing this in stealth, we had the privilege

to get funded by the Israeli innovation authority

and to have partnerships with IBM and the Odessa Medical

Center.

We even won awards from the Prize4Life ALS Foundation

and MedTech Accelerator in Israel--

all of this before coming out of stealth, which

is what we're doing right now.

We're just right now coming out of stealth.

So it's very exciting for us.

[INAUDIBLE] is right here-- you can raise your arm.

[LAUGHS]

And so more than any of these partners

or more than any of these things,

the thing that touched us and helped us the most

was collaborating with individuals

who actually need this.

One of them you might recognize from earlier in this talk.

MATAN BERKOWITZ: The thing is, as we were working on this,

and we were working with people that actually needed it,

we realized the potential of this technology

to disrupt other industries-- like security, gaming, sports.

Just think about scuba diving, for example,

where you're already putting something in your mouth.

Why not make it a smart device?

And this notion is now leading us to first focus

on assistive technology and rehabilitation,

while we're exploring this new space of mouth

based controllers and what it could do.

And out of all of the apps that we've created so far,

I'd love to show you the musical one.

So this is Tongo.

This is the operating system we've built.

It has different apps, and they're all

controlled by the tongue.

[VARIOUS MUSICAL SOUNDS FROM TONGO]

TONGO: Hi, how are you?

I am hungry.

I'm tired.

[SAMPLE OF INIGO MONTOYA FROM THE PRINCESS BRIDE]

[LAUGHTER]

MATAN BERKOWITZ: Recognize this one?

I'm going to try and play a song with it.

[TONGO PLAYS THE RIFF FROM "EYE OF THE TIGER"]

[LAUGHS]

[APPLAUSE]

MATAN BERKOWITZ: Thank you, guys.

I'd like to finish this talk with kind of the main message

from me, which is not really about music, technology,

disabilities-- it's about something else.

I feel like the world right now is

at this point where we actually have a lot of responsibility.

Our generation-- the generation after us--

we're inheriting a world with a lot of challenges

and a lot of potential.

And we have to start figuring out

what we can each do to help.

And in that sense, what I have found

in my own journey is that there is a sweet spot between what

you love doing--

what you would do if no one would

pay you or ask you to do--

and between what you feel like the world needs--

the community around you, people you know,

or the world at large.

And where these two things meet--

where what you love doing and what the world

needs become one thing--

is that sweet spot where doors open

and things start happening.

And I encourage all of us to search for that,

and once we find it, to go with all

of what we have to offer and direct

our energy and our intention to that place.

Thank you for your time and attention.

[APPLAUSE]