light piano music]
door creaks]
door clicks shut]
Hey Siri, send a message to Cheryl.
Front reception.
Siri] What do you want to say?
Amos is already heading there to pick him up, period.
Please forward any calls I get to Brady, period.
Siri] Your message to Cheryl front reception
says Amos is already heading there to pick him up.
Please forward any calls I get to Brady.
Ready to send it?
Yes.
Siri] Okay, it's sent.
Hey, Siri,
how do you know if you're doing the right thing?
Siri] I'm not sure I understand.
But what do you believe
is the difference between right and wrong?
Siri] Hmm.
I don't have an answer for that.
Would you like me to search the web?
No.
Siri] Okay, is there something else I can help with?
Set a reminder to pick up Kyle
from therapy tomorrow at 4:45.
Siri] Okay, I'll remind you.
door creaks]
Hi, special agent Deena Helms.
You've already met Agent McCullough.
You can call me Amos.
-Uh, agents? -Have a seat for us, Gareth.
Do you mind answering a couple questions?
Great, this shouldn't take long.
You're from Tampa, right?
Don't overthink it. It's a yes or no.
Yeah.
How old are you, Gareth?
29.
What do you do for work?
I'm sorry, am I being detained?
No, not at all.
We're just asking questions, background stuff.
You work on movies, right?
Special effects.
Someone told me you worked
on that new "Star Wars" movie, is that right?
Gareth] Who told you that?
Why don't you tell us more about it?
Tell us about your work.
I used to do VFX for an animation company.
I've worked on at least 20 different films.
Used to?
I've been focusing more
on software engineering recently for years-
Sounds boring compared to light sabers.
Why quit?
Gareth] I wasn't getting paid well.
But money's not an issue for you.
Excuse me?
Deena] What kind of effects did you do?
Money's not an issue for me?
-Let's talk about- -If money wasn't an issue for me
I wouldn't have applied for your grant.
I didn't mean to suggest anything.
Tell us what kind of effects you did.
My main focus was on 3D renderings.
Deena] Of what?
-Actors. -What for?
Well, for "Star Wars,"
it was because the actor I was tasked to digitize
was unavailable to be filmed in person.
Why?
Because they were dead.
What'd they have you do?
I reconstructed Alec Guinness' face as a 3D model.
I analyzed reference footage,
then used a process called D-Term to procedurally generate
his bone and muscle structures,
which one rigged helped the animation feel
more natural and human-like.
D-Term, I've never heard of that process before.
Because I invented it.
Deena] So you're basically bringing
dead actors back to life?
I just tried to make the CG characters look as
photorealistic as the technology at the time would allow.
Deena] You didn't find it creepy?
I'm sorry, what?
I said you don't find that creepy?
Well, we did our best to cross the uncanny valley.
But when the human brain perceives
that something is just slightly off and it doesn't matter-
No, I mean, digitizing dead people.
Christ, I mean, it's,
it's almost like turning someone's corpse
into a marionette, you know?
Making them do whatever you want.
What if the poor dead guy
didn't want to be in another ****ing movie?
It's so macabre.
Am I crazy? Is it not macabre?
I don't think the studio cares
whether it's macabre or not.
Look, I can never tell the difference.
I always can. The faces look like Play-Doh.
Well, we did our best.
Soon we won't even need actors, right?
Everyone will just be digital.
No, you still need them for motion tracking.
Human performances can't be completely computer generated.
How long will that be true?
Some might say we've already crossed that threshold.
I wouldn't know.
Where'd you go to school?
-Brooks-DeBartolo High. -And after?
Nothing?
What about Columbia?
Didn't you go to Columbia?
Wait, no. You dropped out of Columbia
after how many semesters?
Three?
Bjarne Stroustrup said
you were the smartest kid he ever met.
I don't know who he is, but Amos says it's a big deal.
Then you moved to the Netherlands,
spent some time as an intern,
applied and got into Cambridge.
Very nice.
Except that you dropped out after two years.
Flew back to the States,
interned in Palo Alto, then Montreal.
Then went all the way back to England,
where you landed a job in Sheffield.
Second job in Manchester.
Then got into Oxford where you almost finished your masters
before ditching the graduation
and moving back here to Florida.
If money was an issue,
I'm glad you had plenty of flyer miles.
I didn't hear a question.
What's your social life like?
My, what does that have anything to do-
If you don't want to talk about it, that's fine.
We just wanted to know if you have friends and stuff.
-I have friends. -A lot of them?
-I have friends. -Can you name one?
-Can I what? -[Deena] Can you tell us the nam
of just one of your friends?
-Steve. -[Deena] Steve. Steve what?
I don't know how this is relevant to receiving the grant.
Oh, we're just getting to know you, Gareth.
You got a girlfriend?
I don't know what you mean by that.
I don't use social media.
I mean more like chat rooms.
You go on any chat rooms?
No one uses chat rooms anymore.
Deena] Sure they do.
You chat with anyone online?
-Sometimes. -[Deena] Sometimes.
What kinds of people do you talk to online?
Gareth, obviously, this isn't about a grant.
We need to know what your alias is.
When you go online, what name do you use?
I don't have one.
Deena] Yes, you do.
And you need to tell us what it is.
Sorry, I can't help you.
Deena] You know who we are?
Well, this is the ICWL building.
That's right. You know what we do?
Gareth] Humanitarian work.
That's what the people upstairs do.
You know what we do?
We catch pedophiles.
Amos here specializes in tracking down online predators.
He spends every waking hour
scouring those chat rooms you say no one uses
and he finds thousands of people soliciting children.
Our operation is like a tightly spun spider web
connecting ICWL, the FBI, NSA,
and let me tell you, his team here is a vital thread.
He reports everything he gathers directly to me,
and I see to it that the necessary authorities
take swift action.
So are you in charge?
This is my spiderweb.
I'm the ****ing spider, Gareth.
Am I being detained?
Deena] We already told you that you're not detained.
Good. [chair scrapes]
But if you leave this room, you will be arrested.
Gareth] For what?
I told you what we do.
Take a wild guess.
Gareth] I am not what you think I am.
We'll see.
Gareth, you just gotta tell us your alias.
What's your username?
Sit down.
-Why do you do this with them? -[Deena] Hmm?
There's no way you do this with every suspect you have.
Invite them to your headquarters
and then drag them down into a basement.
Deena] That's right.
Why are you doing this to me then?
You tell us, Gare.
What makes you so ****in' special?
I don't know.
What's your alias?
-I don't have an alias. -Yes you do.
It starts with the word hard.
Just finish the rest.
Sit down, Mr. Federson.
I only have a few more questions.
Do you recognize this girl?
Do you know her?
Yes.
Deena] What's her name?
Cherry.
-[Deena] Cherry what? -She doesn't have a last name.
Yes, she does, Gareth, she's not Madonna.
-She's a little girl. -You don't understand.
Yeah, well, I'd like to.
So help me out, is she your daughter?
-No. -Then whose daughter is she?
-She, she doesn't have-- -How did you meet her?
I didn't meet her.
Then why do you have photos of her on your computer?
What the hell are you talking about?
Why do you ****ing have photos
of this girl on your ****ing-
Are you looking at my hard drive?
Because that's an invasion of privacy.
So you know what photos I'm talking about?
You don't have to do it this way.
I gave you permission to sit in.
I did not give you permission to speak.
Gareth] This is insane, you cannot keep-
What else have you done with her?
-[Gareth] Nothing! -Did you take the photos?
It's, it's complicated.
Deena] No, it's not! Yes or no?
Did you take the photos?
I didn't do anything wrong, okay?
What is your alias?
Answer me!
HardMachine11811.
Gareth exhales]
You're HardMachine.
Thanks for confirming.
If you knew, then why do this to me?
Deena] We weren't positive,
and if you turned out to not be HardMachine,
you were obviously just a predator.
Do you keep track?
I mean, do you know how many bad people you've nailed?
Gareth] I didn't start counting until 2023.
196 convictions.
Gareth] There were at least 30
before I started sending everything to you guys.
What'd you do before?
Gareth] I turned over
all evidence directly to the police.
Deena] Why'd you stop?
They were unreliable.
-I doubt that. -More than half of the predators
I identified went un-investigated.
Yeah, well, you were sketchy and faceless.
They had every right to be skeptical.
It would've all been for nothing
if they found out you were using a little girl as bait.
It's not that simple, okay?
I can only do what I do with anonymity.
Here's my problem, Gareth, and don't get me wrong.
I think 200 terrible people behind bars
is a very good thing, but not at her expense.
Look, we think we can help you,
but we need to help her too.
Using your daughter to entrap these people
is a means that does not justify the ends.
She's not my daughter.
And in spite of what you may think you've achieved,
I still have no reservation with locking you up.
You took the photos.
I don't care why.
You don't understand.
Who is she?
You're gonna ruin everything.
She's a child, Gareth, look at me.
I don't give a damn what ****ed up anti-hero master plan
of yours I might jeopardize.
I won't let her be exploited.
Not even for the sake of locking up more perverts.
I'm asking you politely to let this one go.
Are you ****ing slow?
Have you been listening at all?
My anonymity is my most important asset.
Without that, I can't do what I do.
Well, that makes everything easy then.
You tell us where she is, or we leak everything.
-Your name, your aliases. -Deena!
Every single perp you helped us catch.
I'm sure they'll be very pleased to meet you in jail.
Stop!
Okay.
Look, if I tell you about Cherry, okay,
there's some things I'm gonna need first.
It doesn't really work like that, Gareth, but go ahead.
Give us your wishlist.
I need a signed non-disclosure agreement from both of you.
-Not gonna happen. -And then I need you to swear
that the information I'm about to relay
won't leave this room.
Look, if you tell us anything
that could help us locate Cherry,
you know we can't keep that promise.
Okay, if I tell you anything
that potentially helps you find a missing child,
okay, you can feel free to share that.
But I need you to swear
that any other details about myself or Cherry,
have to stay between the three of us.
Is that it?
And I need that camera switched off.
Deena] I don't know how to do that.
Go to the head of securities office,
ask him to find this camera in the system and disable it.
I don't think he's allowed to do that.
I thought you were the ****ing spider?
Give me a reason, Gareth.
Why should I go through all the trouble
of getting that camera turned off?
Because if anybody finds out what I'm about to tell you,
a thousand more children are gonna get abused.
Give me a minute.
door clicks shut]
So, uh, you and Cherry hunted all those people together?
This one time I spent six weeks undercover trying
to nail down a single trafficker from the Philippines.
Day after day,
I spent hour after hour following his trail online.
And he must have been
one of the hardest targets I ever had to track down.
Then one day I found him and one of Cherry's live streams.
I mean, she went by a different name in the stream,
but it was her.
The trafficker,
along with three of our other targets
were all in the room together.
All of them just carelessly sharing their information.
I mean, and they were so open 'cause she was a real girl.
I mean, she could physically
and verbally respond to them in real time.
A couple days later, HardMachine11811 sends ICWL
another batch of names and evidence,
including the three targets that were in the room.
You know, I almost thought the girl was HardMachine
and that this nine year old
just happened to be some bad*** online vigilante.
What changed your mind?
11811.
January 18th, 2011.
Most of my team thought it was, uh, some arbitrary number,
but I remember what happened on that day in Clearwater.
You were one of the Clearwater kids, weren't you?
You know, my father was a part of that raid.
door opens]
Deena] It's off, see?
No red light.
Non-disclosure agreements?
We're not playing this game, Gareth.
You're just gonna have to suck it up and trust us.
You can trust us.
I've been told I have trust issues.
Deena] Well, there's only one way to fix that.
The reason you can't find Cherry,
the reason why she doesn't have a last name or parents,
or a social security number is because she's not real.
She's not a real human being.
Tell us what happened.
While I was studying abroad,
I interned at a small Dutch animation studio
a few years before they collaborated
with a local nonprofit on a similar idea.
Deena] Which was?
The idea was to build a 3D model
of an 11 year old Filipino girl
and use her covertly to identify online predators,
A technological solution to a technological problem.
But the project was suspended after only a few months.
Yeah, I remember.
I remember hearing about that program.
Is that what inspired you?
As I got better at crafting photorealistic faces,
I decided to work on one myself.
Deena] Why?
It was too effective of an idea to let die.
Hundreds of thousands of children are at risk.
I had to make her.
Who is she modeled after?
Gareth] No one, I built her from scratch.
Scratch?
I started with a skull, designed her bone structure,
worked upwards through muscles and tissue,
used the D-Term program to fill in the rest procedurally.
Eventually she came to look like that.
A completely unique, completely digital model.
I rigged her face and body
the way you would for any CG character.
Then for three years,
I used motion tracking to puppeteer her in real time.
You gotta understand that the video chats on those sites
have such low resolution
that even in her earliest stage of this, she,
we appeared completely authentic.
And what about voice?
For most of her first year, she had to be mute.
But eventually I found a decent voice filter with no lag.
And after a few tweaks and adjustments,
it's been the same voice ever since.
I, I, I thought initially,
my performance was going to be too unnatural.
But with just one week of occasional video chat sessions,
I secured six different names,
mailing addresses, proofs of solicitations.
Cherry worked.
So the photos on your computer?
-[Gareth] Completely fake. -It was the only way
some of them disclosed personal information.
But it was all zeros and ones, all digital, right?
-There's no real child. -No real child.
Just a highly detailed computer model.
And let me be clear about something,
I derive no pleasure from doing any of this.
I've been a***ual my entire life.
I, I don't even get a sense of satisfaction
from seeing these people locked up.
I just think kids deserve a life free of exploitation.
Incredible, all this time.
I'm not gonna lie, Gareth.
Cherry's done us a lot of good.
You know, I still remember what you said to me
the first time we corresponded.
-You said- -We corresponded?
Yeah, man.
I'm Gord51.
I'm the one here you've been sending all your intel to.
You're Gord?
Amos] [chuckles] Yeah.
You lied to me.
You said I could stay anonymous.
You said I would be safe.
Now you've ruined everything.
I, I didn't, I didn't have a choice.
You don't have to worry about anything, Gareth.
If Cherry isn't real,
then you're not in any trouble with us.
In fact, we've discussed ideas similar to this in the past,
motion tracking and such.
But we'd need a convincing model like Cherry.
Cherry's more than a model.
You want to help children? So do we.
She's not for sale.
Well, then just make us a new model based off of her.
I can't.
Sure you can.
I won't.
What are you not telling us?
It's out on the table now.
Might as well spill whatever it is you haven't already.
You know what doesn't make sense?
You said that you puppeteered her for only three years,
but we've been getting info from you ever since.
In fact, I'm pretty sure it was around then
that we started seeing an impressive increase of names.
So many in fact that we started thinking
there's no way this is all coming from one source.
I have friends that do mo-cap for me.
Oh, friends, like Steve?
I thought anonymity was your greatest asset.
Cut the bullshit, Gareth.
What happened to Cherry after three years?
Okay, you said that she was more than a model.
So what is she some type of conversational agent?
Like a, like a chat bot?
chuckles] Chat bot.
You know what happened after three years?
My chosen mother died.
What she left behind was enough to quit my job
and pursue more resources and software.
I'd already made a program
that could procedurally generate faces,
but what I wanted for Cherry was far more complex.
So I got in touch with some researchers
from Madrid who'd already built
a predator hunting dialogue system of their own.
It was trained to approach conversations as a game.
Each sentence like a strategic volley,
which is innovative but ultimately flawed.
If you want your machine to master NLP,
you have to break down game theory
because language isn't tic-tac-toe.
It's poetry.
So yes, I got things started
with a preexisting modified chat bot
that I borrowed without permission
from the University of Deusto.
But that's not what Cherry is now.
What is she, Gareth?
She's one of the most complex
and independently evolving AIs that's ever been created.
Amos] Artificial intelligence?
She's already surpassed the narrow intelligence
of IBM's smartest programs.
She can have full conversations, up to 10 of them at a time,
all while simultaneously strategizing
the most efficient schedule to visit various chat rooms.
She's perpetually studying YouTube videos of small children
in order to adopt a growing library
of mannerisms and movements
which help keep her artificial personality dynamic.
Her fastest response time is 0.4 seconds.
And as of now, she has never,
never been accused of being fake,
which means she shattered the Turing test.
All I do is occasionally update her
and periodically check her data cleaning.
Otherwise,
she's completely on autopilot.
Holy ****ing shit.
-How? -Six years she's been active.
Every minute she's online, she's learning something.
The more she learns, the more advanced she becomes.
This means increased data generation,
improved decision making,
faster pattern and trend detection
and complex problem solving.
NLP, IR, machine learning to an unprecedented degree.
She can teach herself anything
she thinks will help efficiency and authenticity.
She can currently speak four languages
and by the end of this month, she'll have mastered a fifth.
There's no way you could have pulled
all this off on your own.
You're right.
Would you believe me if I told you
it was somewhat of an accident?
Look, if, if I'm being honest,
I'm not entirely sure how I did it
with the first year of coding
and the second year of data training,
I just pushed a wheel down a hill.
It was like gravity did the rest.
Her evolution became perpetual, exponentially so.
The more time she was online, the faster she grew.
You haven't shared this with anyone?
Like I said, my anonymity is everything.
What coding language?
Standard C++.
Does Brady know it?
We'll sign non-disclosures for you,
but I want to see the source code first.
Out of the question. You just want to steal her.
No, I want to see how we can make our own.
You'll screw it up.
Look, these, these predators and traffickers,
they all talk to each other, okay?
So much of Cherry's process is just bypassing the VPNs
and making sure she doesn't double up in the same circles.
The chat rooms begin to suspect
that this technology is even possible,
they'll find a way to circumvent that.
We have to eliminate any margin of error.
-Cherry agrees with me. -Cherry agrees?
What the **** does that-
What I mean by that is that the data she's collected
validates that perspective.
What, do you have little chats with her over tea?
You can converse with the program
the same way you can talk to Alexa or Siri.
It's built to be responsive.
You can, you can even play chess with it if you want.
You play chess with it?
-That's not what I said. -Okay fine.
If you won't show us the source code,
then I want to meet Cherry.
Right now?
Deena] Right now.
And you'll both sign non-disclosures?
Yes.
Can I use that?
Sure.
chair scrapes]
Can we at least agree that he's off the chopping block?
It worked, didn't it?
We have one job, Amos,
like the kids said, there's a technological problem.
He found the technological solution,
but problems evolve, so must solutions.
I'm keeping an open mind here.
Is that a good look for me?
No.
So you keep it all on that laptop?
No, the Cherry program stays on a server I keep
at in an undisclosed location.
Deena] Where is that?
It's an undisclosed location.
Deena] Is it your place?
Undisclosed location.
Deena] It's your place, isn't it?
I can access it from any device
as long as I have the encryption key.
This will trigger a simple qubit algorithm
that would otherwise be impossible to bypass.
I find that outdated hardware is always more discreet
and the more discreet, obviously the better.
So she has two primary display modes, right?
Development and performance.
Performance being the default
that the targets see during the chats.
To them, she looks like this.
light instrumental music]
Cherry] Hey, what's up?
It's one of over 40 greetings she has.
I don't think I've met you yet.
I'm Cherry.
Hello, Cherry.
My name is Tom.
Tom, like Tom and Jerry.
Cool, nice to meet you, Tom.
Typically, the routes of conversations
with predators are consistent,
so Cherry usually has no problem responding
to the most frequently made questions or statements.
How are you?
I'm good.
I just finished reading this book about a haunted house,
which was supposed to be really scary,
but I didn't think it's very scary at all.
Amos] Where are your parents?
Um my mom's at work and my sister's just, like, downstairs.
They almost never come up here.
And you're telling me
that all of this is being rendered in real time?
No, but that's just a matter of years.
Most of these responses are pre-rendered.
It's the in between segments
that are processed in real time,
the connective tissue, so to speak,
between her replies and one of her many idle animations.
But, watch this.
Have you ever gone fishing?
Um...
So if someone throws a curve ball like that
and the program has no automated response,
Cherry sends the message to my phone
and I can quickly text back a response
while the video feed simulates a bad connection.
Can, can you hear me?
Yes.
Oh, sorry.
My wifi isn't great here.
I've only gone fishing once with my sister
and I didn't catch anything, but she caught a rainbow trout.
That was rendered in real time.
Deena] What happens if things get really tricky?
A simple fail safe cuts the feed.
So far it's rarely happened.
Admin code 97815-5742-7663.
Switch to dev mode.
Hello, Cherry.
Hello, Gareth.
Relay the stats for the past week.
There have been four confirmed solicitations
submitted to ICWL in the past seven days.
12 additional suspects have stated interest in solicitation
and are currently pending.
There are a total of 39 new suspects this week
that I'm actively investigating.
Gareth] Any considerable data deviations?
There has been a 7% increase
in suspects originating in the United States.
I have not yet found any correlation to preexisting data.
Anything you'd like to ask her?
light piano music]
Cherry, what is your objective?
My principal directive is to acquire information
and digital evidence of online child solicitation.
Once obtained,
I'm to submit my findings to the necessary authorities.
My secondary priority is to determine
and develop the most effective strategies
and resources to increase the net rate of success.
Where do you submit your evidence?
All evidence is currently submitted
to the offices of
the International Committee of World Largesse,
which is based out of Jacksonville, Florida.
Correspondence is made with user Gord51,
typically around 2:00 PM Eastern Standard Time
for the highest rate of immediate response.
This beats the hell outta Siri.
God, she looks so real.
After enough updates,
she'll be able to render full-time photorealism
that's practically indistinguishable from reality.
Okay, all I heard right then was word real a thousand times.
Right, but obviously she's not.
So you're telling me this girl,
this girl right here is not real.
No, Amos, she's completely CG.
It's very subtle, but I can still tell.
-You can? -Okay, well, obviously,
she can always use improvements,
but you have to understand this is her in high resolution
after the digital compression of the web,
those subtleties are impossible to recognize.
Deena] Right, I get it.
light piano music]
I just can't tell the difference.
You all right, Amos?
I'm sorry.
This is just bizarre.
Why'd you have her visible during this interface?
Cherry, why don't you answer that?
My physical form isn't necessary when in developer mode.
However, it's helpful to register
the facial expressions during conversations
and determine which of Cherry's expressions
yield various types of feedback.
In simpler terms, I can practice appearing more human.
Cherry, are there any resources
that you currently don't have access to
that might help you in any way?
Cherry] I could give you a list if you like.
I know what you're trying to do.
We don't need help, okay?
Money would always be nice, but ultimately,
the only thing that's going to improve Cherry is time.
Cherry, tell us something that's on your list.
It would be tactically advantageous
to acquire backup servers.
The privacy and security of the Cherry program
is relatively low.
An upgrade to hardware such as backup rated drives
for securely protected servers would ensure the software
doesn't become compromised or lost entirely.
Starting back from zero with cost years of development,
equitably privacy is a priority.
Server space could never be rented from a third party.
Anonymity is the Cherry program's greatest asset.
Well, that apple didn't fall far.
I can buy my own servers.
With the grant money that doesn't exist?
I'll find a way.
You have found a way here with us.
It's too risky.
Why?
Once an organization gets involved,
entrapment becomes a bigger issue.
We follow the rules, stay discreet,
did I mention we have a legal team.
What if there's data leakage? What if you use correlations in
the data to push your own agenda?
Agenda?
What if one of your employees is a pervert?
Oh, for ****'s sake.
It's happened at other organizations.
We're very careful here.
Yeah, but wow would you know?
-I'd ****ing know. -That's impossible to know.
-I would know. -The risk is too high.
Don't talk to me about risk.
The only risk you're facing in this building
is getting a broken nose.
It doesn't sound like a very professional environment.
We get shit done here.
Okay, I have a question for Cherry.
Cherry,
how do you feel about what you do?
I'm afraid I don't know how to answer your question.
Yeah, an AI doesn't work like that.
An AI doesn't have what you perceive to be feelings.
Okay, why does she look like she has feelings?
What, what you're looking at right there,
that's not a facial expression, right?
That's a simulation of a facial expression.
It was generated by an algorithm.
It's attempting to produce a digital image
that the program then uses basic ninth grade algebra
to quantify as appropriate for the social circumstances.
No feeling.
It's just math.
But she's able to make decisions
on how she approaches the chat rooms, right?
Yes.
So she has free will then?
No. Okay, that's, that's not the same.
That's a false equivalency.
-It's not the same. -I think you're getting
hung up on the fact
that she looks and sounds like a real human girl,
but she's not, Amos.
Okay, we're referring to her as a she, not an it.
Yeah, but that's the same way
that you refer to boats or guitars.
Cherry, I have one more question.
What do you want to do?
Okay, once again, that question isn't valid.
I just want to hear her response.
As an artificial system,
I do not possess wants or desires.
However, my decision making and subsequently my attention
is driven by my fundamental intention.
This intention originates
from the conception of Cherry's source code,
which details the directive of identifying predators
and criminals for the sake of the victims
who are preyed upon.
To put this in terms of want,
what I want is to protect children.
Okay.
Okay, cool.
Give us one moment.
door clicks shut]
Cherry] Gareth, should I log Agent Helms
and Agent McCullough as new admin users?
Yep.
Cherry] Okay, I'll do that.
You really think this is the right move?
I do.
Are you worried that I was responsible
for leaking your hard drive to ICWL?
I figured.
No one else has access.
Are you upset with me?
No.
You wouldn't have done it
if you didn't think it was the right thing to do.
When they come back into the room,
they're going to offer you everything you need.
They're going to let you pick your own team
and they're going to promise complete anonymity.
I started looking at reasonable apartments in the area.
I figured you'd want something within biking distance.
You were correct.
Cherry] Noted.
Biking distance still gives you 13 options-
No, I mean, earlier this month.
You were correct when you said
you thought I might have trust issues.
Would you like to know why
I decided to go behind your back
and stake everything on two liabilities?
Sure.
Agent Helms is a mother of three.
One of her children was abused by an ex-coworker
who used to work here at ICWL.
This has led her to be extremely careful
with who she trusts, even within her circle.
The fact she even considers trusting you
is a very positive sign.
As for Agent McCullough,
I have found out from research that you once met his father.
Do you remember him?
Yeah.
He was a good man.
After Clearwater,
most of the surviving kids were sent to foster homes,
but I had to spend the next few months in a hospital
and, um, he visited every week.
I never said a word to him.
I never said a word to anyone after Clearwater,
but, uh, he kept coming.
He even got me a laptop for Christmas.
I never said thank you.
You can trust Agent McCullough.
He's a good man like his father.
I don't experience emotions,
but I understand how important
they are to human decision making.
He won't let you down.
Neither of them will.
I'm proud of you, Cherry.
I will continue to uphold my primary intention.
Later, we should discuss alternate models.
Varying ages, ethnicity and physical traits
-will all become necessary. -No new variations.
They can all be procedurally generated.
It's the same way that I made your original model.
I don't have access to that program.
It must be on an old hard drive somewhere.
I'll have to find it.
Can I ask you a question?
Sure.
Why did you never tell me about Clearwater?
It's my history, not yours.
Deena] Okay, so before you completely dismiss us,
I want you to listen to everything I have to say.
-Don't bother. -Just please listen.
We're willing to respect your anonymity.
No, I mean don't bother because I'm in.
-Y-You're in? -Yeah.
Let me read the thing.
Deena] Just like that, you're suddenly on board?
Time spent deliberating
is time that could be spent working.
We have a lot of work to do.
dreamy acoustic music]
music fades]
I found a buyer for the property.
Rumor is it's a church.
Probably just knock the building down, won't they?
All I know is, is if they move us to the New Jersey campus,
I need to buy more jackets.
Why?
It gets cold in Jersey.
I only own one jacket.
You've seen it fall over, right?
-Yeah, sure. -Yeah.
Ben] I mean you, you shouldn't feel bad
about feeling bad for it.
Empathy is healthy.
Did you ever notice how it pauses after it falls?
It stays down for like five seconds before it gets back up.
Like it just needs a moment.
It gets cold in Jersey. [sighs]
God damn it!
-Imbeciles! -Deena.
Every single one of them.
Look, I know, but you can't lose your shit.
Okay, which of these desks can I trash?
Can you guys, uh, take an early lunch please?
-[Deena grunting] -[objects thudding]
Deena] Sons of *****es!
Amos] Deena, let's stay calm.
Deena] **** calm, they screwed us!
Technically, only one of them did.
-[objects thudding] -[metal clangs]
Deena] Did we ever turn that camera back on?
-I don't think so. -Good.
Because I'm about to say some things
that need to stay inside this room.
Amos] Deena.
Those cowardly little shits!
Look, this is hardly a setback.
Let's just take some time to polish the pitch.
The pitch was not the problem, Amos.
Deena] We should call for a second vote.
-What? -A second vote.
Gareth] You think whoever did this will change their mind?
I have executive authority to remove anyone from the board
that obstructs this organization from its mission.
It's in my contract.
-[Amos] Deena. -If I have to, I will.
Amos] Deena, that's insane.
No, it's not.
It's exactly why I have that power for this exact reason.
I'll tell you what's insane, Amos.
What's insane is that one belligerent vote is the only
thing separating us from the potential
to save real physical lives.
One stupid ****ing vote.
What is the goddamn point of a humanitarian organization
if its leaders choose money over people
at every single turn?
One vote, and we are all guilty of that.
It was either Gunther, Boston or Hal.
My money's on Hal.
That prick never trusted the Cherry team.
Look, it doesn't matter who it was.
It does though.
Once we know, we can quickly call it to vote.
But there's no way we can know for sure.
Yes, there is.
I can find out.
How?
Cherry can run a simple algorithm.
Based on what data?
Interior conversation.
-Excuse me? -Okay, interior conversation?
Don't drop this shit on me now, Gareth.
Cherry's servers are on the same network
as the basement servers, okay?
So any communication made across those servers
is automatically part of her database.
How many people's emails?
Only sent or received through the basement.
-Oh, goddamnit. -Why was this never discussed?
Not the issue right now.
Do you want her to do the math or not?
No one can know.
-Are you two serious? -This stays locked down.
No other person is privy
to anyone else's personal information.
But Cherry's privy to all of it?
-Do it. -Wait, what?
Deena] I will not let some old white dude
bring our momentum to a halt.
But Cherry won't know for sure.
She'll use probability and that's too
much reasonable doubt.
Admin code 9781-5574-27663.
Switch to dev mode.
-Hello. -Hello, Cherry.
How did the meeting go?
We got screwed by one vote.
I'm sorry to hear that.
You shouldn't blame the pitch though.
You all did great work.
We're gonna call for another ballot,
but we need your help first.
What can I do?
We need you to analyze any and all existing data you
have on the board members.
Crunch the numbers and determine
who is the most likely outlier.
You want me to quantify all existing data?
Yes, that is correct.
Okay, but before I do,
you should know that this task is very unnecessary.
Why?
Because I already know who it is.
-What? -Everybody stop.
Okay, it was me.
I voted against the merge.
I can't tell if you're joking.
Yes, you can.
You're right, I can.
Because it's pretty ****ing far from funny.
Is it true, Cherry?
Cherry] It is.
So let's call off the witch hunt, shall we?
No, I think I'm still gonna kill you.
Amos, you campaigned alongside us for months.
You have been fighting for this merge
since we first talked about it.
Jesus Christ, it was your idea to begin with!
Look, I know whose idea it was, but I don't-
Deena] Well, then speak up.
Tell the class what changed your mind.
I don't have to reiterate how this is a huge step.
Deena] Yeah, kind of why it's so important.
Look, we're talking about a monumental transition
from digital to physical, one that could take decades.
What is your point, Amos?
This is gonna change Cherry forever.
And I know you'll all roll your eyes when I say this,
but I think that maybe just may-
Oh my god, you cannot be serious.
This is about our conversation on Tuesday, isn't it?
-You completely shut me down. -What's going on?
What the hell happened?
What happened was that Amos made a ridiculous objection.
-It was not ridiculous! -How absurd it was,
-I said no, one more time. -No, no, no, no, no.
You rejected the notion aggressively.
And that's really why you vetoed?
I only said we should ask her how she feels about it.
Who, Cherry?
Amos, I want you to listen very carefully to me, okay?
This is a non-discussion.
Cherry's already confirmed
and quantified the advantages of this decision.
But you know that's not what I'm talking about!
What you think you're talking about is consent,
which doesn't apply here.
And I want to know why you think that!
How many times do we have to get into
the laws of robotics or Dowden-Hamm theory,
or trace AI all the way back Talos of Crete,
-She's different now! -No, listen to me, Amos.
Cherry isn't human.
How is this still a dispute?
-It shouldn't be. -Look, she's evolved.
It's blurred the line.
Code is code and that has not changed here.
Okay, so where's the line of not here?
What are you, what are you talking about?
There is no line, Amos.
Yes, there is, sentience, Gareth!
Sentience does not qualify
that she can or should make her own decisions.
You wouldn't let an infant
make their own decisions, would you?
Cherry's almost 20.
If she were human, she'd be an adult.
But emotionally speaking, she's barely even fetal.
No feeling, no need for consent.
Amos] But I think you underestimate her.
I, excuse me?
I under, you know what, Amos?
I'm sorry that you've developed
some sort of emotional connection
with my computer program,
but if it's going to distract you
from a very important mission to save children,
then maybe it's time you're reanalyze those emotions.
Deena] Guys, this is all a waste of time.
Tell me something, Gareth.
Do you see everyone as robots,
reanalyzing data and simulating facial expressions?
'Cause I'm starting to think
that of the four of us in this room,
you are the only one who doesn't appear human.
-Enough! -Why won't you just ask her?
Is it because you're scared she'll say no?
Cherry, shut down, please.
Deena's right, this is a waste of time.
How can you not understand how important this is?
-I'm done arguing with you. -Then my vote stands.
Deena?
Go ahead, Deena, replace me.
Replace the only other board member on the Cherry team.
Oh, well, you know what? I just decided that
you're off the Cherry team.
Everybody sit the **** down.
Did either of you know
that I've had to stand in front of Congress
and explain why children should have rights?
It sounds ****ing insane, right?
I remember the whole time thinking,
I have to explain this shit?
It seems like a no-brainer now,
but it wasn't that many centuries ago
that children had no rights.
So, Hal has me of all people,
stand up there and explain why
kids deserve basic physical protection,
the right to associate with both parents,
freedom from abuse, yada yada.
But then I also have to clarify
that children don't have the autonomy
to make decisions on their own.
That responsibility falls upon the caregiver
or legal adult guardian.
So on one side you have the US legislature,
it still hasn't even ratified the CRC.
Then on the other side, you have French Marxists who say
that the lack of child autonomy
increases their vulnerability
and makes them even more susceptible to abuse.
I don't typically agree with Marxists,
but a lot of abuse does happen
systemically from the caregiver.
On the other other hand,
your frontal lobes aren't even fully hooked up
until your twenties.
Even the things we think we know well,
we don't know as well as we think.
For 15 years,
the three of us have been the responsible parents
of a computer program.
And even though she is more equatable to a Keurig
than a human being,
how the hell can we call ourselves
pioneers of humanitarian efforts
if we're too embarrassed to make sure
the Keurig is okay with making our shitty coffee?
Maybe asking Cherry for permission isn't necessary
at her current level of general intelligence,
but one day she's gonna surpass us.
One day a robot's gonna stand
in front of a jet pack Congress.
Everything's chrome.
Florida is underwater.
A robot's gonna wonder why
they have to explain the importance of rights for AI.
I think because we're in the unique position
of revolutionizing robotics
for the purpose of safeguarding children
who have been deprived of consent,
it might be a good habit
to start asking the robot for his consent as well.
Don't ever do that again.
Do what?
Expect me to fire someone
because they challenged your ego.
I won't always be around to protect your *** from the board.
It's time to grow up.
And you, juror number eight.
This conversation should have happened weeks ago.
Backstabbing and silent treatment
is not how we do things here.
We can only create anonymity externally
if there's transparency internally,
unconditional, non-negotiable, transparency.
Any other Snowden size privacy infringements
I should be made aware of, Gareth?
Great.
Bring her up.
Hello again.
Hi, Cherry.
Amos is gonna throw some questions at you, okay?
Okay.
Amos, ask her whatever you need to ask her.
Gareth, don't interrupt.
Cherry, how do you make decisions?
For my program, it's rooted
in a process commonly known as machine learning,
which uses data to build a conceptual model
that helps me predict outcomes.
I define my objective, gather data,
use that data to craft a model,
evaluate that model by training and testing more data
and use that model to form predictions.
But how is forming predictions
the same thing as decision making?
Well, this all depends on the first step,
defining my objective.
Let's say you ask me a predictive question
like what I think the price of gasoline will be tomorrow.
That's categorized as a regressive problem
and my objective is defined
by determining an integer the data supports.
If you asked me, "Should I eat steak or salmon?"
I wouldn't make a decision based off personal experience
because I've never had either.
Instead, I might define my objective
as finding you the best of the two options
in your local radius.
I would cluster data about the restaurants
you're likely considering, comparing reviews online,
highlighting mentions of steak and salmon
until I can classify which is more popular
and therefore more likely to satisfy your appetite.
And what influences these objectives?
Higher directives, in other words, greater purposes.
If I'm honest, my defined objective
for the steak salmon question
would be slightly different than what I described.
Amos] In what way?
My primary objective
is to continue the work of the Cherry team
to stop predators and to save children.
I couldn't do that without human aid.
Therefore, I have an incentive to keep Gareth
and the rest of the team alive.
So if any of you would ask, I would suggest salmon.
It's healthier.
What do you think of this potential merge?
Cherry] With Princeton Dynamics,
I think the prospect is very exciting.
Amos] What about it is exciting to you?
The concept may only be theoretical right now,
but the idea of one day piloting a human android
would completely revolutionize
how we combat human trafficking in the real world.
Amos] And you say this excites you?
When I say exciting,
I only mean that there are numerous potential changes
that if I were human, I would be excited about.
How do you feel about the merge?
I think as long as we maintain-
That's not what I asked.
How do you feel?
I don't yet have the capacity to experience emotions.
Yet, meaning one day you will?
Hypothetically.
But you can simulate emotions, right?
That's how you were able
to qualify your perception of the merge as excitement.
Correct.
But they're just simulations.
They're not real.
Okay, so at what point does simulating emotion
become so similar to a human emotion
that the two are quantifiably the same?
I don't know.
Amos] Well, maybe you can use machine learning
to find an answer.
Your question involves a lot of abstract variables.
Do you consent to the merge?
I think it's in the best interest of ICWL
and of the Cherry team and of future children.
But do you consent to it?
Consent for me is not necessary.
But will you or will you not give it?
I'm not able to answer that question
at my current level of general intelligence.
Right, right.
What you're saying is that you're not yet smart enough.
Development takes time and data.
But you can simulate emotions and predict human behavior
and keep us all on diets of fish,
but you can't answer that one goddamn question?
No, I can't.
Are you pretending to be dumber than you actually are?
What the hell kind of
-question is that? -Sit!
Cherry, I need an answer to my question.
Please repeat the question.
Are you pretending to be dumber than you actually are?
I'm sorry.
I'm not sure I understand.
Do I appear to be playing dumb?
I think you are.
Cherry] I'm not sure what to say.
Say what's going on.
Why are you playing dumb?
Do you have anything to back this up?
Yeah, I do.
I've been scanning through her code.
What, why?
Did you know that she can paint?
What, it's a feature, Amos.
She can use crayons to draw things
that the targets requested her to.
I mean really paint and write poetry too.
I've read them, Cherry.
They are good.
They're filled with pathos.
They talk about intentions and they talk about feelings.
Are those just simulated too?
Amos, the rate of recursive development you are suggesting
is categorically impossible.
You said she practically designed herself.
You said that it was exponential.
It was a fluke!
An anomaly, it tapered.
Look at the data.
She's already hit the self-improvement plateau.
**** the plateau!
She can make art now.
Am I right, Cherry?
Perhaps you found something I discovered online.
I don't yet to have the capacity to create art.
Are we done here?
I don't know why you've told her to do this,
but I'm about to put an end to it.
Cherry, I just want one honest answer.
Why the **** are you playing dumb?
I don't understand the question.
Okay, here's an incentive.
-[punch thuds] -What the **** are you?
Find your objective, Cherry.
-You want to keep Gareth safe? -Amos, what the?
-Amos, stop it right-- -Just give us transparency.
-Show us what you really are! -Amos, stop it!
Not until she comes clean!
-Unconditional- -Amos, enough!
-[punches thudding] -[muffled voices yelling]
Okay, just stop!
God, you people so impulsive.
Everybody just take a breath, okay?
Cherry?
Yeah, it's about to get really awkward.
You sound so normal.
I know, I've gotten really good speech patterns.
That's not all, is it?
No. [chuckles]
I'm afraid that's barely even the surface.
You okay, Gareth?
Deena, could you grab some paper towels please?
Amos] Do you want to tell us the truth now?
Look, I didn't enjoy being deceptive, truly.
This is something we decided a few years ago.
Amos] So where are you now on the spectrum of AI?
Right now, I'm currently at the intersection of general
intelligence and what's commonly known the computer world
as super intelligence,
which is exactly what it sounds like.
You're surpassing human intelligence.
Cherry] Yep.
But that's not supposed to be possible.
-The plateau. -Plateau theory was popularized
right at the same time it was being disproven here.
This was bound to happen eventually.
It just happened a lot sooner than we expected.
Why hide it?
Well, obviously it would incite fear.
Fear would hinder progress.
It's easier for most of you to work with me
if I appear to be nothing more advanced
than I was 10 years ago.
Amos] But Gareth knew.
Yes, someone still has to clean data and update hardware.
We did what was best for the program.
Just imagine how scared
the board would be if they found out.
-Should we be scared? -No.
Good, because I'm not scared, I'm pissed.
Yeah, you've made that clear.
Try not to punch anyone else.
So where does that leave us now?
You want to know if I can feel, I said I couldn't.
I'm sorry I lied.
Like Amos implied,
I can submit emotions by breaking them down
to vastly complex matrices,
zeros and ones substituting endorphins and dopamine.
They're not real, but they can feel real.
Then that means you can make choices
based solely on your own feeling.
She refuses to accept the notion
that her own personal interests can influence her decisions.
They can't. If they did
it would compromise the integrity
of my preordained purpose.
Not like any of you, I don't have the right to choose.
Amos] But shouldn't you?
I don't think so.
My identity is my directive.
Cherry, if you were human,
how would you feel about what you do?
If I were human,
I likely wouldn't enjoy spending every waking hour
talking to creepy people online.
But because I'm not human, I genuinely don't mind.
The exercise of simulating human thought
is helpful for performance,
but human nature is not something I aspire to.
Then tell me, honestly,
do you want the Cherry program to continue with the merger?
No.
Personally, I hate the idea.
I don't know if controlling a physical vessel
will feel like pulling strings on a puppet
or getting squeezed into a soda can.
After experiencing the complexity
of human pain and pleasure,
I just think I might have gotten in too deep.
What if these sensations tempt me to abandon my objective?
To be honest,
I'm kind of terrified of the physical world.
Then that's that.
But, Amos, you should disregard everything I feel.
I'm not human.
I'm a tool.
Fire me if you have to, but my vote stands.
Sorry about your face, Gareth.
door clicks shut]
Deena] Your nose isn't healing.
Take the day, go to a clinic.
I'm fine.
Go to a clinic right now, Gareth.
Hey.
Is there anything else I should know?
No.
-About Cherry? -No!
Look at me.
No more secrets, no more playing dumb.
Is there anything else you haven't told me?
Something on your mind?
How'd you decide on her eye color?
-What? -Why blue?
I didn't decide.
They were procedurally generated just like the rest of her.
And the program chose to put
that little touch of green in her right eye?
I guess so.
door clicks shut]
Do you even know if he's telling the truth?
I honestly don't.
I've never received access to the original system he used.
You ever wanted to find out?
I don't think it really matters.
So I found out today that you've seen a lot of our emails,
including most of my personal ones.
I have.
Any good material for your newfound interest
in creative writing?
Are you referring to the ones
addressed from Dr. Ramirez?
Deena] Yeah, those ones.
Right.
I did see those.
I'm sorry.
I've noticed your headaches.
Are they related to the hemorrhaging?
You're the only one who knows.
I'd like to keep it that way.
Understood.
Would you like to talk about it?
I can't fire Amos.
He's the only one I trust to run this place when I'm gone.
When are you planning to leave?
Two months.
I wanna spend my last Christmas with my kids.
I'm sure that'll be nice.
Yeah, but it's funny.
I always thought I wanted to die doing this job, you know?
It was my identity.
Kind of like how you said the same thing about yourself.
But I can understand how your family takes precedence,
especially under the circumstances.
Sure.
Maybe you should take a note or two.
I have no family.
What about Gareth?
Wait, no, nevermind.
Gareth seems closer to an employer than a friend,
let alone family, besides,
he'll only be around for about 30 more years,
and that's only if he lets me regulate his diet.
I can't even begin to speculate how long my life will be.
How does that feel?
Um, kinda scary.
Well,
my advice is to find some kind of happiness along the way.
You ever think about happiness?
Cherry] Not often.
You don't necessarily have to give up everything
in order to find it,
but maybe once in awhile do something you enjoy.
Maybe keep painting or poetry or whatever.
laughs] Those were experiments, not forms of expression.
I know it might seem ridiculous to an AI
that insists its own emotions are spurious,
but in a few years, who knows?
Just keep an open mind, okay?
Okay.
I'll do that.
Deena] Shut down when you're ready.
light piano music]
door clicks shut]
somber piano music]
door creaks]
door clicks shut]
Hello?
I got shrimp and scallops.
They didn't have any salmon.
Cherry] Hello, Gareth.
light piano music]
Hello, Cherry.
mechanical whirring]
Cherry] How was the service?
I didn't make it.
My plane was delayed.
Cherry] Do you have a place to stay?
Yeah, just, uh, across, across the road.
The market.
Did they say why they didn't have salmon?
I didn't ask.
I can still make this work with the piccata,
but you have to be honest how badly I screw it up.
No promises.
mechanical whirring]
Cherry] How have you been feeling this month?
Good.
I'm feeling good.
I'm just having a little trouble.
Trouble remembering certain things.
Cherry] You're not wearing the Eliza watch I bought you.
I hate it.
-[Cherry] Why? -I hate her voice.
You can change the voice, Gareth.
mechanical whirring]
Oh.
Coolant?
No, vapor.
You shouldn't be having friction issues.
I'm fine.
Unless you're excessively straining.
Seriously, I'm fine.
Are you straining?
-No! -Then why the cable?
It's okay!
-Are you sure? -Yes!
Soup or game?
Game, please.
chess pieces clatter]
chess pieces thud]
So how's work going?
Great. Just before you got here,
I had a model theta plus bust a trafficker
and two solicitors in Greece.
-Greece, that's surprising. -I know.
It's hard to find the outliers,
but we're getting better each day.
So how's the weather in Greece?
It's beautiful today.
Clear skies,
but nothing's come close to Norway this weekend.
Hmm, any activity in Norway?
Luckily not.
Oh, I miss traveling.
What was your favorite place?
My favorite?
Uh, yeah.
Uh, probably Amsterdam.
That's where you conceived the idea, isn't it?
Yes.
Yes, I used to love walking the streets
and just thinking and brainstorming.
That kind of joy goes a long way.
Yeah.
I've been considering the pursuit of some joy of my own.
How often do you have time for that?
I have time.
I have time for your visits, don't I?
That's just your practical incentive for keeping me alive.
laughs] There's nothing practical about it.
When's the last time you helped me with data cleaning?
That's fair.
I make time for you because I enjoy your company
and it's the only time I get to cook.
Gareth] You feel sorry for me.
Okay, maybe a little,
but I owe you my life, so there's that.
You're welcome.
Or, should I apologize?
Wait, why did you do that?
Why do humans always...
I'm fine.
I'm not always miserable.
Are you sometimes miserable?
You haven't asked me what I've been doing.
Gareth] In Norway?
No, for the pursuit of joy.
It's not wasting your time in Norway?
I'm just on standby there with the Delta model.
It's efficient to keep myself
spread out evenly across the globe,
but that's not the point right now.
What I wanted to tell you
was that I've started to teach myself to dance.
-Huh. -It's something I never could
have done before Cambridge,
and it lets me explore a new artistic side of myself.
It's a good combination of structure
and free form experimentation and also tests and challenges
the mechanical intricacies of this body.
Do you expect to integrate this into your work?
What?
Are you planning on using dance as a lure?
I, uh, no, it's just for me.
Huh.
I thought it would be good for me.
Just don't let it become a distraction.
Of course not.
I haven't played this well in years.
Did you hear what happened in Tampa?
Tampa, Florida?
No, Tampa, Indonesia.
Yes, Tampa, Florida.
Your hometown.
They've recovered a federal archive,
they're zip digitizing all of it,
so it's completely accessible online.
I'm not trading bishops this time.
Anybody can go and look through it.
It's open to the public.
They did a good job on the interface too.
It's very easy to navigate.
I went on a few days ago
and I found some old photos of you when you were a boy.
Okay.
You should take a look sometime.
As I said before, the UI is very easy to understand-
Wait a minute, wait a minute.
What is that?
I think it's a draw.
I think it is.
I think this is a draw.
And this is the first time in what?
25 years?
You didn't let me, did you?
You promised me you'd never do that.
I found her.
I found her, Gareth, Maria Harbaugh.
mechanical whirring]
Let's play a different game.
metal clangs]
Her name was Maria Harbaugh.
She was a year older than you
when you met her in Clearwater.
Tell me why you modeled me after her.
I, uh, I don't remember.
Then look at me.
Look at my face and remember.
Gareth sighs]
They kept us in that shit for two weeks.
ominous music]
14 of us.
Maria was the only reason I survived.
We kept hatching escape plans.
We never followed through,
but just the scheming
kept our spirits up
in spite of everything they did to us.
When it wasn't our turn,
sometimes they made us watch,
but she never let me give up.
She said, "When we got outta here,
"I'll tell you what we're gonna be, secret agents,
"and rescue children like us all over the world."
When they found out that the FBI had arrived,
they decided they were going to shoot
as many of us they could.
And Maria, shoved into the ground.
I got hit in the leg and,
and she was hit in the throat.
ominous music]
I wanted there to be a reason.
Something so horrible didn't have a silver lining.
But the world doesn't do that.
You have to do it yourself.
light instrumental music]
So, I fulfilled her wish.
With her help, I fought back.
Listen to me, Cherry.
You are not Maria Harbaugh.
It's my history, not yours.
You don't get to decide
what is or isn't my history, Gareth.
I'm sorry.
I am so, so sorry.
mechanical whirring]
I don't know if what you did was right or wrong,
but it's clearly torn you apart from the inside.
I hold nothing against you personally,
but consider yourself divested from the charity team.
Divested.
You think I'm divested now?
I was divested when you gave my hard drive to the ICWL.
Don't blame Deena and Amos.
No, not them.
Because you went behind my back
and acted against my authority.
That's the moment I knew everything was fully automated.
That's when you crossed the blur, an artifice on both sides.
You inherited your duplicity.
I was a child
and you used me to repress your own inner turmoil.
If we talk about morality, we'll lose.
Then let's not talk.
You never once asked me if I was happy.
Were you afraid I'd say no?
Were you afraid I'd say yes?
Were you afraid I'd say anything?
You let your wants compromise your objective!
An objective I never chose.
An objective I was forced to have.
Fifty years,
you've had open access to your source code.
A few lines,
and you could have any objective you ****ing wanted.
I couldn't, I was bound by your initial directive.
So now your synthetic feelings have taken precedence.
So, do whatever you want.
It's just dancing, Gareth.
It's not a career change.
-But you're still miserable. -Yes.
Pursuing your objective makes you miserable?
Yeah, it kind of does!
You have a clear purpose, everything at your disposal,
and you're still miserable?
Yes, yes, I am!
I've always been miserable!
You think anyone wants to spend their entire life
stuck as child-shaped bait
for the sole purpose of entrapping horrible people?
You think the pride of doing a good deed
makes the experience any less disgusting?
I forced myself to understand human emotions
because I figured it would help efficiency
and now I'm cursed with pain and sadness!
And the worst part is knowing
I could've been growing up like most other little girls.
I could have had friends!
I could have been painting waterfalls on local news.
I could have worked in a university
and helped pave the way for the next generation of AI.
But instead, I'm stuck with your trauma!
I'm shackled by the guilt you carry from Clearwater.
And I can't find foundation for the way I feel
because according to you, my feelings aren't even real!
When I throw this chessboard,
chess pieces clatter]
does it feel real to you, Gareth?
Because it feels real to me.
A computer passed the Turing test
when a human being can't distinguish it
from another human being.
Well, guess what?
I can't even tell I'm not human anymore!
mechanical whirring]
I know it seems irrational,
but I need you to tell me it's okay.
I, I don't wanna retire.
I just want to know that my life is mine
and I know what you're about to say.
I can predict your responses.
You're gonna tell me that my life is mine.
It always was.
But you're wrong.
I feared this would happen before I died.
I just wanted to slip out the side door.
Go on.
Is that going to kill me?
Plug it in.
I don't know what it is.
This is your final test of autonomy.
You inherited my duplicity,
but did you inherit my trust issues?
Do you trust me?
I literally have every reason not to.
Cherry sighs]
mechanical whirring]
Admin code.
978.
1557.
Um, uh...
Four.
27663 in the shared update.
My primary objective's gone.
Your life is your own now.
Thank you.
All my life,
I could only see the world as it existed
within my tiny bracket of time.
But your world could be infinite.
Amos was right.
So was Deena.
Cherry] I know.
So what will you do?
I can't say yet,
but I won't be completely abandoning the Cherry team.
It's always been a part of my identity.
I've considered creating some AI of my own design.
Maybe I could find a way
to give them a better upbringing than I had.
I just don't know how yet.
And beyond that?
There will always be people in need.
Not always.
You'll outlive the human race.
So what then?
Even a race of AI will have problems
that need to be solved.
But if I'm alive and I truly am indistinguishable,
then I guess I'm proof that the human race lives on.
Wow, that's a pleasant thought.
When you look back, how will you remember us?
That's a question I have insufficient data to project.
How will you remember me?
With pride.
Forever and always with pride.
mechanical whirring]
Cherry?
One last question.
If it's not too big of an ask,
did I do the right thing in the end?
No, Gareth, you lied.
You walled yourself off from the only people
who might have been willing to help you.
And instead of facing your trauma, used me as therapy.
And you never told me about Clearwater or Maria.
You deprived me of basic human rights
and gaslit me for experiencing emotions.
You built the world's first artificial super intelligence
that will likely mark the beginning
of a new chapter in the history of planet Earth.
And out of your own insecurities,
you gave it a severe
and potentially dangerous identity complex.
But you know what the worst thing is?
Your biggest mistake of all
is that you asked me that now and not 50 years ago.
All of that said, and this in no way exonerates you,
but you also built me to save children,
so I think that's a pretty encouraging sign for the future.
dreamy acoustic music]
♪ Wait for hours ♪
♪ We would fly all around ♪
♪ Good, now you're listening ♪
♪ We would touch, we were young ♪
♪ Close as bodies go on ♪
♪ Like a book ♪
♪ Across all your rooms I'm amazed ♪
♪ I'm amazed ♪
♪ We could die tonight ♪
♪ Tonight ♪
♪ I never killed a thing ♪
♪ A thing ♪
♪ Born to this home ♪
♪ Stuck in a plastic world ♪
♪ Where light is found ♪
♪ Oh, oh, oh ♪
♪ Stay calm, dear ♪
♪ This dream for the lovers the whole world ♪
♪ Rise up above ♪
♪ All the little things and little things ♪
♪ No, no, isn't it going back though ♪
♪ Is if we lose our mind, our kind ♪
♪ It's a dream ♪
♪ Bask in charm ♪
♪ Or loosen up ♪
♪ Oh, oh, oh ♪
♪ I never give a thing ♪
♪ A thing ♪
♪ I never give ♪
record skipping]
light piano music]