AI in Hiring: The Promise, Perils, and Unintended Consequences

1
00:00:00,640 --> 00:00:18,039
I'll see you next time.

2
00:00:24,974 --> 00:00:25,863
Nola Simon: so much for joining me.

3
00:00:25,863 --> 00:00:26,613
I'm Nola Simon.

4
00:00:26,613 --> 00:00:29,394
I'm your host of the Hybrid Remote
Center of Excellence and joining

5
00:00:29,404 --> 00:00:31,183
me today is Hilke Schellman.

6
00:00:31,183 --> 00:00:32,444
Did I pronounce your last name right?

7
00:00:32,753 --> 00:00:33,104
Yes.

8
00:00:33,114 --> 00:00:33,394
So it's

9
00:00:33,394 --> 00:00:34,883
Hilke Schellmann: Hilke Schellman.

10
00:00:34,884 --> 00:00:35,364
Hilke.

11
00:00:35,663 --> 00:00:37,604
Yeah, it's a, German first name.

12
00:00:37,703 --> 00:00:38,263
Very hard.

13
00:00:38,263 --> 00:00:39,774
Sorry, my dad is actually German,

14
00:00:39,994 --> 00:00:41,284
Nola Simon: but I don't actually speak

15
00:00:41,284 --> 00:00:41,693
Hilke Schellmann: German.

16
00:00:41,693 --> 00:00:44,494
The funny thing is Germans
don't know the name either.

17
00:00:44,494 --> 00:00:46,624
So they're often like Heike, Zilke, what?

18
00:00:48,114 --> 00:00:50,654
So it is, it is an enigma
for anyone I encounter.

19
00:00:51,094 --> 00:00:52,044
Nola Simon: Okay that's fine.

20
00:00:52,064 --> 00:00:53,104
I'm glad I asked.

21
00:00:53,374 --> 00:00:54,114
So thank you.

22
00:00:54,464 --> 00:00:59,324
She's the author of the algorithm,
how AI decides who gets hired,

23
00:00:59,334 --> 00:01:02,384
monitored, promoted, fired, and
why we need to fight back now.

24
00:01:03,114 --> 00:01:06,474
And so it was actually interesting
how I came across your book, and I

25
00:01:06,474 --> 00:01:09,864
want to tell the audience a little bit
about how I became aware of your work.

26
00:01:10,244 --> 00:01:14,644
And so it started, honestly, I
started on my own noticing like AI

27
00:01:14,664 --> 00:01:18,714
being a trend back when I started my
podcast back in like December of 2021.

28
00:01:18,724 --> 00:01:24,744
And I interviewed a guy from Eightfold
AI and he told me all about the way

29
00:01:24,744 --> 00:01:26,634
they were using it in hiring and firing.

30
00:01:27,034 --> 00:01:31,904
But also he was telling me about how
they can identify transferable skills.

31
00:01:31,924 --> 00:01:33,314
And I found this very interesting.

32
00:01:33,324 --> 00:01:37,304
They have contracts with government to
really identify transferable skills that

33
00:01:37,304 --> 00:01:39,734
existed in long term unemployed people.

34
00:01:40,279 --> 00:01:42,149
With the goal of actually
getting them back to work.

35
00:01:42,149 --> 00:01:44,349
And I'm like, wow, that's fascinating.

36
00:01:44,699 --> 00:01:46,859
I never considered that was a possibility.

37
00:01:47,139 --> 00:01:52,329
And he came up with this example
that, if you have somebody who has

38
00:01:52,349 --> 00:01:54,589
went to university, they have a
math degree, but they have always

39
00:01:54,589 --> 00:02:00,489
worked in customer service, you could
literally get them into data analysis.

40
00:02:00,889 --> 00:02:04,489
because the skill sets are really
the same, but data analysis is

41
00:02:04,489 --> 00:02:09,789
really more future focused and
also pays a lot more, right?

42
00:02:09,849 --> 00:02:14,459
So people are sitting on skill
sets that are extremely valuable.

43
00:02:14,519 --> 00:02:17,849
They don't even realize what the
value is of those skill sets.

44
00:02:17,899 --> 00:02:21,759
And that really appealed to me because
I have a degree in math and I spent

45
00:02:21,779 --> 00:02:23,329
20 years working in customer service.

46
00:02:23,769 --> 00:02:26,499
Now that's how it started.

47
00:02:26,519 --> 00:02:31,289
So then I also was aware of a company
called Plum who worked with Scotia

48
00:02:31,299 --> 00:02:33,409
Bank to actually replace their resumes.

49
00:02:33,439 --> 00:02:37,619
And so basically nobody submits a
resume for Scotia Bank in Canada.

50
00:02:38,069 --> 00:02:38,739
They use this.

51
00:02:39,309 --> 00:02:41,039
AI profile called Plum.

52
00:02:41,509 --> 00:02:46,199
Now, I ran a personal profile for
myself, and it came up with all kinds

53
00:02:46,199 --> 00:02:49,719
of interesting things, except one of
the lines was, basically, you're better

54
00:02:49,719 --> 00:02:54,059
doing work that doesn't really take
any initiative, that it's repetitive,

55
00:02:54,079 --> 00:02:58,349
and you need structure, and you need
to be told what to do, basically.

56
00:02:58,349 --> 00:03:00,139
And I'm like, Yeah, that's not me.

57
00:03:00,909 --> 00:03:01,059
Thank you.

58
00:03:01,539 --> 00:03:02,319
I'm working for myself.

59
00:03:02,319 --> 00:03:05,486
And my podcast, I've got, I'm
nearing a hundred episodes.

60
00:03:05,496 --> 00:03:09,526
So it's yeah, nobody's been
telling me to do my podcast.

61
00:03:09,526 --> 00:03:10,566
Thank you very much.

62
00:03:10,616 --> 00:03:15,106
So I wanted, but what I wanted to know
and why I had approached the CEO of Plum

63
00:03:15,526 --> 00:03:21,626
was what is, What does AI actually know
about me that is pulling that result?

64
00:03:21,736 --> 00:03:26,256
Because if I'm running my own
profile and I notice an inaccuracy

65
00:03:26,256 --> 00:03:28,006
in it, so what's causing it?

66
00:03:28,026 --> 00:03:29,196
Where is it pulling from?

67
00:03:29,476 --> 00:03:33,406
How can I influence that result so
that, I'm going to draw something

68
00:03:33,406 --> 00:03:37,706
that's more accurate, but also
what are the repercussions of that?

69
00:03:38,236 --> 00:03:39,586
inaccuracy, right?

70
00:03:39,626 --> 00:03:41,036
And how do you get it removed?

71
00:03:41,036 --> 00:03:42,046
How do you get it fixed?

72
00:03:42,496 --> 00:03:49,176
And how do you do the due diligence on
doing like on any of that information?

73
00:03:49,176 --> 00:03:53,056
Because if I applied for a job and
that came up and the job is about

74
00:03:53,086 --> 00:03:58,406
innovation and, taking initiative
and, every job that I'd be interested

75
00:03:58,406 --> 00:04:02,326
in, and that's what's coming out of
that profile, that's gonna shoot you

76
00:04:02,326 --> 00:04:04,196
dead in the water right there, right?

77
00:04:04,256 --> 00:04:04,616
Yeah.

78
00:04:05,256 --> 00:04:06,886
And then I could not get
an answer from the company.

79
00:04:06,896 --> 00:04:10,326
She didn't initially respond to me
because I'm like, I want to interview you.

80
00:04:10,376 --> 00:04:14,256
And then she goes to me and she even
actually walked by me on a stage.

81
00:04:14,286 --> 00:04:17,606
I had told her that I would be there and
she walked by me and wouldn't talk to me.

82
00:04:17,606 --> 00:04:18,216
So it's okay.

83
00:04:18,796 --> 00:04:22,536
So I, I've given up on, on that, but
I was fascinated with reading your

84
00:04:22,536 --> 00:04:27,981
book because you focused on both of
those companies and I was like, Oh,

85
00:04:28,011 --> 00:04:32,861
somebody else noticed this and put
it together and put it into the book.

86
00:04:33,011 --> 00:04:37,561
And that's where I knew I really
wanted to interview but we started,

87
00:04:37,571 --> 00:04:38,951
there's one more piece of the story.

88
00:04:39,001 --> 00:04:42,291
I didn't, I'd introduced you to
somebody else that I had as a guest on

89
00:04:42,291 --> 00:04:44,361
my podcast, and that's Swetha Redney.

90
00:04:44,641 --> 00:04:48,091
She's a career coach up in Sudbury.

91
00:04:48,401 --> 00:04:51,331
And I had noticed Swetha trying to get.

92
00:04:51,816 --> 00:04:57,446
Information and trying to get media to
actually interview her about concerns

93
00:04:57,446 --> 00:04:59,996
that she had about AI and immigrants.

94
00:05:00,376 --> 00:05:02,986
If you're if English is not
your 1st language, how does

95
00:05:02,986 --> 00:05:04,936
that influence an AI interview?

96
00:05:05,016 --> 00:05:09,566
These 1 way video interviews, do
you have accommodations, right?

97
00:05:09,596 --> 00:05:12,556
If you have a stutter, if you
have a lisp, all of these things

98
00:05:12,556 --> 00:05:14,736
factor into how you perform.

99
00:05:14,996 --> 00:05:18,456
And personally, again, my own
personal story going back, there's

100
00:05:18,456 --> 00:05:22,126
something about my voice that
automated systems do not like.

101
00:05:22,526 --> 00:05:25,186
So Siri and Alexa hate me.

102
00:05:25,601 --> 00:05:29,101
My kids used to razz me because
they could get Alexa to talk and

103
00:05:29,311 --> 00:05:30,781
Alexa wouldn't respond to me.

104
00:05:30,781 --> 00:05:33,171
So they would harass me using Alexa.

105
00:05:33,701 --> 00:05:37,391
I've turned Siri off because it's so
reliably does not understand my voice

106
00:05:37,431 --> 00:05:39,531
that it's more hassle than it's worth.

107
00:05:40,031 --> 00:05:43,851
And one time I had a bad accident
they actually sorry, I've

108
00:05:43,851 --> 00:05:45,311
got music playing somewhere.

109
00:05:45,341 --> 00:05:47,121
Can you just pause for a moment?

110
00:05:47,591 --> 00:05:47,871
Yeah.

111
00:05:48,291 --> 00:05:48,891
Hang on a second.

112
00:06:14,151 --> 00:06:14,761
Oh my God.

113
00:06:14,761 --> 00:06:15,971
It was an electric.

114
00:06:16,651 --> 00:06:17,521
It started playing in my ear.

115
00:06:19,741 --> 00:06:20,871
Now it heard you.

116
00:06:20,891 --> 00:06:21,261
Finally.

117
00:06:21,261 --> 00:06:21,951
Now it hears me.

118
00:06:22,341 --> 00:06:23,581
This is why I hate it so much.

119
00:06:23,581 --> 00:06:26,941
Because, literally, it was played like
the theme song from the Mickey Mouse Club.

120
00:06:27,751 --> 00:06:31,841
I'm like, did I say anything at
all about the Mickey Mouse Club?

121
00:06:33,291 --> 00:06:34,611
Hilke Schellmann: Wait
till it starts again.

122
00:06:34,981 --> 00:06:35,881
Nola Simon: Oh, God.

123
00:06:36,421 --> 00:06:38,201
Apparently that's dangerous to talk about.

124
00:06:38,721 --> 00:06:41,441
Yeah, so I used to have to, I
was checking the status because

125
00:06:41,441 --> 00:06:43,831
I was waiting for my license to
be reinserted after the accident.

126
00:06:44,241 --> 00:06:47,251
And I had to wait my eight year
old up to say the letter S.

127
00:06:48,026 --> 00:06:50,556
Because the system would not
recognize me saying the letter S.

128
00:06:51,056 --> 00:06:55,346
I literally have concerns about if I ever
had to do one of those video interviews,

129
00:06:55,926 --> 00:06:58,286
how would that actually affect my voice?

130
00:06:58,856 --> 00:07:00,206
English is my first language.

131
00:07:00,706 --> 00:07:02,176
I don't have a concern otherwise.

132
00:07:02,556 --> 00:07:08,066
But there's something about how automated
systems, Recognize my voice that I don't

133
00:07:08,086 --> 00:07:10,036
trust that they're going to get it right.

134
00:07:10,336 --> 00:07:11,456
So I have a lot of concerns.

135
00:07:11,456 --> 00:07:14,196
Hilke Schellmann: So we could
totally test that in different ways.

136
00:07:14,966 --> 00:07:16,056
Yeah, exactly.

137
00:07:16,056 --> 00:07:16,576
That's right.

138
00:07:17,016 --> 00:07:21,646
We usually have people who have
maybe a dialect or an accent, right?

139
00:07:21,656 --> 00:07:24,336
Or have a speech disability
that we mostly concerned about.

140
00:07:24,796 --> 00:07:28,306
I've never had somebody who is
a native English speaker who

141
00:07:28,336 --> 00:07:30,066
has automatic systems that.

142
00:07:30,741 --> 00:07:31,701
Don't want to listen to you.

143
00:07:32,051 --> 00:07:32,571
I know

144
00:07:32,601 --> 00:07:34,661
Nola Simon: it's so
aggravating and so annoying.

145
00:07:34,681 --> 00:07:38,441
And I actually I was working for a bank
before my job was restructured and they

146
00:07:38,441 --> 00:07:40,201
had an automated system as well, too.

147
00:07:40,561 --> 00:07:45,941
And I had run tests for them
demonstrating like how badly you

148
00:07:45,941 --> 00:07:49,934
couldn't recognize me and they didn't
know how to, Handle that right?

149
00:07:49,974 --> 00:07:53,614
And again, I volunteered to be a subject
for them so they could test it out.

150
00:07:53,654 --> 00:07:56,674
But I got restructured before
that project could go anywhere.

151
00:07:57,204 --> 00:08:00,534
But yeah, no, I find that fascinating
to me because again, if this can

152
00:08:00,534 --> 00:08:06,404
happen to me, then how does it happen
to, how does it affect people who

153
00:08:06,414 --> 00:08:09,274
have bigger barriers than I face?

154
00:08:09,374 --> 00:08:09,714
Yeah.

155
00:08:09,804 --> 00:08:13,514
And that's where it's the only thing
that I know how to do anything about is

156
00:08:13,524 --> 00:08:15,674
to ask questions and to talk about it.

157
00:08:16,004 --> 00:08:18,874
And so that's why I'm very grateful
to have you on the podcast so we can

158
00:08:18,874 --> 00:08:24,024
amplify this conversation because I do
think that it's extremely important.

159
00:08:24,594 --> 00:08:27,024
Because it's only the tip of the
iceberg where we are right now.

160
00:08:27,344 --> 00:08:27,624
Yeah.

161
00:08:27,634 --> 00:08:30,724
So before we really get into it
you're a professor at New York

162
00:08:30,724 --> 00:08:33,284
University and you're In the journalism

163
00:08:33,324 --> 00:08:34,734
Hilke Schellmann: department,
teaching journalism.

164
00:08:35,324 --> 00:08:38,619
I'm in, I'm a professor in the
journalism department, teaching

165
00:08:38,619 --> 00:08:40,084
journalism, what I love.

166
00:08:40,094 --> 00:08:41,744
So I have the dream job.

167
00:08:41,744 --> 00:08:43,694
I get to teach what I love to do.

168
00:08:43,704 --> 00:08:45,234
I get to do journalism.

169
00:08:45,244 --> 00:08:47,614
So I'm I feel very blessed with that job.

170
00:08:47,939 --> 00:08:48,239
Nola Simon: Yeah.

171
00:08:48,469 --> 00:08:51,729
And the AI really fell in your
lap too, when you were actually

172
00:08:51,759 --> 00:08:56,399
in a cab and this guy told you
he had an interview with a robot.

173
00:08:56,609 --> 00:09:00,349
And that started, that question
really just started opening up the

174
00:09:00,349 --> 00:09:01,939
whole Pandora's box for you, right?

175
00:09:02,149 --> 00:09:02,889
Hilke Schellmann: Yeah, totally.

176
00:09:02,889 --> 00:09:06,269
We always wonder like where do
journalists get their ideas from and

177
00:09:06,269 --> 00:09:10,169
this was literally like me in the back
of a Lyft ride talking to the driver

178
00:09:10,169 --> 00:09:12,949
and he was like, I've had a weird
day that doesn't ever happen to me.

179
00:09:12,949 --> 00:09:14,489
It's usually they say I'm fine.

180
00:09:14,489 --> 00:09:15,059
How are you?

181
00:09:15,059 --> 00:09:16,639
And he's yeah, I've had a weird day.

182
00:09:16,639 --> 00:09:18,239
And I was like, Oh, yeah, what happened?

183
00:09:18,239 --> 00:09:20,519
He's I was interviewed
by a robot for a job.

184
00:09:20,519 --> 00:09:22,159
And I was like, robot?

185
00:09:22,509 --> 00:09:24,089
He's yeah, I got a call from a robot.

186
00:09:24,239 --> 00:09:26,349
So probably like a pre recorded voice.

187
00:09:26,889 --> 00:09:31,289
And he had applied for a baggage
handler position at at a local airport.

188
00:09:31,609 --> 00:09:34,789
And he was just really just almost
speechless about the process, because

189
00:09:34,789 --> 00:09:38,869
he was like, that has never happened
so weird and I took note of that and

190
00:09:38,869 --> 00:09:40,389
was like, Oh, I've never heard of that.

191
00:09:40,389 --> 00:09:43,259
And then, I went to an AI conference
a few weeks later, and somebody talked

192
00:09:43,269 --> 00:09:47,949
about AI and algorithms being used to
check people's calendars and absences.

193
00:09:47,949 --> 00:09:50,529
And she also did mention, oh,
they use it for hiring and.

194
00:09:50,819 --> 00:09:53,089
She told me about, could the
company hire of you, which is one

195
00:09:53,089 --> 00:09:56,379
of the largest providers in this
space and it just started this.

196
00:09:56,379 --> 00:09:59,739
And when I started looking into it, I was
like I didn't know how ubiquitous it is.

197
00:09:59,739 --> 00:10:02,459
And I think talking to your
point of feeling like, whoa,

198
00:10:02,499 --> 00:10:04,509
like if, It's inaccurate for me.

199
00:10:04,509 --> 00:10:06,389
What is it for people
who have way less agency?

200
00:10:06,389 --> 00:10:11,849
And I think that in introduction of
algorithms and AI and hiring does shift

201
00:10:11,929 --> 00:10:15,419
the power balance, it's always been
more power with the employers, right?

202
00:10:15,449 --> 00:10:18,759
Because make the final decision,
of course, but now it feels like

203
00:10:18,819 --> 00:10:22,249
that, you could like Think about
what am I put in my application?

204
00:10:22,259 --> 00:10:23,549
Who do I list as a reference?

205
00:10:23,559 --> 00:10:26,629
So you were a little bit of the
curator of your passport, maybe.

206
00:10:26,929 --> 00:10:32,339
But now companies assess you with AI
or without even without knowing, right?

207
00:10:32,369 --> 00:10:35,129
Like you may be thinking you're doing
a video interview, little did you

208
00:10:35,129 --> 00:10:38,409
know that they use AI on you or you
send in your application material.

209
00:10:38,709 --> 00:10:42,439
And if you send it through like LinkedIn
or any of the large job platforms,

210
00:10:42,439 --> 00:10:47,039
they all use AI, no recruiter wants to
have just a folder with 2000 applicants

211
00:10:47,039 --> 00:10:48,574
Transcribed Their resumes, right?

212
00:10:48,584 --> 00:10:51,734
There's all kinds of ranking in
the way this ranking happens.

213
00:10:51,734 --> 00:10:53,274
It's like really curious, right?

214
00:10:53,284 --> 00:10:56,864
Like, how do we know that the person
ranked at number one is better than

215
00:10:56,864 --> 00:11:00,654
the person ranked at 100 and those
are like feel interesting questions.

216
00:11:00,654 --> 00:11:02,964
And I started looking into that.

217
00:11:02,964 --> 00:11:07,014
And at the time, like a lot of
the I was first used on people.

218
00:11:07,589 --> 00:11:11,469
What the industry calls like
high turnover high volume job.

219
00:11:11,469 --> 00:11:14,599
So we hire a lot of people and you
have a lot of turnover often for like

220
00:11:14,629 --> 00:11:17,679
retail positions, fast food service.

221
00:11:18,049 --> 00:11:21,649
Yeah, customer service call centers
have a really high turnover too.

222
00:11:21,649 --> 00:11:25,209
So we see, in one of the in one of
the tests that we did the company

223
00:11:25,209 --> 00:11:26,519
is actually for a call center.

224
00:11:26,519 --> 00:11:29,669
So the video is like somebody, you
have this irate person on the phone.

225
00:11:29,679 --> 00:11:31,829
How are you going to calm them
down with one of the tests, right?

226
00:11:32,149 --> 00:11:32,889
So we see that.

227
00:11:33,149 --> 00:11:37,159
And I would say that folks who are looking
for those kinds of job have probably one

228
00:11:37,159 --> 00:11:41,109
of the least power in the workplace and
the least time and get don't make often

229
00:11:41,479 --> 00:11:43,449
a living wage or not near living wage.

230
00:11:43,779 --> 00:11:48,359
And they're being subjected to this
first, whereas CEO is another like sort

231
00:11:48,359 --> 00:11:52,539
of leadership positions very rarely
get subjected to these kinds of tools.

232
00:11:52,539 --> 00:11:56,259
We've seen it like climbed a little bit,
I've seen it used for flight attendants,

233
00:11:56,259 --> 00:12:01,339
for teachers now I've seen it being used
for a lot of recent graduates because a

234
00:12:01,339 --> 00:12:05,899
lot of people feel hiring managers feel
like, Oh, there's so many people, they

235
00:12:05,909 --> 00:12:09,489
all have great bachelor's degree, but
they don't have a lot of work history.

236
00:12:09,489 --> 00:12:10,479
So they all look alike.

237
00:12:10,779 --> 00:12:14,999
So let's do a skills capability assessment
because we don't know a lot about them,

238
00:12:14,999 --> 00:12:16,969
so we see it used mostly for that.

239
00:12:16,969 --> 00:12:20,819
And we see it, I think what happened
with the dawn of job platforms and

240
00:12:20,819 --> 00:12:25,669
now with generative AI companies
are flooded by applications.

241
00:12:25,669 --> 00:12:27,839
And I think that is,
there's a real need there.

242
00:12:27,859 --> 00:12:30,599
IBM said they get about
5 million applications.

243
00:12:30,599 --> 00:12:32,929
Google get about 3 million
applications per year.

244
00:12:33,364 --> 00:12:35,854
Goldman Sachs said for their
summer internship alone, they

245
00:12:35,854 --> 00:12:38,354
got over 220, 000 applications.

246
00:12:38,664 --> 00:12:40,754
There's not enough humans
to go through all of them.

247
00:12:40,754 --> 00:12:42,534
And we know that humans are biased too.

248
00:12:42,754 --> 00:12:46,134
So a lot of companies turn to
technology because they, they're

249
00:12:46,134 --> 00:12:47,504
drowning under applications.

250
00:12:47,804 --> 00:12:51,724
We now more than ever, they're drowning
because, a lot of job applicants use

251
00:12:51,724 --> 00:12:56,664
generative AI to generate a cover letter,
to generate Resume and now we have

252
00:12:56,694 --> 00:12:58,384
a I that can actually apply for you.

253
00:12:58,384 --> 00:12:59,854
So you don't even have to do anything.

254
00:13:00,904 --> 00:13:02,914
Nola Simon: The battle
of who has the best a I

255
00:13:03,604 --> 00:13:06,344
Hilke Schellmann: it is a I versus a I
out there and, at one point, you might

256
00:13:06,344 --> 00:13:08,274
have to ask okay, what's still real here?

257
00:13:08,284 --> 00:13:09,214
What are we talking about?

258
00:13:09,274 --> 00:13:10,684
What are the capabilities?

259
00:13:10,694 --> 00:13:13,544
How do we actually hire the best people?

260
00:13:13,584 --> 00:13:16,064
And I don't think we
necessarily have an answer there

261
00:13:16,474 --> 00:13:19,644
Nola Simon: well, and how many people
are actually avoiding that whole

262
00:13:19,644 --> 00:13:24,624
process and just working the network
and, That then becomes problematic too,

263
00:13:24,624 --> 00:13:30,684
because then it's like, who, and in
terms of equity and, equality, that's,

264
00:13:31,549 --> 00:13:38,019
Automatically problematic because, people
tend to have closed networks, right?

265
00:13:38,049 --> 00:13:40,729
Like they, they like people
who are like them, right?

266
00:13:40,989 --> 00:13:46,719
So if you're building a process, that's
automatically going to be biased,

267
00:13:49,489 --> 00:13:54,429
then is it's example of systems
working as systems are designed

268
00:13:54,429 --> 00:13:58,009
to work and that's really what the
outcome is that everybody wants.

269
00:13:58,834 --> 00:14:00,724
Yeah, we don't Diverse workforce.

270
00:14:01,574 --> 00:14:01,814
Hilke Schellmann: Yeah.

271
00:14:01,814 --> 00:14:02,024
That's a

272
00:14:02,024 --> 00:14:03,554
Nola Simon: question that
you have to ask, right?

273
00:14:03,874 --> 00:14:04,264
Hilke Schellmann: Yeah.

274
00:14:04,454 --> 00:14:06,704
And especially like when
you build tools based on the

275
00:14:06,704 --> 00:14:08,114
current workforce that you have.

276
00:14:08,354 --> 00:14:08,414
Yeah.

277
00:14:08,414 --> 00:14:10,244
Most companies are not diverse, right?

278
00:14:10,294 --> 00:14:13,614
So the the problem is that
you might replicate, right?

279
00:14:13,664 --> 00:14:17,504
The historical inequities that you've
built into your workforce for hiring

280
00:14:17,504 --> 00:14:19,364
more men or more white people, right?

281
00:14:19,364 --> 00:14:20,174
Over time, right?

282
00:14:20,394 --> 00:14:21,654
And then the tools pick up.

283
00:14:22,149 --> 00:14:26,279
Their facial expressions, the words
that they use, their manners, their

284
00:14:26,359 --> 00:14:31,389
way of behaving and game playing
might just hire more white men

285
00:14:31,549 --> 00:14:35,589
which I don't think is anyone's
intention, but is a likely outcome.

286
00:14:36,469 --> 00:14:40,029
Nola Simon: Because they the systems
have access to your hierarchy

287
00:14:40,199 --> 00:14:42,629
through performance reviews or
anything that's written, right?

288
00:14:42,629 --> 00:14:47,349
You have an example in the book about,
this AI had learned that the name Thomas.

289
00:14:47,464 --> 00:14:48,467
It was, yeah, it was getting more points.

290
00:14:48,467 --> 00:14:49,345
My hypothesis is that this was built.

291
00:14:49,345 --> 00:14:49,972
This is a resume screener.

292
00:14:49,972 --> 00:14:55,832
And if the resume happened to include the
name, Thomas, it was rated more, it was,

293
00:14:55,842 --> 00:14:57,882
Hilke Schellmann: it was, yeah,
it was getting more points.

294
00:14:57,942 --> 00:15:00,082
That my hypothesis is that this was built.

295
00:15:00,082 --> 00:15:01,332
This was a resume screener.

296
00:15:01,997 --> 00:15:05,387
that it was probably built with a bunch
of resumes of people who are currently

297
00:15:05,387 --> 00:15:08,847
successful in the role, which often means
the people who are doing the job now.

298
00:15:09,027 --> 00:15:09,267
Nola Simon: Yeah.

299
00:15:09,547 --> 00:15:12,137
Hilke Schellmann: So you give the
tool like thousand resumes of the

300
00:15:12,137 --> 00:15:15,587
people you have employed right now,
or you have been recently employed

301
00:15:15,587 --> 00:15:18,267
the last six months or so made it
to the last round of interviews.

302
00:15:18,657 --> 00:15:20,727
And you say those people
are the successful people.

303
00:15:20,727 --> 00:15:22,687
And then the tool does
a statistical analysis.

304
00:15:22,687 --> 00:15:27,887
And apparently in this Statistical
analysis, the word Thomas came up and

305
00:15:27,927 --> 00:15:31,347
became statistically significant, so
then the tool gave people who had the

306
00:15:31,347 --> 00:15:33,427
word Thomas on their resume more points.

307
00:15:33,567 --> 00:15:37,127
So obviously, any human knows
that Thomas, the word Thomas,

308
00:15:37,137 --> 00:15:38,487
doesn't qualify you for anything.

309
00:15:39,427 --> 00:15:42,457
The machine obviously doesn't have
a conscience and doesn't have any

310
00:15:42,487 --> 00:15:47,052
ethical ideas and I should also say,
Humans are also problematic, right?

311
00:15:47,052 --> 00:15:51,142
We know from like social science
that like we send resumes with more

312
00:15:51,142 --> 00:15:54,432
Caucasian sounding names versus
African American sounding names, right?

313
00:15:54,462 --> 00:15:58,992
There's a lot of human bias too,
that we know that people get fewer

314
00:15:58,992 --> 00:16:02,732
callbacks who have African American
first sounding names, but like a machine

315
00:16:02,842 --> 00:16:04,302
was supposed to be objective, right?

316
00:16:04,332 --> 00:16:05,002
That's what.

317
00:16:05,332 --> 00:16:08,302
These AI vendors sell that
it like democratizes hiring.

318
00:16:08,302 --> 00:16:09,442
It has no bias.

319
00:16:09,472 --> 00:16:10,882
It is absolutely fair.

320
00:16:11,282 --> 00:16:12,852
And that has not been the case.

321
00:16:12,852 --> 00:16:16,322
And I was really surprised when I
talked to industrial organizational

322
00:16:16,322 --> 00:16:19,662
psychologists who said like all of the
tools that you looked at had problems.

323
00:16:20,032 --> 00:16:23,682
And other employment lawyer told me
like every fourth tool he looked at

324
00:16:23,682 --> 00:16:26,122
resume parser had problematic keywords.

325
00:16:26,132 --> 00:16:29,002
He found in one of them,
the word Africa and African

326
00:16:29,002 --> 00:16:30,932
American were used as keywords.

327
00:16:31,312 --> 00:16:32,342
Another tool.

328
00:16:32,737 --> 00:16:34,087
The word baseball.

329
00:16:34,137 --> 00:16:36,607
If you had the word baseball on
your resume, you got more points.

330
00:16:36,607 --> 00:16:38,867
If you had the word softball
on your resume, you got fewer

331
00:16:38,867 --> 00:16:40,537
points, which points, which is

332
00:16:40,537 --> 00:16:43,587
Nola Simon: which is excellent reason
that you don't want to put your

333
00:16:43,827 --> 00:16:46,107
activities your after curriculars.

334
00:16:46,122 --> 00:16:47,622
And I don't

335
00:16:47,622 --> 00:16:48,672
Hilke Schellmann: know what
happens for the people who

336
00:16:48,672 --> 00:16:49,832
don't have any hobbies, right?

337
00:16:49,842 --> 00:16:52,062
We don't know if they get
penalized or not, right?

338
00:16:52,102 --> 00:16:53,312
But that's the problem.

339
00:16:53,372 --> 00:16:58,862
A tool will look at everything that's
on a resume, doesn't know that maybe

340
00:16:58,882 --> 00:17:02,702
Python, a programming language, is a more
important skill for a software developer

341
00:17:02,702 --> 00:17:06,770
job than hobbies, which really should
be, not be part of the decision making

342
00:17:06,770 --> 00:17:10,710
at all, because hobbies doesn't say more
about like your socioeconomic background.

343
00:17:10,720 --> 00:17:15,270
If you put snowboarding or skiing,
probably means in some societies

344
00:17:15,290 --> 00:17:17,570
that you have more money than others
that you come from a privileged

345
00:17:17,740 --> 00:17:19,550
background because you can afford it.

346
00:17:19,920 --> 00:17:23,390
That means more that tells you more
about your background than your actual

347
00:17:23,400 --> 00:17:25,240
skills and capabilities to do the job.

348
00:17:25,355 --> 00:17:28,675
Unless it's a skiing instructor,
they probably need skiing skills.

349
00:17:28,675 --> 00:17:31,605
But other than that it doesn't
have any bearing on the job.

350
00:17:31,615 --> 00:17:37,195
And we see this again and again, and in a
lot of these tools and that does worry me.

351
00:17:37,195 --> 00:17:41,505
And we have so little oversight where
in this case is the vendors themselves

352
00:17:41,525 --> 00:17:42,815
didn't find the problem, right?

353
00:17:42,815 --> 00:17:46,345
It was like when a vendor
talked and talked to an employer

354
00:17:46,635 --> 00:17:48,255
and, about using the system.

355
00:17:48,910 --> 00:17:51,040
And then some of the employers
do their due diligence.

356
00:17:51,040 --> 00:17:54,060
They bring in outside counsel,
they start the system, and

357
00:17:54,060 --> 00:17:55,320
then they find the problem.

358
00:17:55,690 --> 00:18:00,240
And a lot of times companies use
deep neural networks, which you don't

359
00:18:00,240 --> 00:18:02,640
need to know what that means, but it
means that you have training data.

360
00:18:03,340 --> 00:18:06,700
to build the model, and you can look at
the results, but you don't necessarily

361
00:18:06,700 --> 00:18:08,350
know what happens inside the machine.

362
00:18:08,680 --> 00:18:11,850
We can ask the machine what
exactly happened, but most

363
00:18:11,850 --> 00:18:12,870
companies don't do that.

364
00:18:12,880 --> 00:18:16,900
So a lot of AI vendors don't know
what is the machine inferring upon?

365
00:18:17,390 --> 00:18:20,730
And what we see a lot of machine learning,
there's been like a famous example

366
00:18:21,000 --> 00:18:24,480
where I think somebody built a tool that
was supposed to understand what's the

367
00:18:24,480 --> 00:18:26,450
difference between huskies and wolves.

368
00:18:26,615 --> 00:18:28,435
and fed a lot of pictures in the model.

369
00:18:28,785 --> 00:18:31,735
And, the model miraculously learned
what's a husky and a wolf because if

370
00:18:31,735 --> 00:18:35,805
you send in a new photo it was like,
oh, husky, yes, until it didn't work.

371
00:18:36,435 --> 00:18:40,535
And then the the folks asked the
tool what did you infer upon?

372
00:18:40,725 --> 00:18:41,755
Was it the nose?

373
00:18:41,785 --> 00:18:44,035
Was it like the fur of
the husky or the wolf?

374
00:18:44,055 --> 00:18:45,175
How did you know the difference?

375
00:18:45,685 --> 00:18:49,960
And the tool highlighted the snow
in the background of the huskies.

376
00:18:50,340 --> 00:18:50,570
Nola Simon: Because it turns out

377
00:18:50,570 --> 00:18:52,310
Hilke Schellmann: the
Huskies had like snow in the

378
00:18:52,310 --> 00:18:53,580
background, the Wolves did not.

379
00:18:53,590 --> 00:18:56,220
So the tool that actually didn't
know what's the difference

380
00:18:56,220 --> 00:18:57,350
between Husky and Wolves, what?

381
00:18:57,670 --> 00:18:58,680
The Wolves didn't have snow?

382
00:19:00,880 --> 00:19:02,530
Or the Huskies didn't have snow?

383
00:19:02,530 --> 00:19:03,840
I don't remember, actually.

384
00:19:03,940 --> 00:19:05,390
I would have expected
Wolves to have more snow.

385
00:19:05,390 --> 00:19:09,000
One of, one of them didn't have snow.

386
00:19:09,300 --> 00:19:09,480
Yeah.

387
00:19:09,480 --> 00:19:11,550
And that was and that's
what the tool learned.

388
00:19:11,550 --> 00:19:13,990
And so that's what we
call, and this is actually.

389
00:19:14,670 --> 00:19:17,720
More prevalent than you think in
these systems, like we think it

390
00:19:17,720 --> 00:19:21,640
does X, but when we tested, we
learned that it doesn't do anything.

391
00:19:21,640 --> 00:19:25,080
So there's another example with The COVID
cough, like during the pandemic, there

392
00:19:25,080 --> 00:19:29,360
were a lot of companies that tried to
build AI systems where you like cough into

393
00:19:29,360 --> 00:19:32,990
the phone to your doctor, the tool will
tell you, do you have COVID or not, right?

394
00:19:33,010 --> 00:19:34,600
And that never worked out.

395
00:19:34,650 --> 00:19:37,790
And what we saw in even like academic
literature where people said, Oh, we

396
00:19:37,790 --> 00:19:39,980
found the COVID cough when you tested it.

397
00:19:40,430 --> 00:19:44,740
What it actually found was that it
had the people who were, had the

398
00:19:44,750 --> 00:19:47,170
COVID cough and most often in the ICU.

399
00:19:47,170 --> 00:19:49,790
So you heard the beeping of the
machines and the people who didn't

400
00:19:49,800 --> 00:19:53,730
have COVID were in, whatever
scenario where they were coughing.

401
00:19:54,090 --> 00:19:58,220
So the tool had learned that COVID means
machines beeping in the background.

402
00:19:58,250 --> 00:20:00,310
So obviously had to learn
anything about COVID.

403
00:20:00,620 --> 00:20:03,600
So we see this again and
again, it's a real problem.

404
00:20:03,920 --> 00:20:04,930
And I'm.

405
00:20:05,430 --> 00:20:09,530
Not too hopeful that obviously
that the hiring space isn't spared

406
00:20:09,530 --> 00:20:12,130
from this problem because I found
this problem now many times.

407
00:20:12,530 --> 00:20:14,890
And that's just, as you said,
the tip of the iceberg, right?

408
00:20:14,910 --> 00:20:15,990
I haven't looked at all the tools.

409
00:20:15,990 --> 00:20:17,540
I don't have access to all the tools.

410
00:20:17,870 --> 00:20:19,370
So we need a lot more scrutiny.

411
00:20:19,400 --> 00:20:20,790
We need a lot more testing.

412
00:20:21,110 --> 00:20:25,170
So I hope that people in talent
acquisition or in hiring will hear this.

413
00:20:25,785 --> 00:20:28,705
Oh, hear me speak like please
test these tools and do these like

414
00:20:28,745 --> 00:20:30,785
super cheaper tasks that I do.

415
00:20:30,785 --> 00:20:35,625
If I can speak German to a tool and still
get a six out of nine English proficiency

416
00:20:35,625 --> 00:20:37,585
score that the tool probably doesn't work.

417
00:20:37,585 --> 00:20:42,195
Do some kind of testing and to really
understand what does this tool do?

418
00:20:42,235 --> 00:20:45,345
And if it doesn't do what it's
supposed to do, you really might want

419
00:20:45,345 --> 00:20:47,175
to rethink using this for hiring.

420
00:20:48,145 --> 00:20:49,685
It matters who gets a job.

421
00:20:49,705 --> 00:20:51,185
It matters to job seekers.

422
00:20:51,185 --> 00:20:54,475
I'm relying on making money to put
food on the table to have an apartment.

423
00:20:54,525 --> 00:20:58,035
I'm nervous before a job interview
because it matters if I get the job.

424
00:20:58,405 --> 00:21:01,115
I know for employers, often it
feels ah, so many candidates.

425
00:21:01,125 --> 00:21:04,010
We reject any, them anyways,
but It doesn't matter.

426
00:21:04,010 --> 00:21:06,680
And I'm sure it mattered to the
hiring manager and the talent

427
00:21:06,680 --> 00:21:10,390
acquisition manager at one point
that they did get the job, right?

428
00:21:10,440 --> 00:21:12,910
Nola Simon: And there's legal
ramifications as well too.

429
00:21:13,430 --> 00:21:18,010
So if you're using tools that have
built in discrimination, like you

430
00:21:18,010 --> 00:21:21,430
said, like they're learning things
that aren't necessarily relevant.

431
00:21:21,510 --> 00:21:24,840
An example would be like just
university related to like

432
00:21:24,840 --> 00:21:26,240
social economic class, right?

433
00:21:26,250 --> 00:21:28,550
If you're filtering out your
AI tools, filtering out,

434
00:21:28,650 --> 00:21:31,320
Harvard or whatever university.

435
00:21:31,370 --> 00:21:36,470
There's long time studies and proof that
show that those aren't necessarily tied

436
00:21:36,470 --> 00:21:41,020
to skills and ability, but those are
definitely tied to socio economic class,

437
00:21:41,370 --> 00:21:41,600
Hilke Schellmann: right?

438
00:21:41,600 --> 00:21:41,930
Yeah.

439
00:21:42,110 --> 00:21:45,880
Yeah, and I feel especially with the
resume parser there was 1 example that it

440
00:21:45,880 --> 00:21:50,670
had highlighted, had learned that Syria
and Canada are, Predictions of success.

441
00:21:50,680 --> 00:21:52,920
So everyone else was fascinated
with that one since I'm

442
00:21:52,920 --> 00:21:53,400
Nola Simon: from Canada.

443
00:21:55,000 --> 00:21:55,480
Hilke Schellmann: Exactly.

444
00:21:55,480 --> 00:21:59,270
I was like, look, the Canadians, like
they're, moving forward at a faster clip.

445
00:21:59,280 --> 00:22:01,900
But you know what this means for
people from other places that

446
00:22:01,900 --> 00:22:05,270
they would be potentially get
discriminated against, which is

447
00:22:05,380 --> 00:22:06,920
illegal in the United States, right?

448
00:22:06,930 --> 00:22:08,840
Like they're illegal place.

449
00:22:08,840 --> 00:22:08,920
Yeah.

450
00:22:08,950 --> 00:22:13,170
I think what's a little bit problematic is
like that the tools themselves obfuscate

451
00:22:13,180 --> 00:22:14,440
some of the decision making, right?

452
00:22:14,440 --> 00:22:19,640
Because we don't actually know that Canada
and Syria were indicators of success,

453
00:22:19,640 --> 00:22:23,430
which we shouldn't be using, because
unless somebody digs deep into the system,

454
00:22:23,960 --> 00:22:25,800
we just see the results of the ranking.

455
00:22:25,980 --> 00:22:28,160
And they look very convincing.

456
00:22:28,160 --> 00:22:32,860
Even when I did tests, I know that
these tests are bogus that I was like,

457
00:22:32,860 --> 00:22:35,210
okay, there's no science behind them.

458
00:22:35,220 --> 00:22:39,170
But when you still see your score,
you're like 78%, blah, blah, blah.

459
00:22:39,170 --> 00:22:42,890
It's really hard to not see that score
and feel like, oh, that means I'm this.

460
00:22:43,180 --> 00:22:47,110
You still put meaning towards it, even
though, it's actually meaningless.

461
00:22:48,300 --> 00:22:49,020
And I think that's.

462
00:22:49,205 --> 00:22:50,575
What's really problematic, right?

463
00:22:50,575 --> 00:22:54,225
As an hiring manager, you get
this list of 1000 people, and

464
00:22:54,225 --> 00:22:55,595
you're going to call the first 20.

465
00:22:55,645 --> 00:23:01,195
I'm not going to call 998 that person,
even though I have no idea how this

466
00:23:01,195 --> 00:23:02,955
banking was really put together, right?

467
00:23:02,985 --> 00:23:06,465
I assume the machine, somebody
did their due diligence, and these

468
00:23:06,495 --> 00:23:09,245
machines work and pick the best people.

469
00:23:09,805 --> 00:23:13,765
But I think it was interesting from one
survey that Joe Fuller did at Harvard.

470
00:23:13,865 --> 00:23:18,165
We know that he surveyed like
2200 or so people in leadership

471
00:23:18,965 --> 00:23:20,955
when their company use an AI tool.

472
00:23:21,005 --> 00:23:24,365
Almost 90 percent said, Oh, we know
that they reject qualified candidates.

473
00:23:25,000 --> 00:23:28,150
So we know that they don't actually
fulfill that promise that they

474
00:23:28,150 --> 00:23:29,780
find the most qualified candidate.

475
00:23:30,060 --> 00:23:33,270
But we still use it because I think
the tools make hiring so much more

476
00:23:33,270 --> 00:23:38,270
efficient, so much easier, cut down
the number of days that you're hiring.

477
00:23:38,280 --> 00:23:41,360
And that is something that companies
care about, which I think is

478
00:23:41,380 --> 00:23:42,790
probably the wrong incentive.

479
00:23:42,870 --> 00:23:44,830
Who cares if you found
somebody in two days?

480
00:23:45,240 --> 00:23:46,360
Are they the most qualified?

481
00:23:46,360 --> 00:23:48,230
Can they actually do the job the best?

482
00:23:48,550 --> 00:23:50,800
If it takes three weeks maybe that's okay.

483
00:23:50,880 --> 00:23:54,600
That isn't necessarily the
incentive that is being used,

484
00:23:54,910 --> 00:23:55,330
Nola Simon: right?

485
00:23:55,330 --> 00:23:56,080
Yeah, exactly.

486
00:23:56,080 --> 00:23:56,480
That's right.

487
00:23:56,490 --> 00:23:59,600
And it comes down to
trust and accountability.

488
00:23:59,600 --> 00:23:59,970
Really?

489
00:24:00,020 --> 00:24:01,340
Are you walking the talk?

490
00:24:01,340 --> 00:24:03,780
And are you being
responsible for what it is?

491
00:24:03,780 --> 00:24:07,540
You're doing and that seems
to be lacking in companies

492
00:24:07,550 --> 00:24:08,660
that are actually using these.

493
00:24:08,870 --> 00:24:10,110
These AI systems.

494
00:24:10,140 --> 00:24:13,020
And again, I don't know that's
their intention, but it is.

495
00:24:13,150 --> 00:24:13,630
I don't think it's

496
00:24:13,630 --> 00:24:14,690
Hilke Schellmann: at all their intention.

497
00:24:14,700 --> 00:24:19,880
Yeah, this is like a tale of like
good intentions, maybe gone bad or

498
00:24:19,880 --> 00:24:21,820
having unintended consequences, right?

499
00:24:21,820 --> 00:24:25,000
I don't think actually anyone
building these systems or using

500
00:24:25,000 --> 00:24:26,750
these systems has any ill intent.

501
00:24:27,140 --> 00:24:28,060
I think everyone.

502
00:24:28,200 --> 00:24:31,020
Really does probably want to
diversify the workforce, want to

503
00:24:31,030 --> 00:24:34,870
give more people a chance but it's
not really fully understanding how

504
00:24:34,870 --> 00:24:36,230
are we actually using this system?

505
00:24:36,230 --> 00:24:38,080
How do these systems make decisions?

506
00:24:38,400 --> 00:24:42,610
Yeah, because they are literally black
boxes, like that, what we always say,

507
00:24:42,610 --> 00:24:45,085
and I think that's, really problematic.

508
00:24:45,085 --> 00:24:46,985
And I think, a lot of
companies come into this.

509
00:24:47,015 --> 00:24:49,245
They want to have they want to save money.

510
00:24:49,275 --> 00:24:51,445
They want to have more
efficient hiring technology.

511
00:24:51,445 --> 00:24:55,325
So they don't want to necessarily
now hire someone to oversee these

512
00:24:55,325 --> 00:25:00,405
systems and test them continuously,
like they're easier and faster.

513
00:25:00,595 --> 00:25:04,045
So they're not going to maybe apply
like a lot of scrutiny to this.

514
00:25:04,075 --> 00:25:05,205
They're just like, okay, it works.

515
00:25:05,205 --> 00:25:06,375
It cuts down hiring.

516
00:25:06,375 --> 00:25:07,975
We need to employ fewer people.

517
00:25:08,350 --> 00:25:09,000
That's all I want.

518
00:25:09,500 --> 00:25:09,710
And I

519
00:25:09,710 --> 00:25:13,400
Nola Simon: mean, it's basically
software as a service, right?

520
00:25:13,400 --> 00:25:17,150
So like the vendors could be updating
the software as they go, right?

521
00:25:17,150 --> 00:25:20,170
So they really should keep on top of that.

522
00:25:20,170 --> 00:25:22,570
Just from a compliance point of
view, you would think that you'd want

523
00:25:22,570 --> 00:25:25,970
to make sure that you're reviewing
the terms and conditions every time

524
00:25:25,970 --> 00:25:27,650
there's an update to the software.

525
00:25:28,120 --> 00:25:30,400
I don't understand what the
risk people are like, where's

526
00:25:30,400 --> 00:25:34,720
the cybersecurity and the risk
assessment that goes along with this.

527
00:25:35,060 --> 00:25:35,370
Hilke Schellmann: Yeah.

528
00:25:35,370 --> 00:25:38,980
And I'm sometimes wondering about
this too, because, like we reported

529
00:25:39,010 --> 00:25:42,420
very early on in 2018, after I met the
Lyft driver and the next year I was

530
00:25:42,420 --> 00:25:43,440
working for the Wall Street Journal.

531
00:25:43,440 --> 00:25:46,880
And we did this like a 10 minute,
as a video investigation, because we

532
00:25:46,880 --> 00:25:50,110
felt like, oh, this is, it's about
video interviews mostly and other

533
00:25:50,140 --> 00:25:53,380
AI and hiring, but it felt like,
oh, we want to show this in video.

534
00:25:53,770 --> 00:25:59,060
And we looked at HireVue and HireVue at
the time still use like facial expression

535
00:25:59,060 --> 00:26:02,575
analysis Ation of our voice and analysis.

536
00:26:02,575 --> 00:26:05,355
And it also transcribed
the words that you used.

537
00:26:05,765 --> 00:26:09,125
And, when I started digging deeper,
at first I was like so surprised.

538
00:26:09,125 --> 00:26:11,765
I was like, oh, I didn't know that
facial expression and job interviews

539
00:26:11,765 --> 00:26:13,445
are predictive of success in a job.

540
00:26:13,495 --> 00:26:16,045
What Interesting way to measure things.

541
00:26:16,405 --> 00:26:18,715
And then when I dug deeper and
I talked to expert, they're

542
00:26:18,715 --> 00:26:20,155
like, there's no science here.

543
00:26:20,155 --> 00:26:20,545
I was like.

544
00:26:21,305 --> 00:26:21,785
Wait, what?

545
00:26:21,815 --> 00:26:22,695
There's no science here.

546
00:26:23,475 --> 00:26:24,915
They're like, yeah, we
don't have any science.

547
00:26:24,925 --> 00:26:28,065
What facial expressions you have to have
in a job interview that predicts how

548
00:26:28,065 --> 00:26:29,805
good you should be at any given job.

549
00:26:29,835 --> 00:26:32,675
And I was like, Oh but they're using it.

550
00:26:33,065 --> 00:26:36,295
And and then I talked to folks, there
is the effective computing companies

551
00:26:36,295 --> 00:26:37,955
that, that, that put out the tools.

552
00:26:38,335 --> 00:26:40,665
They also say we have
the same six emotions.

553
00:26:40,665 --> 00:26:42,105
If you're smiling, you're happy.

554
00:26:42,105 --> 00:26:43,575
If you're frowning, you're angry.

555
00:26:43,625 --> 00:26:47,525
And when I talked to psychologists,
they said Oh yeah, that's 60 years ago.

556
00:26:47,715 --> 00:26:51,645
We thought that, but we know
now that like different cultural

557
00:26:51,655 --> 00:26:55,565
backgrounds, different societies,
we don't have the same emotions.

558
00:26:55,565 --> 00:26:58,935
Like we are likely to have
similar emotions, but not always.

559
00:26:59,045 --> 00:27:00,905
And so this is not
predictive at all either.

560
00:27:00,905 --> 00:27:05,545
And I was like, Oh, I was like, this
is really problematic, but large

561
00:27:05,555 --> 00:27:10,245
fortune 500 companies were using the
technology without questioning it.

562
00:27:10,690 --> 00:27:11,660
From what I know.

563
00:27:11,950 --> 00:27:16,110
So I was really surprised that I was like,
I do wonder, like, where are those people?

564
00:27:16,110 --> 00:27:17,360
Where are the compliance people?

565
00:27:17,690 --> 00:27:19,550
I don't know if each into that.

566
00:27:20,395 --> 00:27:25,525
hiring space or they feel like because
there is like almost no way for

567
00:27:25,525 --> 00:27:27,295
people to get access to these systems.

568
00:27:27,295 --> 00:27:30,965
If the person who builds the system
doesn't know what the tool infers

569
00:27:30,965 --> 00:27:33,985
upon, like the people using it,
the companies don't know it, job

570
00:27:33,985 --> 00:27:35,245
seekers are never going to find out.

571
00:27:35,245 --> 00:27:36,745
So maybe the risk is very low.

572
00:27:37,035 --> 00:27:37,455
Nola Simon: Yeah.

573
00:27:37,595 --> 00:27:38,675
There's no transparency.

574
00:27:38,705 --> 00:27:42,925
Like even as a job seeker, it's only
because of press releases and whatnot.

575
00:27:42,925 --> 00:27:45,815
Like I knew that Scotiabank was
using Plum, for example, and

576
00:27:45,815 --> 00:27:47,275
that's why I was researching Plum.

577
00:27:47,615 --> 00:27:51,740
But most of the time they don't identify
who's providing the documentation.

578
00:27:53,510 --> 00:27:54,730
the services, right?

579
00:27:54,790 --> 00:27:55,180
Yeah,

580
00:27:55,390 --> 00:27:57,310
Hilke Schellmann: I think after,
like some sort of scandals, right?

581
00:27:57,310 --> 00:28:02,200
Amazon tried to build an algorithmic
tool that scans resumes, and they built

582
00:28:02,200 --> 00:28:04,780
a tool that was downweighing women.

583
00:28:04,780 --> 00:28:08,570
If you had the word woman or women on
your resume, it would downweigh you.

584
00:28:09,200 --> 00:28:13,540
So I think, They got a lot of backlash
for that as I think since that companies

585
00:28:13,540 --> 00:28:18,690
have also been very careful to publicize
anything bad about these tools, right?

586
00:28:18,900 --> 00:28:22,260
Like I have talked to, like after
the book came out, I got lucky.

587
00:28:22,260 --> 00:28:24,920
Before too, but like really after the
book came out, I had talked to a lot

588
00:28:24,920 --> 00:28:29,030
of CHROs, like chief human resource
officers, chief people officers,

589
00:28:29,430 --> 00:28:31,090
and Oh yeah, we use that tool.

590
00:28:31,090 --> 00:28:33,810
We have the same questions and
problems that you uncovered.

591
00:28:33,810 --> 00:28:35,330
And then we like stopped using it.

592
00:28:35,370 --> 00:28:36,800
And I was like that's great.

593
00:28:36,800 --> 00:28:37,710
I'm glad to hear it.

594
00:28:37,740 --> 00:28:38,930
Did you tell anyone?

595
00:28:39,220 --> 00:28:41,330
Because Company X is still using it.

596
00:28:41,430 --> 00:28:44,700
Could you publicly say that you
found out these tools didn't work?

597
00:28:44,700 --> 00:28:47,905
And they're like, No, we can't publicly
acknowledge that because they're

598
00:28:47,905 --> 00:28:49,315
afraid of class action lawsuits.

599
00:28:49,625 --> 00:28:52,385
They're afraid of press that
they use the tool that's flawed.

600
00:28:53,475 --> 00:28:54,555
You can understand it.

601
00:28:54,575 --> 00:28:57,915
But the problem is we don't really
have much progress then, right?

602
00:28:57,945 --> 00:29:03,055
No one calls out the bad actors, or not
the bad actors, but like when tools don't

603
00:29:03,065 --> 00:29:04,655
work, they don't call out the companies.

604
00:29:04,655 --> 00:29:09,655
And, I think the vendors are unlikely
to dig deep and really check out

605
00:29:09,655 --> 00:29:11,125
their systems and change them.

606
00:29:11,125 --> 00:29:13,735
They do want to market and sell, right?

607
00:29:13,755 --> 00:29:17,895
And there's still companies that
merely believe them that this works.

608
00:29:17,895 --> 00:29:18,655
So yeah.

609
00:29:19,465 --> 00:29:21,185
Nola Simon: So what's the solution?

610
00:29:21,225 --> 00:29:25,105
Like you've been on record saying
that, regulation is needed.

611
00:29:25,165 --> 00:29:26,855
Government involvement is needed.

612
00:29:27,415 --> 00:29:27,955
Hilke Schellmann: Yeah.

613
00:29:27,955 --> 00:29:31,095
We just did a recent election in
this country, so I'm not sure.

614
00:29:31,330 --> 00:29:34,810
under a Trump administration, there
will be more regulation, right?

615
00:29:34,820 --> 00:29:38,580
In fact, we've seen a lot of deregulation
under the last Trump administration.

616
00:29:38,580 --> 00:29:39,470
So I'm not very hopeful.

617
00:29:39,480 --> 00:29:43,900
We haven't seen some forward movement
under the Biden administration.

618
00:29:43,900 --> 00:29:48,000
There has been like AI not
lost, but like hopeful, like we

619
00:29:48,000 --> 00:29:49,330
should really look into this.

620
00:29:49,520 --> 00:29:54,700
And there has been like progress but we
really haven't seen an agency stepping in.

621
00:29:55,645 --> 00:29:59,105
starting to approve these
tools or test them or anything.

622
00:29:59,325 --> 00:30:02,245
I think that is, there isn't a whole
lot of political will, I think, to

623
00:30:02,245 --> 00:30:03,815
do that, and maybe also not a lot of.

624
00:30:04,380 --> 00:30:09,580
knowledge because it is tricky to, and I
wouldn't recommend that government agency

625
00:30:09,580 --> 00:30:11,390
do this like little testing that I do.

626
00:30:11,820 --> 00:30:15,880
I sometimes do this thing where bring a
question to like computer scientists and

627
00:30:15,880 --> 00:30:18,825
sociologists and we do larger Studies.

628
00:30:19,035 --> 00:30:19,275
Yeah.

629
00:30:19,275 --> 00:30:21,915
That's how we found the problem and
their hallucinations and whisper.

630
00:30:22,195 --> 00:30:26,695
That's also how we found out that some AI
tools who take your personality from your

631
00:30:26,755 --> 00:30:32,015
LinkedIn or your Twitter feed don't work
and that you have multiple personalities

632
00:30:32,015 --> 00:30:36,455
and even the same AI tool has different
things finding out about your personality.

633
00:30:36,815 --> 00:30:39,460
, if it looks to you at your LinkedIn
or Twitter, which is, against the

634
00:30:39,460 --> 00:30:43,990
theory, like if you use personality,
you would assume that it is a stable.

635
00:30:44,315 --> 00:30:44,895
Thing, right?

636
00:30:44,915 --> 00:30:47,085
Otherwise, if it changes every
5 minutes, then you really

637
00:30:47,085 --> 00:30:48,255
shouldn't use it for hiring.

638
00:30:48,625 --> 00:30:49,435
So we had also

639
00:30:49,485 --> 00:30:52,085
Nola Simon: Social media is a
filter to begin with, right?

640
00:30:52,095 --> 00:30:55,815
Who you are on LinkedIn may be very
different from who you present yourself

641
00:30:55,815 --> 00:30:57,895
to be on Twitter or threads or whatever.

642
00:30:57,895 --> 00:30:58,605
I absolutely agree.

643
00:30:58,935 --> 00:31:01,715
You adopt personas, or
at least some people do.

644
00:31:02,165 --> 00:31:04,265
Hilke Schellmann: Yeah but I
think the companies would say you

645
00:31:04,265 --> 00:31:06,095
didn't write this to get a job.

646
00:31:06,365 --> 00:31:08,545
Especially your tweets, so
there's something about you

647
00:31:08,545 --> 00:31:09,665
that you're not hiding, right?

648
00:31:09,665 --> 00:31:12,115
This idea to look under
the hood of a job seeker.

649
00:31:13,055 --> 00:31:14,265
It is pretty prevalent.

650
00:31:14,265 --> 00:31:17,515
I think the same idea with checking
your facial expressions like you

651
00:31:17,525 --> 00:31:21,055
get something up about yourself
that you weren't even knowing about.

652
00:31:21,125 --> 00:31:26,035
And we know from history of
companies analyzing like handwriting

653
00:31:26,065 --> 00:31:28,865
and other things and hiring
that, there is no science there.

654
00:31:28,975 --> 00:31:29,625
There is no science.

655
00:31:29,625 --> 00:31:33,305
But we've seen this again and
again, because it does feel.

656
00:31:34,120 --> 00:31:35,670
So intuitive, right?

657
00:31:35,670 --> 00:31:39,000
It does feel intuitive that something in
your writing says something about you.

658
00:31:39,010 --> 00:31:42,370
So I could analyze that and find
the real you that you can't hide.

659
00:31:42,910 --> 00:31:44,670
But so we fall into that again.

660
00:31:44,680 --> 00:31:47,820
Those are those are assumptions that,
that, that worry me in this space.

661
00:31:47,820 --> 00:31:48,780
Let me see.

662
00:31:49,620 --> 00:31:49,910
Yeah.

663
00:31:50,700 --> 00:31:52,730
Nola Simon: But Europe isn't
having the same sort of problem

664
00:31:52,780 --> 00:31:54,300
because of the GDPR, right?

665
00:31:54,390 --> 00:31:55,240
And the privacy?

666
00:31:57,460 --> 00:31:58,460
Hilke Schellmann: Yes and no.

667
00:31:58,540 --> 00:32:02,740
I do think that Europe is like a
more regulated place on earth, right?

668
00:32:02,760 --> 00:32:05,910
Like we see much more regulation
coming out of the European Union.

669
00:32:06,440 --> 00:32:10,765
And we see, they have, we call
those like omnibus legislation

670
00:32:10,765 --> 00:32:12,545
because there are large laws.

671
00:32:12,565 --> 00:32:16,675
So one is GDPR, the general data
protection privacy law even the

672
00:32:16,675 --> 00:32:19,255
UK adopted, although it's no
longer in the European union.

673
00:32:19,285 --> 00:32:23,985
And I was like lucky to find who I call
like patient zero Martin Birch who got

674
00:32:23,985 --> 00:32:29,065
his who had to take a plum assessment,
like a gamified personality test to,

675
00:32:29,115 --> 00:32:30,975
to really basically describe it you.

676
00:32:31,055 --> 00:32:34,445
Get some, I don't know, you
do some Tetris thing games.

677
00:32:34,445 --> 00:32:36,335
And you have to answer
a couple of questions.

678
00:32:36,335 --> 00:32:39,075
What would you in the workplace,
if you encounter this or that and

679
00:32:39,075 --> 00:32:42,985
they come up with the personality
and capability analysis of you.

680
00:32:43,495 --> 00:32:47,415
So he was he had applied
for a job in data.

681
00:32:47,455 --> 00:32:50,445
And he was like, I was data
scraping all day at my current job.

682
00:32:50,715 --> 00:32:54,550
So he was like, I was really surprised
that, I wasn't, asked to do an assessment

683
00:32:54,560 --> 00:32:58,240
on my capability of data analysis and
data scraping, which I do in my job,

684
00:32:58,250 --> 00:33:00,170
which I applied for at Bloomberg.

685
00:33:00,550 --> 00:33:03,210
But I was asked to answer these
questions about my personality and

686
00:33:03,210 --> 00:33:05,320
puzzle Tetris and other things.

687
00:33:05,370 --> 00:33:09,000
And he was also surprised that
he pretty much got an instant

688
00:33:09,010 --> 00:33:10,640
within a day, a rejection.

689
00:33:10,710 --> 00:33:11,590
He was like, Huh.

690
00:33:12,185 --> 00:33:15,505
For a job that I'm already doing that
I applied for at this other company.

691
00:33:15,505 --> 00:33:16,395
That's interesting.

692
00:33:16,645 --> 00:33:19,235
So he was in a journalism
adjacent job, right?

693
00:33:19,305 --> 00:33:21,205
He was for a news organization.

694
00:33:21,485 --> 00:33:24,675
He wasn't like a reporter,
but he knew what reporters do.

695
00:33:25,075 --> 00:33:26,525
So he thought about the GDPR.

696
00:33:26,695 --> 00:33:28,055
He asked for his data.

697
00:33:28,735 --> 00:33:31,815
And then he got the data and he
scored really low and then he found

698
00:33:31,815 --> 00:33:38,285
out that Bloomberg had automatically
rejected him, which is under, some

699
00:33:38,285 --> 00:33:42,975
other laws, is not legal, you have
to disclose that if you use automatic

700
00:33:45,555 --> 00:33:46,785
And Bloomberg didn't disclose that.

701
00:33:46,785 --> 00:33:50,205
So he started, he started like
a inquiry, a legal inquiry.

702
00:33:50,205 --> 00:33:54,215
And he settled with the company, but,
it's one of the first people that I ever

703
00:33:54,225 --> 00:33:58,955
heard from that like actually had the
data to look at and ask about it exactly.

704
00:33:59,215 --> 00:34:01,365
And I think that happens so rarely.

705
00:34:01,815 --> 00:34:03,595
And it doesn't happen very
often in Europe, right?

706
00:34:03,595 --> 00:34:06,165
Because a lot of Europeans don't
even know that they have this

707
00:34:06,165 --> 00:34:07,555
right or they know how to get it.

708
00:34:07,565 --> 00:34:11,115
And then when you get the data, it's
also really hard to really understand it.

709
00:34:11,115 --> 00:34:11,615
Okay.

710
00:34:11,695 --> 00:34:14,515
Capability or not, I'm, I don't
know if it's right or not.

711
00:34:14,515 --> 00:34:17,435
There's no authority to go to
and say I don't think it's right.

712
00:34:17,735 --> 00:34:18,775
Who would you call?

713
00:34:18,835 --> 00:34:22,795
We see a little bit more and now we see
the EU AI Act, which actually outlaws

714
00:34:22,825 --> 00:34:24,675
outright the emotion recognition.

715
00:34:24,965 --> 00:34:28,815
So that kind of tools would
not, will no longer exist in

716
00:34:28,815 --> 00:34:30,545
the European Union legally.

717
00:34:30,545 --> 00:34:31,965
If you use it, it would be illegal.

718
00:34:32,225 --> 00:34:35,985
And there's a high bar for hiring,
which I'm really happy about because

719
00:34:35,985 --> 00:34:39,145
I've been saying for a while now that
I think, hiring is high risk decision

720
00:34:39,145 --> 00:34:43,460
making, like it makes It is important to
understand who gets hired and we're not.

721
00:34:43,460 --> 00:34:46,690
So it should be up there with
like other high risk things about

722
00:34:46,690 --> 00:34:49,760
who gets a mortgage, who gets a
loan, who goes to jail, like those

723
00:34:49,760 --> 00:34:51,100
are or how long you go to jail.

724
00:34:51,360 --> 00:34:54,870
Those are also high risk decision
making and hiring should be one of them.

725
00:34:54,870 --> 00:34:58,600
So there is a little bit more
scrutiny that companies have to face.

726
00:34:59,470 --> 00:35:02,720
We not exactly know how it's going
to interpret it in every country and

727
00:35:02,720 --> 00:35:04,890
every like sort of societal system.

728
00:35:05,310 --> 00:35:06,840
There's like general guidance.

729
00:35:07,195 --> 00:35:08,765
How is this actually
going to be implemented?

730
00:35:08,765 --> 00:35:09,695
It's going to be interesting.

731
00:35:10,125 --> 00:35:13,115
So we're not 100 percent sure,
but we see, a little bit of that.

732
00:35:13,145 --> 00:35:17,215
We've seen promising things in California,
but they've recently been turned down

733
00:35:17,215 --> 00:35:18,825
by the government, governor there.

734
00:35:19,115 --> 00:35:21,655
With the Trump administration,
I'm not seeing that we'll

735
00:35:21,655 --> 00:35:22,845
have a lot of regulation.

736
00:35:22,855 --> 00:35:25,145
Maybe on the state level,
but not on the federal level.

737
00:35:25,165 --> 00:35:25,895
Probably not.

738
00:35:26,445 --> 00:35:27,545
So we'll see what happens.

739
00:35:27,545 --> 00:35:30,805
I do think there needs to be more
regulation, but I think actually, I don't

740
00:35:30,805 --> 00:35:32,705
know if every regulation will catch.

741
00:35:33,845 --> 00:35:35,045
Problematic use case.

742
00:35:35,515 --> 00:35:40,085
So I think that like companies need
to speak out and push the vendors to

743
00:35:40,085 --> 00:35:44,675
do better, or we need to have more
non for profit initiatives to build

744
00:35:44,675 --> 00:35:46,235
tool in the public interest, right?

745
00:35:46,255 --> 00:35:49,745
I think as the barriers of building
some of these tools come down and

746
00:35:49,745 --> 00:35:52,105
make it easier for a lot of people.

747
00:35:52,470 --> 00:35:56,440
To do the question is could we, could
civil society build tools in the public

748
00:35:56,440 --> 00:36:01,160
interest that are better than some
of the commercial companies put out

749
00:36:01,160 --> 00:36:05,350
there and we can actually make public
how the tools were built, what the

750
00:36:05,350 --> 00:36:09,010
assumptions are built in and all of those
things that that we criticize that we

751
00:36:09,010 --> 00:36:10,830
don't know from commercial suppliers.

752
00:36:11,175 --> 00:36:11,915
That's all I do.

753
00:36:12,475 --> 00:36:13,675
Nola Simon: Are you
familiar with Amy Webb?

754
00:36:13,905 --> 00:36:14,805
She's a futurist.

755
00:36:14,885 --> 00:36:15,235
Hilke Schellmann: Yes.

756
00:36:15,285 --> 00:36:15,665
Yes.

757
00:36:15,935 --> 00:36:17,585
I've interviewed her
actually for an article.

758
00:36:17,645 --> 00:36:17,925
Oh, have

759
00:36:17,925 --> 00:36:18,025
Nola Simon: you?

760
00:36:18,135 --> 00:36:18,525
Okay.

761
00:36:18,525 --> 00:36:20,855
What do you think about her point?

762
00:36:20,875 --> 00:36:24,265
Her point is that she thinks
that regulation is going to be

763
00:36:24,265 --> 00:36:28,115
so challenging, especially to get
it consistent across the globe.

764
00:36:28,535 --> 00:36:35,065
And really it has to be a financial
incentive for companies to really build it

765
00:36:35,145 --> 00:36:41,805
ethically and Responsibly and if companies
then hire the are used the AI produced

766
00:36:41,805 --> 00:36:47,455
by that company, then they get a reward
for choosing the ethical AI company.

767
00:36:48,215 --> 00:36:49,095
Do you think that's viable?

768
00:36:49,605 --> 00:36:49,975
Hilke Schellmann: Yeah.

769
00:36:50,065 --> 00:36:51,855
I don't know if it's
viable because I think.

770
00:36:52,760 --> 00:36:56,190
I do agree that regulation
can't catch them all, right?

771
00:36:56,230 --> 00:36:57,520
Like it's not really going to work out.

772
00:36:57,520 --> 00:37:00,380
It might work for like the European
union, but that doesn't necessarily

773
00:37:00,380 --> 00:37:02,230
transfer to all these other jurisdictions.

774
00:37:02,270 --> 00:37:04,760
So it is really problematic
and it can't catch them all.

775
00:37:05,120 --> 00:37:09,520
So if we rely on industry itself, like
maybe a system where we reward, or

776
00:37:09,580 --> 00:37:14,040
we have some sort of certification we
require we publicly shame companies

777
00:37:14,040 --> 00:37:15,590
who don't use certification.

778
00:37:15,840 --> 00:37:16,950
But the question is like.

779
00:37:17,295 --> 00:37:18,705
does the certification work?

780
00:37:18,705 --> 00:37:22,315
And I would argue that we've seen
in some industries that we have

781
00:37:22,315 --> 00:37:24,895
like thick leaf as certifications.

782
00:37:24,895 --> 00:37:27,605
We've seen this about diamonds
where diamonds come from.

783
00:37:27,805 --> 00:37:30,015
We see this, we call
it greenwashing, right?

784
00:37:30,015 --> 00:37:34,355
That companies like buy a certification
that they look like a green, organic,

785
00:37:34,605 --> 00:37:37,925
super environmentally friendly
company, but then they dump their

786
00:37:37,925 --> 00:37:40,915
trash in the landfill and they
actually don't recycle, whatever

787
00:37:40,915 --> 00:37:42,305
their promise it's actually not true.

788
00:37:42,305 --> 00:37:43,315
They're just buying.

789
00:37:43,595 --> 00:37:44,275
The certification.

790
00:37:44,935 --> 00:37:49,965
So I am worried about that too, that we
then have another level of that looks

791
00:37:49,965 --> 00:37:54,995
like this is certified and this is a
good algorithm, then no one looks at it

792
00:37:55,015 --> 00:37:56,965
and knows is this actually good or not?

793
00:37:57,005 --> 00:38:01,245
And the, I wish we had like clear
standards on how to judge algorithms.

794
00:38:01,715 --> 00:38:04,745
They're being developed in the European
Union, but there's still a lot of

795
00:38:04,755 --> 00:38:06,075
we don't know a lot of things here.

796
00:38:06,125 --> 00:38:10,375
So I hope we will have standards
and maybe if we had a little

797
00:38:10,375 --> 00:38:14,965
bit more transparency and demand
accountability and explainability.

798
00:38:15,185 --> 00:38:18,095
A lot of times when I do some of
these tests, like talking German to

799
00:38:18,095 --> 00:38:21,935
the tool and then, I get a, I got a
six out of nine English proficiency

800
00:38:22,145 --> 00:38:23,375
and I talk to the developers.

801
00:38:23,465 --> 00:38:25,565
Oh, before we publish, we always go back.

802
00:38:25,955 --> 00:38:26,195
Nola Simon: Yeah.

803
00:38:26,755 --> 00:38:29,665
Hilke Schellmann: Ask for comment and
also maybe we made a mistake, right?

804
00:38:29,685 --> 00:38:30,915
Like we want to know.

805
00:38:31,175 --> 00:38:32,695
So we go back to the developers.

806
00:38:32,765 --> 00:38:35,345
And I was just, I was
listening and listening.

807
00:38:35,345 --> 00:38:36,105
I was like, wow.

808
00:38:36,105 --> 00:38:39,825
He was like, it's maybe because German
and English is in this 5D space.

809
00:38:40,370 --> 00:38:41,490
The languages overlap.

810
00:38:41,500 --> 00:38:44,950
And I was like, 5D, I was like,
I'm really not understanding.

811
00:38:44,950 --> 00:38:46,130
I was like, I can't follow.

812
00:38:46,180 --> 00:38:48,890
I was like, can you just explain to
me if you were in front of a judge,

813
00:38:49,300 --> 00:38:51,710
how, why I was scored six out of nine?

814
00:38:51,760 --> 00:38:54,510
Can you just explain how you
would explain in front of a judge?

815
00:38:54,940 --> 00:38:55,940
And he couldn't.

816
00:38:56,010 --> 00:38:59,340
And I was like, I, 5D
doesn't mean anything to me.

817
00:38:59,340 --> 00:39:00,420
It seems very complex.

818
00:39:01,030 --> 00:39:02,170
Can you just explain it?

819
00:39:02,500 --> 00:39:04,220
Like how the score came together.

820
00:39:04,550 --> 00:39:07,630
And I think if people can't do
that tells us something like if you

821
00:39:07,630 --> 00:39:12,890
can't explain why this person was
scored this way, we have a problem.

822
00:39:13,020 --> 00:39:16,220
So I think we need much more
explainability, transparency.

823
00:39:16,530 --> 00:39:20,050
If he could get that through government
regulators, that would be great if he

824
00:39:20,050 --> 00:39:24,740
could have like benchmarks standards,
we've now seen in New York, there is

825
00:39:24,740 --> 00:39:29,740
a law that companies who use AI and
hiring or algorithms and hiring they

826
00:39:29,740 --> 00:39:31,360
do have to get an audit every year.

827
00:39:31,670 --> 00:39:34,380
It's up to the companies
to actually decide how much

828
00:39:34,390 --> 00:39:35,730
AI and algorithms they use.

829
00:39:35,730 --> 00:39:36,500
So that's flawed.

830
00:39:36,880 --> 00:39:37,780
And then the audits.

831
00:39:38,335 --> 00:39:42,775
It's also unclear necessarily it's like
usually follows a lot of people follow the

832
00:39:42,775 --> 00:39:44,795
four fifth rule which is actually already.

833
00:39:45,295 --> 00:39:49,535
a guideline by the Equal Employment
Opportunity Commission in, in, in the US.

834
00:39:49,545 --> 00:39:52,805
So it's like just auditing
what's already being audited.

835
00:39:52,805 --> 00:39:55,615
And we know that those kinds
of audits are really flawed.

836
00:39:55,655 --> 00:39:58,965
They just look at different
genders, like male and females and

837
00:39:58,975 --> 00:40:00,415
different sexes against each other.

838
00:40:00,415 --> 00:40:04,585
They don't look at the intersection
of Black, Black women versus

839
00:40:04,585 --> 00:40:06,465
Caucasian men, for example.

840
00:40:06,465 --> 00:40:11,095
We know at those intersections where
marginalized identities crossover That's

841
00:40:11,095 --> 00:40:14,595
where I see most of the problems in
the system because marginalized groups

842
00:40:14,595 --> 00:40:19,065
are underrepresented in the tool and in
the training data, often not the tools.

843
00:40:19,965 --> 00:40:22,405
Then they become underrepresented
in the tools and the models.

844
00:40:22,745 --> 00:40:25,445
And then we have the
the problem multiplies.

845
00:40:25,775 --> 00:40:26,715
And so we don't look at that.

846
00:40:26,755 --> 00:40:28,125
There's no mandate to look at that.

847
00:40:28,235 --> 00:40:31,645
And we've actually, when somebody
looked at that, we found problems.

848
00:40:32,195 --> 00:40:33,765
I'm hopeful ish.

849
00:40:34,885 --> 00:40:35,695
That this will help.

850
00:40:36,235 --> 00:40:36,765
A little bit.

851
00:40:37,085 --> 00:40:38,935
Nola Simon: Let's talk
about the whisper network.

852
00:40:38,935 --> 00:40:42,035
So you just published an
article in the Associated Press.

853
00:40:42,035 --> 00:40:43,035
I read it this morning.

854
00:40:43,515 --> 00:40:48,045
And it's basically about how
you found problems in terms of a

855
00:40:48,045 --> 00:40:50,815
transcription having a hallucination.

856
00:40:50,835 --> 00:40:55,175
And honestly, this is 1 of the
use cases that I use AI for often

857
00:40:55,175 --> 00:40:58,395
when I do podcast, I'll take the
transcript and I'll just put it into.

858
00:40:58,700 --> 00:40:59,370
perplexity.

859
00:40:59,370 --> 00:41:00,080
I prefer that.

860
00:41:00,490 --> 00:41:03,840
And it spits out my show
notes and I read it briefly.

861
00:41:03,960 --> 00:41:08,095
And usually it's I don't
really have a concern about it.

862
00:41:08,095 --> 00:41:10,385
Also it's, that doesn't
have a lot of effect.

863
00:41:10,435 --> 00:41:13,155
Couldn't tell you anybody who's
ever called back to me saying you

864
00:41:13,155 --> 00:41:14,455
wrote really great show notes.

865
00:41:14,455 --> 00:41:16,565
So I don't spend a lot of time on it.

866
00:41:16,615 --> 00:41:20,615
But the whisper network is actually
being used in hospitals and this is

867
00:41:20,625 --> 00:41:25,585
where it can be very interesting in terms
of transcripts for medical use that.

868
00:41:26,280 --> 00:41:29,030
Are then actually they're
deleting the audio.

869
00:41:29,220 --> 00:41:32,860
And so the only record that
remains is this transcript.

870
00:41:33,230 --> 00:41:38,920
And you found in your test that there are
hallucinations and the transcripts are

871
00:41:38,920 --> 00:41:41,140
including text that actually was never

872
00:41:41,530 --> 00:41:41,660
Hilke Schellmann: said.

873
00:41:42,280 --> 00:41:45,410
Just chat GPT, but this is
a transcription tool, right?

874
00:41:45,420 --> 00:41:46,620
Like a benign.

875
00:41:47,025 --> 00:41:50,395
Tool that companies use every
day people use every day.

876
00:41:50,395 --> 00:41:51,605
It's very ubiquitous.

877
00:41:52,535 --> 00:41:55,695
So let me take one step back and tell
you how we got to this because it

878
00:41:55,695 --> 00:41:59,895
actually relates to AI and hiring and
it relates to you know that I work with

879
00:41:59,895 --> 00:42:01,415
computer scientists and sociologists.

880
00:42:01,415 --> 00:42:03,565
So I've looked into AI.

881
00:42:03,870 --> 00:42:05,420
one way video interviews, right?

882
00:42:05,430 --> 00:42:08,930
And even regulators have asked me, and
I've wondered about this too what happens

883
00:42:08,930 --> 00:42:13,480
with people who have a speech disability,
or have a dialect, or have who have an

884
00:42:13,480 --> 00:42:15,080
accent, exactly what we talked about.

885
00:42:15,340 --> 00:42:19,240
If they use these systems, or are
being asked to use these systems

886
00:42:19,520 --> 00:42:22,760
the tool then takes the audio
and transcribes it into text.

887
00:42:22,990 --> 00:42:23,870
Is it fair?

888
00:42:24,165 --> 00:42:27,985
To people with different dialects
or accent and speech disabilities.

889
00:42:28,285 --> 00:42:31,965
So I was asked by regulators and others
and I'm like, why are you asking me?

890
00:42:31,965 --> 00:42:32,975
I'm just a lone journalist.

891
00:42:34,505 --> 00:42:36,645
But I was like I guess no
one is looking into this.

892
00:42:36,655 --> 00:42:39,985
So I talked to a computer scientist
and a sociologist and I was like,

893
00:42:39,985 --> 00:42:41,045
I think we should really test this.

894
00:42:41,755 --> 00:42:44,495
So I asked the companies
which tools do you use, right?

895
00:42:44,495 --> 00:42:47,065
Because most companies in the
space don't build their own tools.

896
00:42:47,065 --> 00:42:50,705
You buy one on the market because
there's large companies that use those.

897
00:42:50,715 --> 00:42:55,505
So we took those tools and then, as
we were doing, we found a a database

898
00:42:55,505 --> 00:42:59,435
with audio recordings of people who
have a speech disability and people

899
00:42:59,435 --> 00:43:00,815
who don't have a speech disability.

900
00:43:00,815 --> 00:43:04,325
And we had about, 13, 000
recordings as a whole.

901
00:43:04,695 --> 00:43:07,385
And we ran it through these
transcription software.

902
00:43:07,385 --> 00:43:10,495
And as we were starting to do the
experiment, we also heard about open

903
00:43:10,495 --> 00:43:14,055
AI snooze, new whisper tool, which
everyone was like raving about.

904
00:43:14,055 --> 00:43:16,375
And millions of people were
already using it's open source.

905
00:43:16,375 --> 00:43:17,595
It's very cheap.

906
00:43:17,875 --> 00:43:22,165
It has a very high accuracy rate
that everyone was raving about.

907
00:43:22,165 --> 00:43:25,379
It can also use it can translate between
different languages and be like, okay,

908
00:43:25,379 --> 00:43:28,872
this becomes, seems to be like one of
the market leaders, we should also.

909
00:43:28,873 --> 00:43:33,833
And when we looked at the scatter plots,
we did see that for Whisper, like a

910
00:43:33,833 --> 00:43:37,933
lot of, there were very low error rates
and some, but there were some plots

911
00:43:37,953 --> 00:43:40,263
that had like ginormous error rates.

912
00:43:40,263 --> 00:43:42,903
And we were like, how do you get
a thousand percent error rate?

913
00:43:42,923 --> 00:43:47,343
Like everything must be wrong in these
transcripts compared to the ground truth.

914
00:43:47,733 --> 00:43:49,352
So we did a human.

915
00:43:50,022 --> 00:43:52,692
Intelligence investigation and
looked at these two because you

916
00:43:52,692 --> 00:43:56,672
know we were like did we upload a
wrong clip like what happened here.

917
00:43:56,942 --> 00:44:00,052
And when we looked into it we saw
that it like in some instances

918
00:44:00,052 --> 00:44:04,602
just hallucinated whole stories
like made up whole paragraphs that

919
00:44:04,602 --> 00:44:06,022
weren't there in the first place.

920
00:44:06,372 --> 00:44:10,122
And when we did a close analysis
like it also had facial commentary,

921
00:44:10,123 --> 00:44:14,952
it, it had hallucinated a new
medication, like all these new

922
00:44:14,982 --> 00:44:16,842
things that like, were never there.

923
00:44:17,212 --> 00:44:19,212
A medication that didn't actually exist.

924
00:44:19,942 --> 00:44:22,812
Yes, it was, wasn't it psychoactivated?

925
00:44:23,012 --> 00:44:24,602
Yeah, it was a hyper, hyperactivated.

926
00:44:25,867 --> 00:44:26,837
Antibiotics.

927
00:44:26,838 --> 00:44:30,138
It also like hallucinated violence
and all kinds of things, because, it

928
00:44:30,138 --> 00:44:31,678
learned from everything on the internet.

929
00:44:31,698 --> 00:44:34,058
So like the stuff that we put
on the internet and in books

930
00:44:34,058 --> 00:44:35,898
and stuff, it learns from that.

931
00:44:36,258 --> 00:44:38,508
But we hadn't seen this in
a transcription software.

932
00:44:38,628 --> 00:44:41,838
And when I started looking at GitHub,
where a lot of like open source developers

933
00:44:41,938 --> 00:44:46,158
congregate, there were just like so
many entries of what is happening?

934
00:44:46,158 --> 00:44:47,118
What is this tool doing?

935
00:44:47,118 --> 00:44:48,898
I was like, Okay, we're not the only one.

936
00:44:48,898 --> 00:44:51,998
So it wasn't like a problem
that only we encountered.

937
00:44:51,998 --> 00:44:53,468
This is like much, much bigger.

938
00:44:53,758 --> 00:44:56,898
And then I found this company
called Nabla that does we see this

939
00:44:56,898 --> 00:44:58,478
a lot now, medical transcription.

940
00:44:58,478 --> 00:45:01,878
So where doctor patient
conversations are being recorded.

941
00:45:01,898 --> 00:45:05,552
And then a system conscribes
it and summarizes and you.

942
00:45:05,553 --> 00:45:06,733
Does these doctor notes, right?

943
00:45:06,733 --> 00:45:09,453
So your doctor can spend more time
with you instead of documenting

944
00:45:09,453 --> 00:45:10,903
everything and taking notes.

945
00:45:11,223 --> 00:45:16,043
It sounds like a great idea, but some
companies, I use whisper and not the

946
00:45:16,393 --> 00:45:20,223
only one that uses whisper, but it is
the, I think the, the only one that

947
00:45:20,223 --> 00:45:24,903
I talked to that, so they work with
about at least 30, 000 or so medical

948
00:45:24,903 --> 00:45:26,633
providers in the U S and large.

949
00:45:27,703 --> 00:45:29,173
systems and hospital systems.

950
00:45:29,593 --> 00:45:33,053
And, they built their own tool
on top of the whisper interface.

951
00:45:33,413 --> 00:45:39,303
And it summarizes the recordings
and what the tool also does, they,

952
00:45:39,353 --> 00:45:42,973
it, because of privacy concerns, it
throws away the underlying recording.

953
00:45:42,973 --> 00:45:47,778
So if I am a doctor, I look at my notes
and I'm like, Was that really sad?

954
00:45:48,138 --> 00:45:52,018
I can go back to the transcript, but
the transcript might not be accurate

955
00:45:52,038 --> 00:45:54,998
because there might be hallucinations
in there, and I can't go back to

956
00:45:55,038 --> 00:45:56,568
the actual recording to check.

957
00:45:56,883 --> 00:46:00,263
Was that actually said did anyone
talk about antibiotics or, whatever

958
00:46:00,553 --> 00:46:03,493
the hallucination might be.

959
00:46:03,833 --> 00:46:08,413
And, to a lot of people who look at
AI, this was really problematic that

960
00:46:08,413 --> 00:46:11,963
a company was using that, not, and
I also asked them, I was like, we

961
00:46:11,963 --> 00:46:14,683
know that whisper hallucinates and
they're like, oh yeah, we know too.

962
00:46:14,683 --> 00:46:15,663
We try to mitigate it.

963
00:46:15,673 --> 00:46:16,453
And I was like, oh.

964
00:46:16,773 --> 00:46:19,243
Were you able to get rid of it 100
percent because I've never heard of

965
00:46:19,253 --> 00:46:22,353
anybody being able to mitigate it
100 percent and they were like, no,

966
00:46:22,663 --> 00:46:23,983
but I think we have it under control.

967
00:46:24,043 --> 00:46:26,206
And I was like, okay So
what are the, what are

968
00:46:26,206 --> 00:46:29,393
Nola Simon: the impacts of having a
medical transcript that is inaccurate

969
00:46:29,393 --> 00:46:30,943
and includes a hallucination?

970
00:46:30,953 --> 00:46:33,833
So that could be, it could go to court.

971
00:46:34,673 --> 00:46:42,033
It can be used as evidence to actually
deny coverage for insurance claims.

972
00:46:42,393 --> 00:46:45,453
Hilke Schellmann: Yeah, it also
is in your presumably in your

973
00:46:45,483 --> 00:46:47,393
electronic medical records forever.

974
00:46:47,433 --> 00:46:49,973
So you might have so it
could lead to misdiagnosis.

975
00:46:50,378 --> 00:46:53,588
Yeah it could have a
lifelong implications.

976
00:46:53,598 --> 00:46:56,878
You might also get a prescription
for the wrong medication

977
00:46:57,398 --> 00:46:58,948
all kinds of consequences.

978
00:46:58,998 --> 00:47:02,598
Nola Simon: It can even, honestly, with
the changes in law, if it picks up somehow

979
00:47:02,598 --> 00:47:08,888
that you had an illegal abortion, does
that then lead to legal consequences?

980
00:47:09,523 --> 00:47:09,893
Hilke Schellmann: Yeah,

981
00:47:09,893 --> 00:47:13,362
Nola Simon: if that's not
something that haven't happened.

982
00:47:14,202 --> 00:47:16,702
Hilke Schellmann: Yeah, and I think
that's, it could be that it hallucinates

983
00:47:16,712 --> 00:47:21,072
something, but it could also be like what
happens if the transcript is actually

984
00:47:21,242 --> 00:47:25,152
accurate and the transcript is used for
more, it's not protected by HIPAA, right?

985
00:47:25,172 --> 00:47:29,932
Which is the privacy loss in
the United States because that.

986
00:47:30,372 --> 00:47:33,872
mandates that medical information
stays private between a patient

987
00:47:33,872 --> 00:47:37,202
and a doctor, but it's also now
shared with other companies.

988
00:47:37,232 --> 00:47:40,072
And what we've seen is like now
patients are asked to sign a

989
00:47:40,072 --> 00:47:42,232
release to actually sign away.

990
00:47:42,592 --> 00:47:46,902
Their HIPAA rights that, says I
acknowledge that the transcripts or the

991
00:47:46,912 --> 00:47:51,592
end of recording sometimes is being shared
with some large tech companies, these

992
00:47:51,602 --> 00:47:55,632
providers and, that does worry me too,
that when we go to the doctor, we get

993
00:47:55,632 --> 00:47:58,812
this tablet where you just sign things
and, just sign here, you don't even read

994
00:47:58,812 --> 00:48:03,372
it, and people aren't even aware of the
potential consequences that these very

995
00:48:03,372 --> 00:48:07,032
personal I think a lot of things that
we tell our doctor, we might not want

996
00:48:07,032 --> 00:48:11,232
to tell others very private information
gets shared with large tech companies.

997
00:48:11,232 --> 00:48:14,912
Then they use it against for for
training data and other things.

998
00:48:14,912 --> 00:48:19,032
And who knows who uses it, who is being
sold to the next time what happens if

999
00:48:19,032 --> 00:48:23,832
a company goes bankrupt, like that is
it's usually very valuable material

1000
00:48:23,832 --> 00:48:25,362
that then the next company buys.

1001
00:48:25,392 --> 00:48:26,912
And, yeah, or even just

1002
00:48:26,942 --> 00:48:29,632
Nola Simon: like you run the test
and go, tell me what you know about

1003
00:48:29,632 --> 00:48:34,022
this person and all of a sudden your
private medical records show up in

1004
00:48:34,032 --> 00:48:38,522
like a general inquiry that anybody
in the world could actually ask.

1005
00:48:39,032 --> 00:48:42,082
Hilke Schellmann: Yeah, and, I
find that, they're like anonymized,

1006
00:48:42,112 --> 00:48:46,362
but we also know from like
anonymized data that they can be.

1007
00:48:46,418 --> 00:48:48,483
Pretty easily right there.

1008
00:48:48,833 --> 00:48:50,983
There's a lot of data on us out there.

1009
00:48:51,223 --> 00:48:52,943
So it does worry me.

1010
00:48:52,943 --> 00:48:56,853
And I know of companies use this
kind of medical data to then predict

1011
00:48:57,483 --> 00:48:58,913
this is what happens to people.

1012
00:48:58,923 --> 00:49:00,003
They need a therapist.

1013
00:49:00,013 --> 00:49:01,223
They need back surgery.

1014
00:49:01,423 --> 00:49:02,623
We already know it's not.

1015
00:49:03,133 --> 00:49:04,713
100 percent predicted protected.

1016
00:49:04,713 --> 00:49:07,923
I'm not saying it's the specific
transcript, but there is like data

1017
00:49:07,923 --> 00:49:12,473
lakes of medical data on Americans and
other people out there that is already

1018
00:49:12,473 --> 00:49:16,613
used for kind of predictions and and
things that I think people would find

1019
00:49:16,613 --> 00:49:21,273
very creepy if they know why they're
being suggested that everyone has a bad

1020
00:49:21,273 --> 00:49:22,893
day and they should see a therapist.

1021
00:49:23,393 --> 00:49:27,893
I might, When my company sends me a nudge
like that, I might be like, Oh, yeah,

1022
00:49:27,893 --> 00:49:32,413
maybe I'm having a bad day if I would
know that the company has derived this

1023
00:49:32,413 --> 00:49:37,773
information because I through my spouse of
my medical benefits, and that might be a

1024
00:49:37,773 --> 00:49:39,903
signal of divorce and I need a therapist.

1025
00:49:40,203 --> 00:49:42,633
I might feel very different that
my company shares that kind of

1026
00:49:42,633 --> 00:49:44,943
information with a third party provider.

1027
00:49:45,408 --> 00:49:47,278
Nola Simon: Yeah, it's so concerning.

1028
00:49:47,838 --> 00:49:50,458
Hilke Schellmann: Yeah, that's
probably going to be the subject of

1029
00:49:50,468 --> 00:49:52,228
my next book is AI and health care.

1030
00:49:52,858 --> 00:49:57,098
Yeah, that's fascinating because
we see a lot of AI also moving

1031
00:49:57,098 --> 00:49:58,158
into the workplace, right?

1032
00:49:58,158 --> 00:50:00,028
That can find out like, are we depressed?

1033
00:50:00,048 --> 00:50:00,938
Are we anxious?

1034
00:50:00,938 --> 00:50:03,958
And I think there is something
that like, if you share this with

1035
00:50:03,958 --> 00:50:06,888
your doctor, I hope gets protected.

1036
00:50:06,888 --> 00:50:10,488
We just talked about that the protection
isn't ubiquitous, but but also What

1037
00:50:10,498 --> 00:50:13,848
happens in the workplace with like, when
we have these kinds of tools are they

1038
00:50:14,078 --> 00:50:16,448
used only for the benefit of employees?

1039
00:50:16,488 --> 00:50:18,278
Are they also being used against them?

1040
00:50:18,308 --> 00:50:18,368
That was

1041
00:50:18,558 --> 00:50:21,258
Nola Simon: one of the questions in your
book that I found the most fascinating,

1042
00:50:21,298 --> 00:50:24,918
because it goes back initially to my
interest in the eightfold, which was,

1043
00:50:25,118 --> 00:50:30,533
if you identify transferable skills
that are valuable, but that Employee

1044
00:50:30,543 --> 00:50:34,673
doesn't necessarily want to use that
skill because they find it draining or

1045
00:50:34,673 --> 00:50:39,403
they have trauma associated with, that
work in the past or whatever reason

1046
00:50:39,403 --> 00:50:40,653
they don't want to do it anymore.

1047
00:50:41,143 --> 00:50:44,373
What are the repercussions for
turning down what the companies

1048
00:50:44,393 --> 00:50:46,203
perceives to be an opportunity?

1049
00:50:46,553 --> 00:50:46,923
Yeah.

1050
00:50:47,133 --> 00:50:48,493
To benefit from your skill.

1051
00:50:49,903 --> 00:50:50,273
Hilke Schellmann: Yeah.

1052
00:50:50,483 --> 00:50:53,873
And I think also what I thought is
interesting about eightfold is like,

1053
00:50:53,873 --> 00:50:57,763
it has these like career laddering
that people in your position did

1054
00:50:57,763 --> 00:51:01,033
this, they became vice president five
years because they did X, Y and Z.

1055
00:51:01,033 --> 00:51:04,483
Here's some, LinkedIn learning or
whatever learning platform that

1056
00:51:04,483 --> 00:51:06,063
you can use to get those skills.

1057
00:51:06,973 --> 00:51:07,933
And that's interesting.

1058
00:51:07,943 --> 00:51:11,833
So but it could also be used for
managers to find like high performers

1059
00:51:12,153 --> 00:51:13,913
and it could also be used to find.

1060
00:51:15,088 --> 00:51:18,418
Maybe what we might see from the
platform is low performance right

1061
00:51:18,418 --> 00:51:21,448
people who didn't advance in five years
to vice president who took longer.

1062
00:51:22,338 --> 00:51:25,998
But the platform doesn't tell you
why that is are they slow learners or

1063
00:51:25,998 --> 00:51:29,348
did they have difficult pregnancy or
did they have to take care of their

1064
00:51:29,348 --> 00:51:31,138
parents and you know didn't want.

1065
00:51:31,138 --> 00:51:33,018
Yeah, there's multiple
reasons why you might.

1066
00:51:33,568 --> 00:51:34,048
And like.

1067
00:51:34,593 --> 00:51:36,633
When I asked them that question,
I was like it could also

1068
00:51:36,633 --> 00:51:38,473
be used to penalize people.

1069
00:51:38,473 --> 00:51:39,993
They're like everyone knows about it.

1070
00:51:40,003 --> 00:51:42,983
The employer knows about it,
about our assessment of them

1071
00:51:43,033 --> 00:51:45,203
and the employee knows as well.

1072
00:51:45,203 --> 00:51:49,273
And the employer you could question
it, but I don't think it uses it.

1073
00:51:49,633 --> 00:51:52,183
Nola Simon: If you're in a state
that has at will employment and

1074
00:51:52,533 --> 00:51:55,583
you don't like the answer to that
particular question, then you could

1075
00:51:55,583 --> 00:51:57,263
also use it as a reason to terminate.

1076
00:51:58,473 --> 00:51:59,413
Hilke Schellmann: Yeah, totally.

1077
00:51:59,433 --> 00:52:00,033
Exactly.

1078
00:52:00,073 --> 00:52:00,593
Exactly.

1079
00:52:00,593 --> 00:52:02,653
And I think there, there isn't
a whole lot of protection.

1080
00:52:02,653 --> 00:52:06,443
It could also be that maybe you used
another learning platform, right?

1081
00:52:06,473 --> 00:52:10,363
To have new skills and that get
never registered in the company's

1082
00:52:11,033 --> 00:52:13,793
system, which uses maybe another
event or you read a book.

1083
00:52:15,613 --> 00:52:18,693
Yes, and that never is part
of the system either, right?

1084
00:52:18,693 --> 00:52:21,933
Like we can think of so many use
cases that are actually not part of

1085
00:52:21,933 --> 00:52:24,163
the what a company collects on us.

1086
00:52:24,583 --> 00:52:28,143
And there's no way to like manually
entry that and be like, no, but I

1087
00:52:28,143 --> 00:52:29,883
did get these skills somewhere else.

1088
00:52:30,213 --> 00:52:34,323
And we don't necessarily know how
company executives or managers

1089
00:52:34,403 --> 00:52:35,743
actually use these systems.

1090
00:52:36,063 --> 00:52:39,323
And I think that opens it
up for some doubtful things.

1091
00:52:39,323 --> 00:52:42,503
And we've seen this, unfortunately,
again and again, that companies

1092
00:52:42,503 --> 00:52:46,353
like collect data for one thing,
and then they want to analyze it.

1093
00:52:46,443 --> 00:52:49,733
And it's actually not then their
layoffs, they want to look at the data

1094
00:52:49,763 --> 00:52:51,633
that they collected for one thing.

1095
00:52:51,673 --> 00:52:54,943
It's actually not used for layoffs,
but it's not supposed to be used

1096
00:52:54,943 --> 00:52:57,383
for layoffs, but they still use it
because they have the data, right?

1097
00:52:57,583 --> 00:52:59,093
Once you have it, I know it.

1098
00:53:00,403 --> 00:53:01,203
Exactly.

1099
00:53:01,393 --> 00:53:02,993
And I think that's another problem.

1100
00:53:02,993 --> 00:53:06,223
Once you have the data, once you do
the analysis, like if I think that

1101
00:53:06,253 --> 00:53:09,353
you are at flight risk, meaning
you're going to leave the company,

1102
00:53:09,373 --> 00:53:12,603
the, there are these indicators that
say this person is 80 percent likely

1103
00:53:12,983 --> 00:53:14,833
to leave the company in a year.

1104
00:53:14,843 --> 00:53:18,133
Am I in a manager and I want to keep
you maybe I'll give you a promotion.

1105
00:53:19,123 --> 00:53:20,053
Did you deserve it?

1106
00:53:20,083 --> 00:53:20,623
I don't know.

1107
00:53:20,623 --> 00:53:23,863
Probably somebody else is not gonna
get a promotion or a raise because

1108
00:53:23,863 --> 00:53:27,583
there's only a limit, a limited pod,
and I only do that based on this.

1109
00:53:28,088 --> 00:53:29,558
flight risk indicator.

1110
00:53:29,618 --> 00:53:30,618
How can I unsee that?

1111
00:53:30,628 --> 00:53:33,658
Am I going to put you forward
for leadership training if you

1112
00:53:33,668 --> 00:53:35,088
think you're already one way out?

1113
00:53:35,168 --> 00:53:35,898
I don't know.

1114
00:53:36,138 --> 00:53:39,588
But the question is you don't even
know that the tool predicted that

1115
00:53:39,598 --> 00:53:42,528
you're with one foot out the door
and you may or may not be, right?

1116
00:53:42,548 --> 00:53:43,138
Like it's right.

1117
00:53:43,168 --> 00:53:43,638
Exactly.

1118
00:53:43,638 --> 00:53:45,598
Nola Simon: There might be
something keeping you there like

1119
00:53:45,598 --> 00:53:48,488
a personal relationship that isn't
going to show up on any tool.

1120
00:53:48,768 --> 00:53:49,068
Hilke Schellmann: Yeah.

1121
00:53:49,118 --> 00:53:51,858
Nola Simon: You follow the
advice to have friends at work.

1122
00:53:52,123 --> 00:53:52,813
I

1123
00:53:53,093 --> 00:53:56,183
Hilke Schellmann: mean, there are
so many things connected to this.

1124
00:53:56,203 --> 00:53:57,553
It's just, it's wild.

1125
00:53:57,553 --> 00:53:57,883
So

1126
00:53:57,923 --> 00:54:00,533
Nola Simon: I wanted to make sure
that we were going back to the book.

1127
00:54:00,543 --> 00:54:02,243
You've fabulous book.

1128
00:54:02,243 --> 00:54:04,413
I highly recommend it
really got a great job.

1129
00:54:04,413 --> 00:54:04,873
Thank you.

1130
00:54:05,273 --> 00:54:08,073
What are the successes that
you've had with the book that

1131
00:54:08,413 --> 00:54:09,723
you really want to highlight?

1132
00:54:09,723 --> 00:54:12,353
Because it's done so well.

1133
00:54:13,038 --> 00:54:14,908
Hilke Schellmann: Yeah, I got
really lucky that I think, it was

1134
00:54:14,908 --> 00:54:18,138
like a, it was like a a topic that
is really in the zeitgeist, right?

1135
00:54:18,158 --> 00:54:21,638
And I think a lot of job seekers
also feel like I've heard of

1136
00:54:21,638 --> 00:54:23,378
algorithms, how do they really work?

1137
00:54:23,378 --> 00:54:26,838
And do you think that people in
talent acquisitions Are also like

1138
00:54:26,848 --> 00:54:28,318
thinking do these tools work?

1139
00:54:28,318 --> 00:54:30,078
And there may have like similar questions.

1140
00:54:30,088 --> 00:54:32,508
So I think it was also
really interesting for them.

1141
00:54:32,508 --> 00:54:37,128
So I've been lucky to talk in actually
like HR, talent acquisition conferences.

1142
00:54:37,128 --> 00:54:38,668
And I'm very grateful for that.

1143
00:54:38,668 --> 00:54:41,528
Cause I think those are the people
that I actually have maybe more

1144
00:54:41,528 --> 00:54:44,648
than job seekers, actually maybe the
power to actually investigate these

1145
00:54:44,648 --> 00:54:49,228
tools, maybe make changes, I've also
spoken at data science and technology

1146
00:54:49,228 --> 00:54:53,838
companies for the people who build
these tools to be like much more.

1147
00:54:55,583 --> 00:54:59,593
thoughtful and think through and talk
to different people and not just assume

1148
00:54:59,593 --> 00:55:02,553
that they're domain experts about hiring.

1149
00:55:02,563 --> 00:55:06,143
And really, they have no clue how to
really hire and what are their best case

1150
00:55:06,143 --> 00:55:08,583
scenarios and how to not discriminate.

1151
00:55:08,803 --> 00:55:11,533
So I also tell them like, here's
how, what you should avoid.

1152
00:55:11,553 --> 00:55:12,933
And here's exactly how you avoid it.

1153
00:55:12,933 --> 00:55:14,223
Don't do this, don't do that.

1154
00:55:14,583 --> 00:55:18,753
So I'm really grateful that I feel like
people who built the technology and use

1155
00:55:18,753 --> 00:55:20,333
the technology are actually listening.

1156
00:55:21,128 --> 00:55:24,628
And then I love just connecting
with like readers from all over the

1157
00:55:24,628 --> 00:55:28,148
world, it's amazing when you get a
message from like Brazil and the next

1158
00:55:28,148 --> 00:55:31,208
day from Sweden, somebody read your
book was really meaningful to them.

1159
00:55:31,218 --> 00:55:33,748
Like I was just like, very honored.

1160
00:55:33,858 --> 00:55:36,828
And then recently I gave a
TED talk about the subject.

1161
00:55:36,878 --> 00:55:37,698
Oh, I listened to the podcast.

1162
00:55:38,028 --> 00:55:38,388
Yeah.

1163
00:55:38,843 --> 00:55:41,193
And getting new finding new audiences.

1164
00:55:41,203 --> 00:55:43,393
So hopefully, it's also
reached some lawmakers.

1165
00:55:43,403 --> 00:55:44,823
So hopefully there will be change.

1166
00:55:44,823 --> 00:55:47,503
And I'm just like, so grateful
for people who pick up the book,

1167
00:55:47,503 --> 00:55:48,733
spend so much time with it.

1168
00:55:48,763 --> 00:55:52,363
And I hope to illuminate a new world
that maybe they didn't know about.

1169
00:55:52,363 --> 00:55:57,013
And it's endlessly fascinating to
me how we quantify human beings.

1170
00:55:57,123 --> 00:56:01,093
So I, I hope that like rubs off on
people too, that it is interesting, like

1171
00:56:01,093 --> 00:56:02,813
how we try to make sense of the world,

1172
00:56:03,313 --> 00:56:04,963
In in, in different ways.

1173
00:56:05,308 --> 00:56:06,608
Nola Simon: Yeah, no, it's fascinating.

1174
00:56:06,718 --> 00:56:09,738
And it validated me because I'm like,
I don't have a journalistic background

1175
00:56:09,738 --> 00:56:13,948
at all, but I went back and I'm like,
Oh, look, I was hearing really a lot of

1176
00:56:13,948 --> 00:56:15,668
the similar work and I'm like, exactly.

1177
00:56:15,748 --> 00:56:16,738
And I think that's what I want to do.

1178
00:56:17,288 --> 00:56:22,033
journalistic instincts to say,
Hey, I'm noticing these things.

1179
00:56:22,033 --> 00:56:22,423
This is good.

1180
00:56:22,423 --> 00:56:22,813
Yeah.

1181
00:56:22,813 --> 00:56:24,103
Hilke Schellmann: Welcome to our world.

1182
00:56:24,103 --> 00:56:28,463
Like the next book about AI and
hiring and in the world of work.

1183
00:56:28,463 --> 00:56:30,503
But I actually think I always
want to encourage people.

1184
00:56:30,503 --> 00:56:32,373
I tell them like steal my methods.

1185
00:56:32,383 --> 00:56:34,663
Like some of them are super basic.

1186
00:56:34,723 --> 00:56:38,373
Some of them are more elaborate, but
yeah, these methods to scrutinize

1187
00:56:38,373 --> 00:56:42,303
these AI tools, because as we can see,
like technology companies just put

1188
00:56:42,303 --> 00:56:47,068
out these tools and, for example, with
Open AI knew about the hallucinations.

1189
00:56:47,068 --> 00:56:48,388
In fact, they put it in the paper.

1190
00:56:48,438 --> 00:56:50,588
They did say in one of their
disclosures, it shouldn't be

1191
00:56:50,588 --> 00:56:52,938
used for high risk use cases.

1192
00:56:53,708 --> 00:56:55,358
Lots of people don't need the model card.

1193
00:56:55,358 --> 00:56:57,348
So they ignored the
advice from the company.

1194
00:56:57,348 --> 00:57:00,878
They took it from it's not like they
didn't know, but you wonder like,

1195
00:57:00,878 --> 00:57:02,538
why did they release a flop system?

1196
00:57:02,768 --> 00:57:04,828
There's all kinds of questions
that we can ask about this.

1197
00:57:05,148 --> 00:57:05,768
So I feel like.

1198
00:57:06,178 --> 00:57:10,148
We need to be much more
skeptical users, testers of

1199
00:57:10,168 --> 00:57:12,248
these technologies and push back.

1200
00:57:12,528 --> 00:57:17,578
So I want everyone to steal my methods
and do, invent your own ones, test

1201
00:57:17,588 --> 00:57:21,798
these tools push back against the
companies so we build better tools.

1202
00:57:22,168 --> 00:57:25,328
Because you think that AI can
be transformative technology.

1203
00:57:25,498 --> 00:57:29,258
The jury is still out if we use it
to predict the future of job seekers.

1204
00:57:29,258 --> 00:57:31,928
It's really a question if we
actually, if AI can help with that.

1205
00:57:32,278 --> 00:57:35,768
But I think it's really helpful
in in, in other use cases.

1206
00:57:35,828 --> 00:57:38,278
But in some maybe we shouldn't
use it and we should push back and

1207
00:57:38,278 --> 00:57:40,348
like highlight how it doesn't work.

1208
00:57:40,348 --> 00:57:41,698
So I think everyone can do the work.

1209
00:57:41,728 --> 00:57:43,648
We don't have enough
journalists to do all this work.

1210
00:57:43,843 --> 00:57:44,123
Nola Simon: Yeah.

1211
00:57:44,153 --> 00:57:46,563
This is what my fourth
podcast episode about it.

1212
00:57:46,573 --> 00:57:46,813
So

1213
00:57:49,503 --> 00:57:50,473
Hilke Schellmann: I'm doing my part.

1214
00:57:51,563 --> 00:57:52,043
Thank you.

1215
00:57:52,043 --> 00:57:52,293
Yeah.

1216
00:57:52,293 --> 00:57:54,843
And if anyone wants to be in
touch with me I'm on LinkedIn.

1217
00:57:54,893 --> 00:57:56,763
You find my email on my pet faculty page.

1218
00:57:56,763 --> 00:57:58,043
There's only one Helga Shellman.

1219
00:57:58,083 --> 00:57:58,793
Please be in touch.

1220
00:57:58,813 --> 00:58:02,753
Like I'm happy to I always love
to hear from job seekers or people

1221
00:58:02,753 --> 00:58:04,163
who work in talent acquisition.

1222
00:58:04,473 --> 00:58:05,753
I love talking to folks.

1223
00:58:05,753 --> 00:58:08,033
It might take a few days to like respond.

1224
00:58:09,873 --> 00:58:11,543
And, I have a bunch of obligations.

1225
00:58:11,913 --> 00:58:14,693
I also have a four year old
kid I like to spend time with.

1226
00:58:14,693 --> 00:58:18,283
And, she demands a scavenger
hunts for sweets with her mom.

1227
00:58:18,783 --> 00:58:21,513
But, I love to be in touch with
people and I, I'm so grateful for

1228
00:58:21,513 --> 00:58:25,633
everyone who's listening and has wrote
me and, has engaged with the book.

1229
00:58:25,643 --> 00:58:26,523
It's just wonderful.

1230
00:58:26,933 --> 00:58:27,283
Nola Simon: Okay.

1231
00:58:27,323 --> 00:58:27,683
That's good.

1232
00:58:27,883 --> 00:58:30,483
I think make sure that everything
is linked in the show notes.

1233
00:58:30,513 --> 00:58:32,363
And I really appreciate
you making the time.

1234
00:58:32,423 --> 00:58:33,548
So thank you so much.

1235
00:58:34,588 --> 00:58:51,988
I'll see you next time.