Dimitri Boylan
Welcome to another episode of the talent transformation podcast. Today we’re joined by Dr. E.J. Lee, an assistant professor of computer science at the National University of Singapore. Dr. Lee’s research focuses on conversational user interfaces, human AI collaboration and the social impact of artificial intelligence. Dr. Lee, thank you so much for joining us.
EJ Lee
Thank you for having me today.
Dimitri Boylan
Yeah, I appreciate you taking time out of your research into artificial intelligence. We know that AI is moving at such a rapid pace, I can’t imagine how busy you are. But I thought we might start out, since you’re the first academic I’ve had on the podcast, with me describing what AI research is like in the National University of Singapore in general.
And then we could drill down into the things that you particularly have been working on.
AI research in NUS, we do have a different direction, especially under the computer science department, from the fundamental AI-related theory to more practic al research in NUS.
And my research mainly focuses on human-AI interaction or human-computer interaction. The part is like taking care about the human’s perspective when we design the AI system and how the AI system will affect human decisions or even change their attitude.
Dimitri Boylan
So you are particularly focusing on this and have interaction. In what specific way? I mean, and how did you come to do this kind of research? Because you’re a computer scientist, correct?
EJ Lee
Right.
Dimitri Boylan
You’re not a social scientist or a psychologist. What led you to this type of research?
EJ Lee
Right. So, I think this is a very good question. So I’m so interested in human-AI interaction or human-computer interaction. And as you found from my previous work, we do integrate a lot of theory from social science, from psychology, or even from cognitive science, because we believe that, in the future, we’re developing this AI technology and making it integrate into our daily life, right? And how this system or this technology can change humans’ lives positively instead of making trouble for our lives.
So I think that’s why I’m so interested in this area, because for a lot of the computer science research, we focus more on the performance and efficiency of the system. But do those performances really matter for human beings? And that is a big question. Improvement of 1%, does that really change a lot for a human?
But now I think everyone can access the large range model easily by using the live chat GPT or other free services. Then we access the information from this type of the technology and people start to worry about the hallucination issue. So this is the thing that really changed people, really made our society have a different perspective or different troubles. So, that’s why I’m thinking, as a computer scientist, we have the capability to design, to develop this type of the technology to affect people’s lives, then how we should be thinking about its impact and how we can evaluate its impact to make sure in the future, when we design the technology, we can make things in a positive way instead of the negative way.
Dimitri Boylan
Yeah, this is very interesting to me because this is really an expansion of the domain of computer scientists, right? I mean, historically, computer scientists focus on creating new software or new hardware or new algorithms. So, you know, how do you go about doing this? I mean, do you just put a bunch of people in a room and wait and force the machine to hallucinate and figure out what the people do about it? Or how are you studying this?
EJ Lee
So, the research in HCI, we really hired human participants in the study. So of course, we may be interested in some specific topic. So for example, if we’d like to design AI to help people to make that decision, right? If we’d like to hire this person or not, and we have the AI to help us review the research or even summarize their interview.
We probably summarize their interview and we probably know that AI can summarize the things in a good way, but also in a bad way, right? So I think recently there’s news talking about in academia, people have embedded some hidden words in their research paper. If the reviewer used a large range model to help review the paper to make a summary, then the large range model will generate the positive review and make their paper have a higher chance to get accepted by the conference.
Dimitri Boylan
Right.
EJ Lee
You know that large language models could be manipulated in this way. When humans make the decision with AI together, they need to be very, very, very careful to consider the outcome. So, for this type of design, we think we’ll design the AI system, really implement a prototype, and hire the target users.
So, for example, if our research is related to HR and we hire the real HR people to come to our research and say, “Okay, now we have developed a prototype and the system is trying to help you make the decision if this person is a good candidate or not. And based on this information, how would you make that decision?” Then we have multiple different designs and see which design will bias people, will make the better decision, and what kind of impact will be observed through this type of the study and experiment. So we really develop prototypes and see its impact.
Dimitri Boylan
Okay. So you’re essentially building different types of models. And are you also just studying existing models? I mean, you can create a model that has a particular bias, right? And see whether or not a human can detect it. Are you looking at the models that are being created and put out into the commercial market?
EJ Lee
I think we do both. So the existing model we’ll see as a control group, right? Because we would like to see, okay, now without any further change or tuning, what the outcome will be by using this existing model. But we’d like to propose a better way to make the decision. So then we tune this model or even embed some additional module for this existing model. And we’d like to see if this design will make the decision ah more accurate without bias.
Dimitri Boylan
How long have you been doing this? And maybe this is a complicated or a naive question, but how far along are you in sort of understanding how people interact with these things? Are you at the very beginning and just beginning to categorize some of the ways people behave? Or do you really feel like you’ve made a lot of progress in understanding this challenge?
EJ Lee
So, I think now, especially at this moment, because the large language model makes the AI system more accessible for everyone, right? So, I think this kind of change just started. So we are seeing that there are very different impacts in our society.
For example, there are a lot of deep fake videos or videos created by a generative AI, and they share them on YouTube. And some groups of people are really engaging with those types of videos. It’s a very surprising thing from my perspective. Probably I’m too old school. I don’t like to watch that AI-generated content. But we found that a group of people really liked that.
And what’s happened? Because of this… things we probably have for the previous 20 years, we haven’t seen this type of phenomenon or behavior in our society. But now, this type of behavior is just coming up, so we are interested in what happens behind that, right? And why in the current society, people have the motivation to watch this type of video online instead of watching the real person perform.
Dimitri Boylan
Yeah. So, the minute I identify it as a fake, I just flip beyond it. I have no interest in looking at it. I would just kind of assume that everybody is like that, but obviously these things are popping up in the algorithms because people are watching them. So, you start to look at these different groups and how they behave.
Are you looking at all, I mean, the whole world it’s big and there are lots of different types of people in the world. You’re obviously looking at and creating different kinds of models that give different kinds of feedback or input to the human. Are you looking at different demographic groups, different age groups, different classes of society and in various countries? I mean, how specific are you getting about the humans themselves?
EJ Lee
Right. So it’s a good question. These are the things I’m working on. I’d like to use one example. We are working on understanding humans’ stigmatized attitudes toward mental illness. So I have been doing this type of research for a long time, like I design conversational AI to help people improve their mental wellbeing. But we found that there’s a big challenge or a big obstacle for people living with mental illness because they have a strong, same-type attitude toward mental illness. And their society also has this stigma and makes them feel uncomfortable to reach out to the right person to help them deal with their illness.
That’s why we started thinking about whether we can understand stigma properly, and if we can, then it will help people improve. So we designed a conversational AI to do the conversational interview to make this and deployed that in the US and the UK. We collected a bunch of data and analyzed the different structures and concepts from these different cultures and countries.
Now we are expanding this research to some other Asian and European countries to make a close cultural comparison and see what made the difference.
And our recent research also did a comparison between the US and China, how people perceive conversational AI. And our findings are very interesting because we found that people in China see the conversational AI as a companion and they can make friends with the conversational AI. But in the US, people are learning to see conversational AI as a tool to help them solve the problem. So from this perspective, these cultural differences may change how we design the product in the future, right? Because we know that under different cultures, people will see the same technology in different ways. So, this makes us really think about how to design the technology properly for different cultures, different age groups.
Recently, we have also been working with an aging population because aging is a big issue in Singapore or some Asian countries; people are trying to design technology to help all their doubts in their community. So now we are also thinking about designing conversational AI to help older adults.
But we noticed that older adults interact with the conversational AI differently. So compared to the generation that uses the mobile phone a lot, right? So, how to design properly to really help them improve their digital literacy is the way that we are trying to do in the current research and make a comparison between younger generations and older generations.
Dimitri Boylan
Yeah.
EJ Lee
What is their difference?
Dimitri Boylan
Yeah.
EJ Lee
Yeah.
Dimitri Boylan
Isn’t it difficult… And I wonder if you’re looking at this: When you see an image and it’s a deep fake, very often today, the images are just crazy, right? Somebody created some crazy thing, and you look at it, and you’re like, oh, this is obviously, uh, you know, a fake.
With conversational AI, it’s much harder to identify something that is contrived and not real. How do you develop trust in a system and then not get deceived in an even bigger way because you are now starting to trust it, but it is actually gamed to your disadvantage?
Because in a conversation, it’s easier to shift people’s way of thinking than it is sometimes in an image. Because the image sometimes it’s just obvious, today anyway, most of the images that are being created are sort of silly, crazy, you know, like a bird eating an elephant, this kind of stuff that I see. But in a conversational mode, it might be much more difficult to detect that the conversation is being controlled, not to your advantage, I guess, or you understand what I’m saying?
EJ Lee
Right, right. So…
Dimitri Boylan
Manipulation that you’re being manipulated by the conversation.
EJ Lee
Yeah, so I think there are a lot of these types of issues and things in our society. I think we haven’t seen an AI policy to constrain how AI or conversations should be designed. Of course, it can bring a lot of benefits and advantages for our society. But at the same time, we also noticed that there are some potential concerns in the future.
For example, I did research related to how to design conversational AI to improve people’s self-disclosure. So, the purpose for that bot is positive. I mean, it’s for improving people’s mental wellbeing. And surprisingly after like three weeks long, kind of a long-term study, we noticed that people start to disclose a lot to their conversational AI because they tell us in their real life, they don’t have someone who can listen to them almost every day and check out if they feel good, if they are doing good each day, but conversational AI can really do that.
So, when AI is trying to, it’s not even that intelligent. When they are trying to ask them, “How do you feel today? Is there anything you’d like to share with me?” They kind of build up a relationship with the conversational AI and self-disclose themselves. So, of course, in three weeks, they feel their mental well-being enhanced, which is good.
But at the time, when we check the content of what they disclose with the conversational AI, we notice that there is a lot of yeah ah sensitive information that they disclose with the conversational AI. So, of course, this is research for academic research. So of course, we won’t use that data for any purpose, business purpose, right?
Dimitri Boylan
No, of course not, right?
EJ Lee
But we can imagine that if there’s a company you can also develop this type of conversational AI as your companion and check your mood every day.
Dimitri Boylan
It’s going to know a lot about you.
EJ Lee
And this will be very sensitive, right? And also, they can use this data to sell you things.
Dimitri Boylan
Right, right.
EJ Lee
So, yeah.
Dimitri Boylan
Exactly. Yeah. So, let me go specifically to the practical world of HR and HR technology. The human resources organization is responsible for the fundamental relationship between the employee and the organization, and there have been many issues that HR has faced where they have not been able to accomplish their stated goal.
You know, the CEO could come to the HR organization and say, “How do the employees feel about my speech? I gave a speech last week to all the employees, and I’d like to know how they feel.” And the HR organization would talk to one or two or 10 or 15 people, or maybe do a survey. But the survey would be a communication between the employee and the HR organization, right?
The idea of getting to sentiment, how people feel, how are people. Do people believe in the mission of the organization? Do people believe in the future of the company? Are people committed to the organization? Are people thinking about leaving the organization? Are people helpful to each other? Are people not helpful because they’re very competitive and they see their cohorts as people that they need to succeed against, as opposed to succeed with?
Are these the types of things that you’re looking at? I mean, do you think… that the conversational paradigm is going to open up, finally, an understanding of these things at scale?
EJ Lee
Yeah, that is one of the things that I believe. So that’s why I’m starting social stigma, right? Because this is things people don’t really like to talk about. But we developed a conversational AI to chat with people about this topic, because this is kind of not like a one-on-one chat. It’s like a one-person chat with their conversational So sometimes they will feel more comfortable to share themselves with the conversational AI.
And because the conversational AI won’t judge anything, right? So they’re just collecting the data.
These things can give you more information compared to traditional questionnaire surveys, which is why we are interested in this topic. I think, as an example, we’ve just given about the organization: They like to evaluate their employees’ experience. So, I think having this conversation with AI could have the potential to make their employees really disclose themselves and help them figure out what the issues behind things are.
And then there comes another challenge. How can we evaluate this type of data? First, it’s just a survey. So it’s just a 1, 2, a Likert scale, 1 to 5, 1 to 7. It’s easy to evaluate.
Dimitri Boylan
Yes, and we can add it up.
EJ Lee
But for this qualitative data, for this conversational data, how can we really evaluate that, especially if this comes on a big scale? My research lab is now also developing a solution. How can we integrate human experts with AI technology to evaluate this type of like kind of conversational data automatically? So this direction is recently very popular. People are thinking about how we can extract more meaningful information from this conversational data and use this way to help us have more insight from the survey.
Dimitri Boylan
So, how do you do that? I don’t want to get too technical, but you have a lot of conversations, which are between people and the AI. And they may have taken very many different forms, but you now have all of that data. And then you’re using a different AI system to look at all the conversations and try to extract the overall meaning or the overall impact of all of those conversations.
And how accurate do you think you are today if I were to give you a thousand conversations between employees and their AI chatbot about their commitment to the organization? Would you be able to come back and say these employees appear very committed, somewhat committed, or not committed at all? And would that be meaningful data? Or do you think you really couldn’t come to a conclusion?
EJ Lee
So for the conversational data, I think to make some classification, right? If they feel positive, negative, I think i think based on the current NLP technology, you can already achieve really good outcomes, right?
But we are more interested in their logic, their reasons behind that. And I think that is the most challenging part of coding this type of conversational data.
So, traditionally in social science or in some qualitative research area, they manually code this type of data, which means they hire three people looking to these interview conversations and the label the features. “Okay, this is the motivation. This is the reason.”
And it takes forever, right? If you have 1,000 employees, then how can we analyze? That’s why we need to develop a new way to use AI to assist this type of coding. We’d like to know more about this conversation, this interview.
So, in my research lab, the things we are doing is like, we hire the experts to help us coding part of the data. And, based on this information, we fine-tune the large language model. But to make sure the model is not wrong… Okay, this time they coded the data from ten interviews and we come back and see, and get the expert to check if the large-language model really does the work in the right way and if there’s something we need to revise again and again. So we kind of developed this procedure to make sure the outcome really align with humans’ expectations.
But we found some interesting things. AI sometimes has good ideas compared to human experts because, as humans, when we read these words, our personal experience may affect how we interpret the information that we see from this conversation.
And so when they see the AI’s or logic model’s output, they say, “Oh yeah, I think AI probably is good.” Then you know that it’s kind of the coding process, they also affect each other, right? So it’s not just from the human’s perspective, also from large language model perspective.
And I think there are some discussion, or say, a criticism about this, right? Because they will change the way how people are thinking about this evaluation and what is the right way to evaluate that. And I think this is the ongoing thing, right? We haven’t had a good conclusion for this.
Dimitri Boylan
Yeah, so it’s the standard pattern of how we’re deploying AI, in a sense. It’s the same thing that we did with the radiology example, where we had the top 10 radiologists review data and annotate it. The machine was basically as good as the top 10 radiologists, which turned out to be a lot better than the average analyst looking at an X-ray or an output from a medical machine.
So, I suppose you can do a whole lot more analysis, an infinitely large amount of analysis, whereas before you were constrained by the time it took, the number of experts you had to have… And of course, if you start cutting that analysis across generations and cultures, the data just gets really hard to economically analyze.
EJ Lee
Right, right. We are developing technology to help HR, so we’re also thinking about developing conversational AI to help the interview process.
So, recently, there has also been some news. There’s some big company, and they fire a lot of HR from their company, right? because they like to use conversational AI to replace interviewers.
Dimitri Boylan
Yes, yes.
EJ Lee
But we don’t think that is the right way to do this, right? So we like to develop some tools to, I think, human-human communication, especially in HR interviews, is very, very important. That’s a thing I like to emphasize, right? But the key is how we can design the AI tool to help this type of real-time interaction to make sure they can record important information, to make sure HR can use the technology to help them understand their account candidates better. And in this way, can also help the company efficiently to recognize their good candidate for the kind of company they are.
Dimitri Boylan
Yeah, getting the right fit is not just good for the company, it’s good for the candidate. And I think that it’s still a very hard problem to solve. It’s not a solved problem. So, I think with a new tool like AI and large language models, I think companies have the opportunity to do better at it. And I think ultimately for companies, whether they have more people, less people, or the same people, really comes down to how they deploy the technology, which has always been the case.
You can deploy technology to do the same thing with fewer people, or you can deploy the technology and do more things. Some companies will go one way, and for other companies, they’ll expand their products, their services, they’ll expand their market share, and they’ll expand their geographic reach because of the large language models. They’ll maybe operate in countries that they never operated in before.
Humans will continue to interact with these machines the way they interact with other machines. We interact with machines every day all the time, right? I mean, I don’t think there’s a question that humans interact with machines. I think it’s just a machine that seems so different right now.
It may not… For the next generation, it actually won’t feel different at all. We already know that paradigm, right? The kids come up searching on Google, and they assume that everything was always available… That knowledge was always at the fingertips. And for my generation, knowledge was sometimes a guarded secret that you had to pay particular people with expertise to give you. So the paradigm is there already. But definitely new challenges here. And I think it’s really critical that we explore at a very, very deep level, you know, how people interface with these machines.
Dimitri Boylan
So, let’s talk about… I know you’ve been traveling a lot. You’ve been going around to conferences. You’ve been to Japan, the UK, et cetera, presenting your research. What other kinds of research has struck you as particularly interesting that’s in your domain? I mean, are you hearing interesting… What other things are people are doing that you find very interesting that we might find interesting?
EJ Lee
So I think there’s one interaction that’s very popular, it’s like a co-creation. Now we use large language model to write the article, write research paper and even make the art pieces. So, this type of co-creation process is becoming a hot topic. But they also raise a lot of questions, right? When you work with the LLM to create these articles, then how would you think… Who is the owner of the article, right?
Dimitri Boylan
Yeah.
EJ Lee
If this article delivers some misinformation, who should take responsibility for these things? Also, for authorship. When we work with AI together to co-create things, then what is the authenticity of human beings, right?
Dimitri Boylan
Yeah, it’s interesting that large language models are popping up everywhere. I mean, they’re certainly popping up everywhere in our software, and they’ll pop up everywhere in our customers’ use of software. But, the other thing that I’m concerned with when you talk about this co-authoring is eventually, does the skill level…
I mean, I write similar to ChatGPT, but I wonder if the next generation will, if they don’t spend too much time writing, write like ChatGPT or need ChatGPT to do the writing for them? That’s kind of a question.
EJ Lee
Yes. So I think this is a very good question. So that’s why people are thinking that… I think we can think of this differently, right? So, of course, if a person just uses a logic model to generate everything and they don’t read it, they don’t reflect the content, they just copy and paste, then of course they won’t know anything. They won’t learn anything. But, for example, for the second language learner, right?
Dimitri Boylan
Oh, yeah.
EJ Lee
As a non-native English speaker, if they need to write an English research paper, they need to learn from the very beginning, right? And, I think in this case, large language model can really be helpful because they can provide examples for writing and they can help them to express themselves better in the research paper. So, my personal observation is that, now as a reviewer for research papers, I noticed that almost all submissions, their writing is really good. It’s easy to understand.
Then, for reviewing those papers, I can focus on what the real contribution of that paper is instead of focusing more attention on the writing itself. So, I can focus more on the contribution of the self. So I think this is the good a good way to thinking about this technology…
Dimitri Boylan
We’ve seen a huge impact on global companies’ ability to produce their job descriptions in other languages, search through references in different languages, and write search strings that can find expertise. Before, they just had to find somebody who was native to actually do that job.
EJ Lee
Right. So, I think it does provide some good things for our society, but how can we design that properly to work with humans together to prevent over-relying on the AI to replace your skill?
So, this is the thing I worry about because, as faculty members, as a professor in the university, we need to teach the next generation to do research, how to write a good essay, and how to write research papers. And we also worry about whether they just use large language models to do everything, then they will miss that skill.
So, at this moment, I think there are no clear conclusions about how we can do that in the right way. So, I can only say to my students, I say, “Okay, now, it’s your own responsibility to learn these things. If you don’t like to learn this, then in the future, you won’t even have the ability to recognize if AI really generated a good thing for you. You cannot really know what the good quality of the writing is because you all heavily rely on the response from the AI, not yourself.” So yeah, I hope they know learning this is very important.
Dimitri Boylan
Yeah. It’s a fascinating time. It’s really good to know that computer scientists, and in academia, are focusing on some of these social issues. I think it would be very difficult, especially from an experimental perspective, for social scientists to do this alone, right? Because they really don’t have the expertise in the models themselves.
This kind of research is, well, obviously one of the reasons that we’re supporting research at your university, is that this topic is very relevant to us. But I think that the human-machine interface is the place where we’re going to see a huge amount of change over the next couple of years.
We really appreciate your time talking to us here and also the collaboration with your lab and your research. The HR community is profoundly interested in the type of research that you do and we want to stay in touch and I want to help the community get closer to the type of research you’re doing.
So, once again, I think it’s great research. We love it. It’s so key to the success of the future success of human resources and people responsible for building that fundamental relationship and between the company, which is, an entity. It’s kind of hard. It’s not a thing. It’s not very tangible, right? The entity is really a group of people and with objectives that are shared and then the individual person with their objectives.
EJ Lee
Right.
Dimitri Boylan
And the match between those things is subtle and complex. So fascinating research. And once again, thank you so much. I look forward to having you back again in the future.
EJ Lee
Thank you. Thank you for having me.
Go back to podcasts 
