Podcast transcript - Test and Learn: Exploring Technology for a Human-First Approach

Original story was published here on October 26, 2020: https://www.rga.com/futurevision/pov/creative-companies-push-tech-boundaries-podcast (no longer online)

Transcript:

SHIRLEY: [00:00:05] Hi everyone, my name is Shirley Brady, and welcome to R/GA’s FutureVision Conversations podcast, where we explore fresh thinking and new ideas around technology, creativity and business and how all of that can help create a more human future. Today, we’ll be talking about something you may have heard buzzing about: GPT-3 or Generative Pre-Trained Transformer 3. It’s the third generation language prediction model in the GPT-N series created by Open A.I., which is a for-profit San Francisco-based Artificial Intelligence research laboratory.

SHIRLEY: [00:00:39] You can think of it as a powerful autocomplete, but it’s much more than that. And if none of that makes any sense to you, don’t worry. We have two people here today to help explain what GPT-3 is and why our company, R/GA, is not only paying attention to it, but kicking the tires on it. So I’m joined today by Nick Coronges, R/GA’s global chief technology officer, and John Tubert, who’s our SVP and head of technology based here in New York. So welcome to you both and thank you so much for making time today.

SHIRLEY: [00:01:11] I thought we’d start off big picture and really just frame this in, why are we even talking about this today? So as we all know, R/GA is known for exploring technology on behalf of our clients and helping advise and guide companies’ tech investments they could and should make, not just to be pioneers for the sake of being pioneers, but to really help solve problems, foster innovation, and then think about what people need sometimes before they even realize they need it. We call this designing a more human future. It’s right on our home page at rga.com, it’s our value proposition and our purpose. And I just thought we’d start off from that piece. And maybe, Nick, this is a good kind of global macro piece for you, just in terms of how we think about emerging technology before we get into the specifics of GPT-3.

NICK: [00:01:57] Sure. I started at R/GA over 12 years ago, and kind of the backdrop of R/GA is we started when the Internet kind of exploded. There was this new landscape that we all found. We saw that we were able to take a lot of the the kind of linear content forms and sort of static content forms that everybody was used to at the time. And they suddenly became interactive. They became kind of living and breathing. You could instead of just kind of like watch a film linearly, you suddenly had tools that could turn a film into an interactive, almost choose your own adventure experience. And that gave birth to a whole kind of commercial business — now obviously massive and almost the lion’s share of economic growth — but at that time, it was really a shift towards things like direct-to-consumer channels and brand activations and digital. And I think for myself and a lot of the people here, I know John as well, when this first was bubbling up and we weren’t quite sure how it was going to turn into commercial opportunities, we got sucked into it and we felt that there was something there. We felt that there was this opportunity to innovate and bring creativity into it, even though we didn’t know at the time what shape that would take. And there have been sort of breakthroughs since then. There have been certainly mobile and other things of that nature.

NICK: [00:03:26] And I think when GP2–3, which itself is a kind of an incremental kind of evolution of the GPT models that came before it, but when GPT-3 kind of reached the maturity that it did in the last few months, when we started to see the power of this thing, it felt a lot like kind of the early Internet and early sort of sense of possibility. And we’ll do our best to explain kind of why and why we think that this is so exciting for us. And some of it really is that we don’t know yet what the applications are. But when you see this kind of new medium, you begin to see that this is computers able to almost talk back to us. This is where in the early Internet it was, the power was how could we program computers to do things? How could we use code to create these deeper, richer, interactive experiences, these different ways of transacting and different ways of connecting with people and with businesses. With this, you get a glimpse of what it looks like when computers have the ability to kind of play back ideas and have a voice of their own, even if that voice comes from the start of assimilation of lots of other voices and sort of an ingestion of data and conversations that are on the Internet, when you begin to feel like these models are, have a life of their own, that kind of wakes you up to this idea that media is going to change again. The way we design is going to change. The way we write is going to change. The way we interact with brands is going to change. It’s going to affect things like what your role is as a copywriter, what your role is as a programmer, what your role is as a as a strategist, and all that is going to take a little time to unfold. But I think for many of us who have been playing with this stuff, it feels very much inevitable. GPT-3 is, I think it is what it is. And in itself, we’ll talk about what we’re doing in this space. But I think just putting it into context, it’s one of those moments that I think will be a harbinger for this, for kind of a new wave of kind of creation. And it’ll be something where we’re going to see a lot of really cool things coming out over the next year or two in this space. We’re just really excited about it.

SHIRLEY: [00:05:44] John, what would you add to that from your perspective — again, framing it within the sort of bigger R/GA point of view?

JOHN: [00:05:51] I agree with a lot of what Nick said. I think it’s, that’s what’s fascinating about this new technology. It’s going to it’s going to change the way we we do things. As I mentioned, it can do a lot of tests that were pretty much impossible to do before and also because of the way that it was generated, because of the way that it was trained, it might do things that we don’t even know about. It’s almost like a child, as a child’s brain is developing, it’s starting, you don’t know what this person will be able to do. And and we’re starting to see that, starting to see, even though you build it to do something very specific, it might unveil and start doing something else really well-related to that. Very interesting what this is going to turn out to be.

NICK: [00:06:40] You know, just building on that, I think the other big thing about GPT-3 is that it challenges the idea of authorship. That’s what happened, I think, with the Internet in the early days as well, where it’s kind of if you were a copywriter in the pre-digital world where you sort of work with an art director and a copywriter and you create a thing and that thing has a shape. The line breaks are all in the same place. The content and the text are fixed. You can kind of get it right or you can get it wrong. But the thing that you output is fixed. When digital happened, that changed, right, because as an author or as a creator of a digital platform, there’s no way to actually map out all the states that the platform will be in. In other words, you can design the website, but it’s not until the website goes live and users start to interact with it and they upload content and they transact with it and data builds up in ways that you never predicted. In many ways, that’s where R/GA excelled was taking kind of design principles and applying it to this now non-linear space.

NICK: [00:07:45] And that is exactly what is happening, but at another level, with GPT-3. It’s a bit like designing games rather than websites, right, where what’s going to happen is we’re going to be creating the rules, the style guides and the guidelines, and we’re going to be putting, kind of putting the guardrails and sort of the training in place to do these things, but slowly but surely the process of making digital things is going to look more like that, where we’re putting these guardrails, we’re giving guidance, we’re training or shaping, but the thing itself is going to have many, many more states and many, many more outcomes than you can possibly predict or design for. So it’ll be a bit more like acting like a gardener and taking care of these things and building them in the right way and sort of interacting with them, than it will be kind of a landscaper where the thing stays put. You go away and you come back in the morning and it’s kind of still the way you left it. This concept of authorship is going to be much more interactive than it was before.

SHIRLEY: [00:09:02] Maybe in layman’s terms, as simply as you can, just kind of explain what it is and how it works for anybody who doesn’t even know what GPT-3 really is, how would you explain that?

NICK: [00:09:21] Yeah, at this point, it would be good to to back up and kind of talk about really what GPT-3 is and how you interact with it, how it actually comes to life and all the things that we’ve been talking about more more specifically. Well, GPT-3 is a is a language model that was built essentially by ingesting available data, mostly textual data from the Internet, and building essentially a compressed version of those interactions. And so it understands not only all the text that it’s ingested, but the patterns and the sort of the grammar and the logical relationships between between concepts. When you interact with GPT-3, essentially what you’ll see if you if you use it and there’s a few different places where people can go and try it out for themselves. Probably the first one where I really got to get excited about it was playing an application called AI Dungeon, which is if you paid for the dragon model, I think that’s the way it works right now, you get access to the GPT-3 version of it. And essentially it’s a kind of a choose-your-own-adventure interactive story where the user can input text and the AI Dungeon kind of narrator responds and builds the story around you. So what’s happening there is that GPT-3 is essentially interpreting your inputs as prompts and responding back. And that particular model has been fine-tuned, which is another way to say that sort of there’s this base model, this this giant base model that it’s built on, and the fine-tuning is is adding a more specific, more domain-specific, kind of universe of language on top of that. So it’s a language model that understands — I’m using the term potentially debatably — but understands and models the relationships between concepts and can do a remarkably good job of essentially carrying a narrative forward, a verbal text-based narrative forward. So that’s the beginning of it, and I think that all the things that we’ll talk about are a little bit sort of extensions of that concept and what it will mean for other for other domains. But that’s what GPT-3 is in its essence. And it can be applied to many different language-based tasks, narration being just one of them.

JOHN: [00:11:55] To explain it in a little bit different way, in simple terms and how it’s different than other models or even other iterations of GPT, by giving them very small examples of, let’s say, label data. You can learn from that and easily respond. So, for example, if you want to do translation, you might give it a phrase in English and a phrase in Spanish, and then you do that three or four more times, and then the next time you give it a phrase in English, it will return the phrase in Spanish. Or for example, figuring out even labeling things, before you will have to give it so many examples, maybe even thousands of examples of what you want, what the input is, and then what the expected output is. And then for you to learn and understand and then for giving input, give you what you expect. So with GPT-3, it it really simplifies that, as you mentioned, it is a pre-trained transformer model that allows you to, you don’t have to train it each each time, as you had to do in previous iterations.

SHIRLEY: [00:13:05] I thought it would be also helpful maybe at this point to frame this again within the AI space. Many people listening to this may not realize that R/GA actually has many years now of AI explorations for clients, things that are in the world, from chat bots and beyond, and maybe just quickly encapsulate what we’ve been doing in A.I. and now what this evolution brings. This is beyond a chatbot obviously. So could you talk a little bit about that piece?

NICK: [00:13:35] I think sort of in the sort of 2015, 2016 period of time, there was an explosion of interest and excitement and real breakthroughs, and in the AI space you started to see things like computer vision, image recognition, object detection and NLP, natural language processing, kind of took a bit of a quantum leap forward. Now — this in the context of A.I. research that has been underway for many decades — and I think this was similarly one of those moments where probably the hype got a bit out ahead of what the actual capabilities were at that time. And our role in looking at this kind of technology is to sort of differentiate where is the technology ready for mainstream? Where is the technology sort of being where we kind of almost anthropomorphizing and projecting a human-level intelligence on some of the stuff that was actually happening? Think about A.I. There’s different levels. And now, of course, A.I. refers to anything that computers do that’s that’s smart. I think in some ways we’d been doing AI, in other words, algorithm-based design and development, pattern recognition and using algorithms and data to drive experiences, for many years, really from the beginning of the platform development we work we did with Nike Plus and others. But what happened with machine learning is artificial neural networks. And that was a space that again kind of blew up at this time. And we did a lot of experimentation. We did a lot of work around NLP and chatbots. We built an R/GA in-house bot that does a lot of very functional things for staff: looking up people, HR tasks, and it uses a conversational interface to do that. In many ways, I think that’s technology that had achieved mainstream as broad applications. It’s something that we use every day on the projects that we do. But I think there’s also areas of AI that are that are more cutting-edge. And these are things that for our clients we’re advising them to experiment, to sort of explore the possibilities of — and to place some bets on — a few things that we think have a lot of potential to scale. Again, certainly areas like computer vision have really exploded; things like retail, things like image detection. You see a lot of applications now, things like TikTok, for example, that whether you call it AI or you don’t, it’s essentially an algorithm-driven experience. And so a lot of that stuff is beginning to sort of pervade and become woven into everything that we do. And so our AI practice, I think it started out more as a place to incubate ideas and a place to create some of these pilots. But it’s become more embedded so that now essentially we have data scientists, we have data engineers, analysts, and we have machine learning engineers that we bring into the work where where we can and where we see an opportunity to use data and to build more complicated models. That’s some of the stuff that that is, I’d say, just kind of baked into the work that we do.

NICK: [00:17:05] GPT-3 at this point is still, I would say, an R&D phase technology. It’s something where we’re spending a lot of time putting it out there, not exactly knowing where it’s going to go. And one of the first explorations that we did is to create a GPT-3 chatbot, essentially an interactive Slackbot that sits in our Slack channels and receives the inputs from everybody in the channel as prompts and is able to essentially assemble its own almost personality on the basis of those interactions. And so this has become an incredibly surprising and stunning display of the technology and how it works. Like we have different channels. The GPT-3 Slackbot exists in those channels, in different personalities, has a different way of talking, has different topics of conversation in each channel. So it gives you an idea of how this thing is very different from the chatbots that we would have thought of even a year or two years ago. So that’s an example of how we kind of, we put it out there over the last month or so. And this way designers, copywriters, other folks around R/GA can explore and can play with the bot and get a sense of what the capabilities are. And we think that’ll lead to other sort of inspirations for how to put this thing to use in ways we can’t sort of imagine right now.

SHIRLEY: [00:18:38] John, as somebody who’s been really steeped in a lot of our AI and ML work, and as you’re looking and assessing what’s possible with something like GPT-3, what has been in your mind and what’s it making you think about?

JOHN: [00:18:52] There is definitely so many different uses for GPT-3, and we’ve seen a lot out there in the world. There are people generating code. So by just giving them a few examples on any particular coding language, you can generate others, others creating Figma plug-ins for engineers. You know, I think for us, we also looking at what creative uses we can have with this. So think about creative writing as an example. Think about coming up with new taglines even or things to inspire copywriters. We’re thinking about how we can use GPT-3 for, you know, and this is early on, but kind of primarily in three areas: one for our internal products. How can we make our internal products that we’re developing better, smarter? How we can use this for our client work and talking to our our clients? And we can talk more or less about some of the client work we have done using AI, not necessarily GPT-3, but AI in general. And also how GPT-3 can help us with our creative process in by inspiration. Let’s just call it that. And I think that’s where GPT-3’s really good at. As Nick was talking about, kind of the chatbot kind of seeing some of the unexpected personalities or outputs, thinking about how we can use that for our creative process, for brainstorming. As far as work that we’ve done in the past, for AI and ML, it really, really varies, and we have worked with many of our clients. We have done stuff for Google, for Shiseido, for Samsung, for Siemens, you know, just just to name a few. For example, for Google, we used something for a campaign called Doodle for Google, where kids from from schools can submit their interpretations of the Google logo, like their doodles. So what we did is we used ML to detect to get hundreds of thousands of submissions. Some of them are not real submissions. Before, that was something that manually somebody had to go in and do and parse them through all the submissions. So now what we’re doing is we’re using machine learning to detect if those are doodles or not. So that’s one example. For Shiseido, we also worked on a project called Beyond Time that basically we created a real-time face-aging simulation. And using AI, we were able to kind of see how a person’s face will kind of change over time. We also did, this was more like a something internally for the holidays, we released something Holid.ai, or Happy Holid.ai, that basically we fed in all these different lyrics, holiday lyrics, and it will generate new ones. So this is another example where you can use, you know, GPT-3 will be great for those kinds of use cases. For Samsung we did something for a project called Samsung Galaxy Innovation, and we used AI for sky segmentation. So, you know, so training your model to be able to see, OK, where is the sky in this image or in this video and then be able to separate the sky from the rest of the image. Last one I will mention for Siemens. Within that, we used AI for content categorization. Again, something that was done before manually, you had to read the content and give it a category. Now we can automate some of that process with AI.

SHIRLEY: [00:22:38] Nick, I’ve been watching you in particular have conversations with the GPT-3 bot and what sort of strikes me, again, as a complete layman in this is just how poetic some of it is. There’s a lot of utilitarian possibilities, which, again, is very useful for companies in terms of just efficiency. But for us, being at the intersection of creativity and technology and seeing how, it made me think of two years ago going to the Brooklyn Museum and seeing the David Bowie exhibition, how he used hired a young computer scientist to help him sort of cut up ideas and see what that inspired him in terms of song-making. And I think he got that from William Burrows. And I think there is actually a really rich legacy of artists, writers, creators, etc. And we’re in the creative industries, you know, bringing using these sorts of tools to really inspire creativity and new thinking. I mean, Nick, you’ve been literally having conversations and testing this out yourself. What do you think your impressions of somebody just again in that sandbox and what’s it making you think about?

NICK: [00:23:43] I think certainly for language, the impact is going to be significant because it’s actually very smart. And the concepts are not just regurgitation. The concepts are legitimately compressed information that can give you new ideas. And I don’t mean just sort of sort of Pollock style, throw something up on the wall and it’s inspiring. But I mean, it will give you a different way of looking at things. And you can see from the chats that I’m having in the channels, the way that it’s I find it very remarkable how how a lot of the answers that it gives are quite spot on. But I think the other part of it is that in terms of generation of content and of of of moments where if you, to John’s earlier point about, I think what’s different here is that if you train the model. On a certain style and a certain conceptual pattern, you can then ask the model to replicate that style and pattern. And as an example, R/GA launched a new homepage last week, and we have a series of differentiators that we use to describe ourselves and we just we have a certain way of talking about ourselves, a type of language that’s difficult to describe. It would be difficult to brief somebody on “We want it to be brash. We don’t want to use jargon. We want to, et cetera.” But the better way to do it is to show, to actually give “here’s examples.” And I trained the model on those differentiators. Now the model will generate new differentiators and they are spot-on and hilarious in some cases. The one that I just generated while we were talking as our new differentiator for R/GA is “We run away from bullshit, but our cone is “no lie, no bullshit, no excuses.” It’s how” — I’m putting this up on the website, John — “It’s how we that’s how we keep score internally and how we aspire to work with our partners. We bring each other down to earth when we drift into our own bullshit, We ask rude questions and tell uncomfortable truths. We’re not really about innovation for innovation’s sake, but innovation for human impact. It’s our North Star and it brings a palpable reality to every conversation we have with clients.” So when you hear and you sort of see the kind of outputs like that, you begin to think, OK, so that has like a very real potential application in sort of creation of content and inspiring copywriters. I don’t see this necessarily generating outputs that go out the door to end users. But I think first it’s about inspiration and then it’s about the writer goes in and edits the outputs from this.

NICK: [00:26:57] But the other part of it is like world generation. Like if you think about a website as a it’s kind of this, ou go down these rabbit holes right in a website and you sort of you want to learn about certain things. And as you learn about those things, there’s this kind of you click on this page, then there’s a set of other pages and then there’s links out to other content and you can go — different people want to go at different levels of depth. But it may be that websites aren’t just like whatever it is, the 1,200 different pages that a brand has in their corporate website. But it may be that actually there’s a kind of a language style guide and a content training that happens. And the website, the experience that you have when you’re interacting with the website are generated. And certainly that’s the case if you think about game, modern game design and things like Fortnite and Minecraft and everything else where the states are not predetermined, they’re generated. And I think that, again, with something like AI Dungeon, that’s certainly the case. There’s no nobody has written these narratives. These narratives are are yet to be generated. And that change is totally what you do as an author. If you think about writing as an act of training, then you have this kind of fractal infinite set of outcomes that you’re creating. So I think there’s a change there and it sounds very esoteric and all of these things. But so did the Internet when it first started, right. It was like, what are we going to do with TCPIP and HTML and hyperlinks?

NICK: [00:28:34] I think that’s where we are with this tech, is that it sounds esoteric. It sounds like it’s just kind of a toy at first, but then it sort of sucks you in and it sort of takes its own shape. And I think that’s what we’re going to see here. I definitely think there’s going to be other types of content and other types of interactive experiences that this sort of becomes. If not, GPT-3 will be one among many steps that this takes. But this is one of those moments where it’s a step change and you sort of see where this stuff is is headed, certainly.

SHIRLEY: [00:29:08] I also want to ask — again, it’s very early days, as you both mentioned. It’s nascent. We’ve seen Microsoft just this week license it. They’re playing around with it, too. And I’m just curious, again, as you take a step back and as you look at these kinds of technologies and thinking — like you Nick explained, I thought really well — that this is just part of an evolution. It’s not necessarily the answer to everything, but it definitely makes you think and see possibilities. When do you say, “OK, this is maybe ready, if not quite for prime-time, but let’s try something.” Like what needs to be in place for you to say, OK, really interesting. And is it looking at the right sort of client or opportunity or how do you know when to get beyond kind of prototyping something to really take it up and try something with it out in the real world?

NICK: [00:29:55] All right. So there’s two parts that I think one is like there will certainly be places where you can you’ll start to, you’ll see this stuff in kind of campaign activations, I think they’ll be fun, experimental executions that we’’ll see in certainly in the next sort of six, nine months, I think within GPT-3, though, and sort of the kind of the domain that it sits in, there will absolutely also be kind of robust sort of enterprise tech applications of this. But it won’t be in the generative language. It won’t be in generating language. It’ll be in things like language search, language classification. I think even now you can look at using GPT-3 for things like classification and language search that that you can’t do with other models. So what you’ll see is you’ll start to see these kind of these kind of more specific use cases that get pulled into enterprise applications or, you know, sort of production applications where you’ll see GPT-3 in use. And then in the generative space I think I think we’re in early days. I think this is one of these things where I don’t think that it’s going to make its way into the sort of everyday copywriter’s kind of arsenal tomorrow. But I think that’s one where where there’s going to be the most potential as well. So on the one hand, I would say parts of it are ready now. On the other hand, I would say the more I think expansive generative side of it, you’re going to see in campaigns, more creative executions, but the the actual sort of industrialization of that that aspect of it will take time, many years.

JOHN: [00:31:41] Yeah, I agree with that. I feel like for certain things it’s super accurate right now. And it’s probably ready for prime-time, especially if they’re very if you’re asking GPT-3 for very specific things. So it has a very clear pattern that it can recognize and it can do. You know, I’ll give you give you a quick example. We have this game at R/GA, this card game called Not So FAQ. So I use GPT-3 to generate new cards, so icebreaker cards. So I already have the ice breaker questions that we’re using on the game and so generating more questions that are ice breakers, it can do that really well because it’s seen the pattern, it understands language, the pattern understands that OK, these are questions, these are questions about ice breaker, so it can do that. And even to take it a step further, you can do that in multiple languages. So I try and generate a deck in Portuguese and Spanish and generate new questions. And it does that real easy. If you like for other things, it’s still not there yet and it will still require some type of moderation in one company, one of the things they did is kind of like a A/B, using for A/B testing. So you can try a, let’s say, a new title, a new header for your web page, of your website, and you might have a copywriter write one or you might have GPT-3 generate one and then see which one does better. I mean would I do that blindly and press a button and let it send to the world or like Nick was saying, add it to rga.com, kind of like the new the new values like the ones that are generated? Probably not. Right. Probably not. Not ready for, not ready for that.

NICK: [00:33:39] And if we did, we wouldn’t share the URL for that page with you guys.

SHIRLEY: [00:33:44] I think this is all just incredibly valuable insight and context for this as people look at these new technologies and what it actually means in the process, I think has been fascinating and useful. So thank you both. And with that, I think we’ll wrap it up. Again. Thank you for your time. And I’m personally, of course, just really excited to see where this goes and what the applications may be once we do try this, potentially with some clients or internally keep pushing some of the tests that we’re doing. So it’s very informative. Thank you for joining us today. And once again, you’ve been listening to Nick Coronges, our global chief technology officer here at R/GA, as well as Jon Tubert, our SVP of technology, who is based here in New York as I am. Again, I’m Shirley Brady, and thank you so much again for visiting us. Please do visit rga.om. You can take a look at what Nick was referencing. You can see our latest work, our thinking, and please join us again next time. Thanks so much.

Originally published at https://www.rga.com.