Google Brain: Andrew Ng on AI, Coding Agents, and the Future of Software Engineering

It's great to have you back for year two. It's good to be back. This has always been such a cool gathering that you put together. I was telling you this earlier, but Andrew's fireside chat last year was the most liked and the most watched on YouTube afterwards, so we're thrilled to have him back for year two. Yeah. Maybe jumping off of that, in the year since we've been here, I feel like there's been a ton that's happened in the AI space. What has happened faster than you would have expected and what's been slower? So I think the hype has exceeded my expectations, and then also the doomsaying narratives also got more traction, unfortunately, I would have guessed, including the jobpocalypse, which I don't think is going to be a thing. On the more positive side, coding agents probably took off faster than I would have guessed, and the frontier coding agents. I know the hype is that everything in AI changes every few months or whatever, and that hype isn't entirely true, but that does feel true for coding agents, where it feels like the frontier of what we can do with coding agents, and it's so competitive, right? I think six months ago, I was almost all cloud code. These days, I'm still a lot of cloud code, but also increasingly open AI codecs with a mix of Gemini CLI and want to support open code as well, because it's open code. So the mix of coding agents we use has been changing rapidly. I wouldn't have guessed a year ago that I'd be coding so much on my phone as well, right? So these workflows, like many of you have a Mac mini in my office, so all these workflows changing really rapidly, and then agentic workflows are really starting to make their way into enterprises. That feels pretty good as well. So on that note, on the software engineering note, what does the future of software engineering look like in your minds? So about a year ago, I was writing about the product management bottleneck, which is this observation that if building becomes much faster than deciding what to build, or the product management work of scoping the project, we're getting customer feedback, deciding what to build becomes the bottleneck. And it feels like over the last year, the product management bottleneck has become much worse in a good way, because software building has become much faster. But it turns out when writing software becomes 10 or 100 times faster, not only is there a product management bottleneck, pretty much everything else becomes a bottleneck. So some of my teams have seen the marketing bottleneck, because we can build so many features, our marketers are scrambling to figure out what on earth the engineers did in order to figure out how to communicate it. Previously, if you had a product that needed legal compliance, if you took three months to build it, waiting a week for legal to sign off is maybe okay. But now, if you're building in a day, then wait a week for legal to sign off, that's a legal compliance bottleneck, and there's a design bottleneck, and so on. So I've been thinking a lot about how software engineering teams in the future will be organized, and I don't think I know the answer. But increasingly, I find myself setting up very small pods, anywhere from one to ten engineers in the team, of often generalists, high-context, highly-empowered generalists that are given a set of very wide guardrails, within which they can just run like crazy and build and ship code, and even drive decisions like writing marketing copy, that are traditionally outside the purview of engineering. Maybe by definition, let's say you have a team that needs software engineering, product management, a little bit of, you know, you need a new terms of service, you need some marketing copy, you need some design, say you need five functions represented in a team of two humans, then by definition, or by the pigeonhole principle, these two humans have to play more than a single role per human. But the good news is, it turns out, I don't think I'm a very good marketer, but when I use, you know, AI, right, I'm still not a good marketer, but I'm slightly less bad compared to if I had to do it without AI assistance. So I find these small teams of high-context generalists that are deeply technical, but able to use front-end technology to do a little bit of other roles as well, like engineers, you know, frankly, use AI to take the first draft of a terms of service, to then take to a lawyer, for the lawyer to finally polish it before it goes out, but I find that these processes allow teams to move much faster, so that's been really exciting. And what's the right background for these folks? Are these engineers by background, are they coming in from different disciplines and learning to code for the first time, what are you seeing here? Yeah, so a lot of the people I work with most closely have deep engineering and technical expertise, and then also are highly empowered, slight generalists that step into other roles and acquire this mix of skills, often with AI supporting where, you know, they didn't have that training. I think people can succeed from any direction. I've seen product managers become much stronger at coding and then participate in these teams. I think just because so much of AI coding and engineers maybe had a natural advantage understanding the frontier tech, so I see engineers play this role successfully the most often, but there's definitely a smaller number of product managers, I've seen marketers learn to code in pretty effective ways, I've seen operations people start to build more and more products. So it's possible for people of, you know, any background to learn to do a lot of this, but the largest number of bodies doing this well right now seems to be people that came from an engineering background. But we encourage everyone from whatever background to see if you can play these roles. What advice would you have for folks who are trying to get more into this new software engineering space, whether it's particular tools to try, or mindsets to have, or skills to pick up? Yeah, so there's one way I've been thinking about the future of software engineering. This is a mental model I have in my head, which is that because of a lot of providers of many tools like, you know, RAG, agentic frameworks, things like evals, guardrails, it turns out that, as well as, so these are AI building blocks, they're also non-AI building blocks, things like user interface components, or identity of mechanisms, and, you know, front-end, back-end persistence databases, and so on. So I think that in computer science, we've always had a wonderful set of building blocks, and with agentic coding, building blocks are proliferating because more and more people are building open source or proprietary API-based or whatever types of building blocks. So they're just wonderful building blocks all around us. And so I find that developers that have a good mastery of enough building blocks can often put them together in combinatorially many ways to very rapidly build software. Right, maybe, if you picture a building with Lego bricks, if all I have is like a white-colored Lego brick, you know, I can build some stuff that's not that interesting, but I mix in a black Lego brick, a yellow one, and a brown one, and a green one, and throw in some squiggly pieces of Lego, then what I can build grows combinatorially or grows exponentially as a function of the number of Lego bricks I have. And I think of a lot of building blocks we now have access to as being akin to that, too. So I find that the developers, they're able to have a good sense of a lot of what these building blocks do, and DeepLearn.ai offers a lot of long-form short courses to help people master these wonderful building blocks provided by tons of people. And then the other challenge is to use coding agents to rapidly assemble these building blocks into the software that you want. One challenge that coding agents have is a lot of building blocks are so new that the coding agents do not know how to use them, right? I think until recently, the leading coding agents were, you know, NanoBanana was released after the knowledge cut-off date of a lot of leading models, so they just had no idea how to call the NanoBanana API, or even knew that NanoBanana existed. So one project that one of my friends, Rohit Prasad, and I have been passionate about is Context Hub, which is kind of a stack overflow for AI agents, for AI agents to get the latest documentation on what are the latest APIs, SDKs, building blocks they can use, as well as a way for agents to give feedback to the documentation to try to improve it for everyone. As it turns out, there are a number of APIs that I personally use that, you know, I find the syntax slightly annoying to remember, but I find that by using Context Hub to load the latest docs, I let the coding agent accurately make all of these API calls for me, right? And so it actually has helped my own coding work, you know, accelerate quite a bit. Maybe two notes there. We also launched something called Context Hub, so we're colliding on names, but it's a very different type of Context Hub, and so, yeah, his is much more useful for working with coding agents, so you should go use that. And then deep learning. So I think we at LangChain, we're the second people behind OpenAI, I want to say, to do a deep learning class, and I know I've talked to so many people who have heard of LangChain through deep learning, and I'm sure a lot of the audience members here as well. So if you haven't tried out deep learning, you should absolutely go take some courses. Maybe on that note, how do you think about education changing in this new world of AI, and have you started to bake any of those practices into how the deep learning courses are run or how you think about that? Yeah, so we're trying a lot of ways to improve the education experience. So maybe in terms of training, the thing that's clear is that what people have to learn has changed significantly. I think for developers, we have to learn coding agents, learn these building blocks, maybe learn some product management or these sort of general skills to make us more effective. So what we learn is changing, and deep learning AI, Coursera, UW tried to provide a lot of that training. But separately from what to learn, there's the delivery of training, and it feels like we've been thinking about how will learning transform for a long time, and it feels like it's actually not quite here yet. One thing we launched just several weeks ago is a new website in our preview called CodeDream.ai, in which the vision was, rather than taking an online course, come and have a conversation. So it's not a course, it's a conversation, where the experience we tried to build was for you to come and get on a simulated video call with me, say, where if you feel like leaning back and listening to me talk and present to you in a one-on-one simulated video call about context hub and coding agents, you can lean back and I'll just show you some ideas. Or if you want to interrupt me or interrupt my AI and ask a question, you could do so at any time. And one thing I see personally playing around a lot with is replacing videos and slides with JavaScript, and what that means is instead of a video, when I present, when I demo something in a video, if you could, you know, click into the video and type your own prompts or type your own queries into the video window, so instead of a static video that just plays, the video area is interactive and just creates more ways for people to interact. So check out Codream.ai if you want and have a kind of a conversation with me or with my AI, where I'll present to you in AI form how to use context hub and these coding agents, but I think we're still, we're actually frankly still iterating and improving these experiences every day. Is that clicking into the video and typing, is that live today or is that more of a future direction? That's live. So imagine if instead of me screen sharing in a video, I am, you know, JavaScript sharing in a video, and so you can interact with whatever I'm, quote, screen sharing of, because it's running JavaScript code, not a canned video. So yeah, we're still, actually, if any, if you, favorite, we'd love your feedback, but it feels like, I think the transformation of education has been overhyped. I think something is coming, but today, taking online courses, you know, I wish we had something way better than online courses, and I would say we have something better than the courses we had 10 years ago, they're much more interactive. It turns out for a lot of our courses, this morning, we actually just, we just launched a new course on Transformers taught by A.M.D's Sharon Zhou, and I find that rather than just having videos, which is mostly what we had, you know, a decade ago, we now build much more interactive visualizations, there's a lot more fun stuff to play with than courses we just had a decade ago, but that biggest transformation, I actually spend a lot of time thinking about that, yeah. Maybe going from software engineering to everywhere else, how do you see enterprises adopting AI, and is it faster than you expected, slower, what's the right way to do it, what lessons can people learn? So I think every enterprise, I'm guessing all of yours, are excited about AI adoption. One thing we've seen is, over the many businesses, it turns out one of my teams, AI Aspire, which is an AI advisory firm that my business partner, Kirstie Tan, and I co-founded, we talk to large businesses all the time, so the largest, kind of Fortune 50, Fortune 500, G2000 businesses about the AI strategies and transformation. So some consistent themes, right? All of us have invested in the bottom-up innovation, let a thousand flowers bloom strategy, and for the most part, it's not paying off. So CEOs and boards are asking, where is the ROI for AI? I think we should keep on investing in bottom-up innovation, let's keep on doing that, but it turns out that bottom-up innovation often results in point solutions that drive incremental efficiency gains, which are actually a good thing, but not the transformation, not the broader transformation that AI has been promising us and that I think we should work to deliver. So just to illustrate this with an example, my teams work with a number of banks, we actually do a lot of work in financial services, but work with a number of banks, and if you think about the process of underwriting a loan, there may be, you know, five steps. You've got to market a loan product, get the application, review and approve the loan, do the final diligence, and execute the loan, right? So there's five steps of that. Number of teams have noticed that the step in the middle of loan approval, we could use AI to do that, and if we can automate that, then instead of a human spending an hour reviewing the loan application, we could have AI do it with, you know, and that's great, we should absolutely do that. But it turns out that if your entire process of underwriting the loan stays the same, except for automating what was previously one hour of human time, that's a small incremental efficiency gain. So what a number of banks have said is, you know what, instead of doing this efficiency gain, which is worthwhile, let's rethink the entire workflow and market a get-approved-in-10-minute loan product, because rather than waiting around for a week for a human to be free for an hour, we can send the loan application to AI right away for a decision. But the challenge with implementing this in a lot of businesses is, this takes someone with a broader scope to rethink and redesign the entire workflow, because now you have to market a get-approved-in-10-minute product, you have to route the application to approval not in a day, but, you know, right away. So marketing, data infra needs to be involved, then yes, AI can make the initial decision, and then final diligence execution probably needs to scale up as well. And so I find that bots for innovation is really valuable, it generates lots of ideas, but often it takes, and that has to be complemented with a top-down motion of having someone with a broader scope to change how all of these steps operate, to then create growth. And I find that many businesses talk about cost savings, you know, that's fine, it's worth doing, but I try to push for more imaginative things we could do with AI, which is drive growth, because we can only save so much money, but growth has almost no practical ceiling, so the more exciting ideas I find usually relate to driving business growth rather than savings. Are there any good examples of driving that business growth that you've seen that you're particularly excited about, or think other people should look at and learn from? Yeah, so, let's see, the bank example is a real one, right, we work with a number of banks on that, and financial institutions on that, and other businesses have also done that. I don't know, I find that, let's see, maybe one other pattern, customer service, contact centres, often viewed as a cost-saving thing, and that's fine, cost savings are nice, but when you can automate customer service, or automate or augment part of it, then the ability to serve customers many more, much faster, that delivers a more delightful customer experience and drive growth. I think that, speaking with a number of businesses that have been working on automating the drive-through voice, drive-through order process, I think that also results in a more delightful customer experience and drive growth, so, I'm seeing that there are more and more of these examples popping up in different places in the economy, there's some I know about that don't have permission to talk about, but I'm actually pretty confident with the number of things businesses are working on, that there will be more and more examples of these things. You mentioned ROI earlier, and from a lot of conversations that I've been having yesterday and today, I know a lot of people are thinking about that, and thinking about how to measure that, and in, I don't actually know if, in some scenarios, maybe in the cost, or in the call centre, it's maybe easy to think about measuring that, but how would you advise people to think about measuring ROI, and any advice for them there? Yeah, I wish I knew. I find that, I think part of the challenge is businesses are so diverse, so measuring ROI is like measuring business, which is, you know, very hard for a one-size-fits-all answer for. But there is one thing, I feel like the projects that excite me most are measurable, should be measured, deserve measurement, but there are some things where we swing for the fences and create so much value, you know, we're not debating, did this drive 2% growth and then minus the 1% cost of implementation, it's just glaringly obvious this is transforming the business. And of course, we still need to measure it, especially when a publicly traded company, blah, blah, blah. But I find that those things, there's actually one thing I've learned, you know, sometimes driving incremental gains is harder than driving transformative gains, because if you tell someone to improve their business results by 2% next year, then they're like, alright, my boss is telling me to work 2% harder, or 5% harder, or whatever. But if you try to search for ways to drive, you know, 20% business growth, or 50% business growth, then you can't just get everyone to work 50% harder in the company, and you have to come up with more creative solutions, and that often leads to, oh, just one lesson I've learned. At AI Aspire, we've had many businesses literally send us spreadsheets with hundreds of ideas, like, sorry, I'm sorry, one financial institution sent us a spreadsheet with like over 300 ideas, and asked us to help them sort out of these 300 ideas, which ones to, you know, put real capital to invest behind. And it turns out that the analysis is really difficult. I wish I was smart enough to glance through it and say, oh, this idea and this idea. But I find that when faced with that many ideas, often brainstorming in a top-down and bottom-up motion, it's actually a lot of work to do the technical analysis to figure out what is possible, and then also the business analysis to figure out which ones, you know, could drive meaningful change, and then it's actually a lot of work on the technical and the business analysis to then narrow it down to what's a small handful of bets to put very meaningful resources behind. And these swing-for-the-fences type things, you're often seeing these being the top-down ones. That's correct. Yeah. Yeah. I find that businesses hopefully not take one wild swing for the fences. It's more a portfolio of a handful of thoughtful bets, where if anyone pays off, it will be meaningful for the business. But it turns out, one of the things I love about, you know, agentic coding is we run tons of experiments, right, prototype all the time, so the cost of prototype has plummeted. But sadly, you know, you can't do everything, right, on a $100,000 budget, and at some point, putting meaningful resources behind every one of a small portfolio of a handful of projects does make sense. But so, because of the resource allocation needed at that level, it often takes, you know, a little bit more of a top-down motion to allocate the amount of resources needed. One of the things that I feel like has been talked a lot recently about enterprise adoption of AI is forward-deployed engineers. Will every company have forward-deployed engineers? How do you see, why do you think they're so impactful, and how do you see this playing out in the future? So I think the Silicon Valley buzz, you know, thing is definitely having a moment of FDEs. I think, and I know Aaron was on stage, he wrote a very thoughtful tweet about it, like one or two days ago as well. So I think FDEs are a great idea, and many businesses, but I think, looking to the future, what do you think is the ratio of FDEs in the company versus the number of just AI engineers employed by the company, right? I think that most businesses will have a lot more in-house engineers and a smaller team of FDEs maybe embedded. So that's why I like FDEs, I'm excited about the growth, let's help more people get jobs as FDEs. But I think the hype is also, as is often the case, a bit greater than, you know, the actual reality. But it is a good thing. But building agentic workflows is, you know, it's hard. It requires understanding the business, it requires customer-facing skills, often to make it reliable you have to drive observability, evals, work with the customer to push back if something is not actually technically feasible, work with the stakeholders to figure out what workflow to automate, help with the change management. So this is a very valuable role that takes deep technical judgment and having FDEs embedded can really accelerate projects. There's one other thing I see as challenging for a lot of businesses, which is, is there any way to get a vendor-neutral FDE? That turns out to be challenging, and depending on what vendors you want to be really embedded with. Because what we see in AI is that the leading AI model rapidly changes. So I have no idea what will be the leading AI model a year from now. I'm actually not at all sure what will be the leading coding agent a year from now. And so in moments of uncertainty like this, optionality is very valuable. So candidly, many vendors are coming to all of our businesses and offering 20-30% discounts for signing a three-year contract, right, whatever. Not giving anyone advice, just saying what I do, I personally almost never sign longer than a one-year contract, regardless of the discounts offered, because I value that optionality to work with whatever vendor would be the best in a year's time that I don't know about. And then when we work with FDEs, one question that I think businesses are asking is, when you have a handful of FDEs from one company in your company, how much does letting them embed everything with one AI model or whatever, reduce your optionality one or two years from now? So I think these are some of the businesses that companies are wrestling with. And this is why, by the way, I think I personally have used LangSmith a bunch of times. I think Harrison did a great job making it so easy to use. I think those types of more vendor-neutral tools are very valuable for observing, maintaining optionality for the long term. And the vendors are great. Work with the vendors. But preserving optionality for yourself in the long term also seems important. On the topic of vendor-neutral, especially in the model space, one of the things we've talked about a little bit throughout the past few days is open-source models. How do you see those progressing, and how do you see them relative to the frontier models? It's been fascinating how it's remained persistently, I'm going to say maybe six to nine months behind the frontier models, but the frontier models are expensive enough that for many use cases, my teams use a lot of the open-weight models, sometimes fine-tuned, sometimes without fine-tuning. So I hope we can all keep on supporting the open-weight model. Over the last two weeks, there have been concerning noises out of the White House about inspecting models before they're released, so I'm actually quite concerned about that, and touched with a few friends in the administration about this. And I feel like a lot of the war on open-source, open-weight models is still being waged.