As artificial intelligence advances at breakneck speed, Daniel Castro, President of the Information Technology and Innovation Foundation, offers his perspective on how policymakers should approach both its risks and opportunities. In this conversation, he makes the case that America’s AI challenge is not just developing the best technology, but ensuring it is widely adopted across the economy. He also discusses competition with China, cybersecurity and other emerging risks, the debate over slowing AI development, the growing demand for data centers, and why he believes AI policy should be driven by evidence rather than speculation about worst-case scenarios.
Key Takeaways
The biggest AI race may be about adoption, not just invention. On a No Labels call, Daniel Castro, President of the Information Technology and Innovation Foundation, argued that U.S. policymakers should focus not only on developing the best AI systems, but on deploying them across healthcare, education, manufacturing, financial services, and other sectors. Castro warned that if China leads the United States in AI adoption, it could have long-term consequences for U.S. competitiveness and economic growth.
Pausing AI development is not a realistic solution. Daniel Castro, President of the Information Technology and Innovation Foundation, told a No Labels call that even if the United States could pause AI development domestically, it could not stop development in China or the spread of open-source models. Castro also argued that slowing AI research could slow the safety research needed to make AI systems more secure.
Policymakers should focus on concrete AI risks like cybersecurity and biological threats. On a No Labels call, Daniel Castro, President of the Information Technology and Innovation Foundation, made the case for an evidence-driven approach to AI regulation focused on identifiable threats. Castro called for more federal AI safety research, a stronger cybersecurity focus, and targeted safeguards for especially sensitive AI applications rather than broad restrictions on AI development.
Data centers are the “factories of the future,” but communities should judge projects individually. Daniel Castro, President of the Information Technology and Innovation Foundation, told a No Labels call that the United States will need significant data center infrastructure if it wants to lead in AI. Castro said individual projects should be evaluated based on local energy and water conditions while also accounting for potential benefits including tax revenue, infrastructure investment, construction, and jobs.
Castro is skeptical of claims that superintelligent AI could soon threaten humanity. On a No Labels call, Daniel Castro, President of the Information Technology and Innovation Foundation, said those making extraordinary claims about AI posing an existential threat should provide concrete evidence and a plausible pathway for how that harm would occur. Castro said he remains open to the possibility but does not believe policymakers should design AI policy around speculative worst-case scenarios.
Transcript
Ryan Clancy: Good afternoon, everybody. This is Ryan Clancy, Chief Strategist for No Labels, and we are here for one of our periodic expert briefings on an issue that we think is of utmost importance – artificial intelligence. I don’t have to tell all of you how this has very quickly become one of the most important, if not the most important, policy issues in Washington today. There’s so much discussion about the implications of AI, how fast it’s moving, and obviously the big question is what role government should or should not play to shape its direction and ensure that it evolves in a way that contributes to human flourishing, and not something worse. For some thoughts on that, we have somebody who’s spent a lot of time thinking about this and has spent his whole career on the frontier edge of technology issues. That’s Daniel Castro, president of the Information Technology and Innovation Foundation. He’s been with ITIF for several decades and ascended to the presidency earlier this year, taking over from the longtime founder, Robert Atkinson, a guy we know well. You can often see Daniel’s perspectives all over the place, including in front of Congress, where he very often testifies. Daniel, thank you for joining us today.
Daniel Castro: Yeah, Ryan, thanks so much for having me here.
Where does AI development stand right now?
Ryan Clancy: Just to start, could you set the scene for us in terms of where you think we are in the evolution of AI and how we should be thinking about this moment?
Daniel Castro: Yeah. First, again, it’s really a pleasure to be here. I started working on AI policy over 10 years ago, before AI was cool, when it was mainly something computer scientists were talking about. And then, of course, the futurists and the science fiction folks. Obviously, we’ve gone through a few waves of AI since the ChatGPT moment that really brought it to the forefront for a number of people. We had that moment where I think there was this realization that AI is big and consequential. The adoption curve of ChatGPT was unlike any other technology in history. The speed at which it got to a million users and beyond was remarkable. And then we had the DeepSeek moment, which was the moment we realized, wait a second, this isn’t just American technology, it’s also Chinese technology, and we’re in a real race to see who will be the leader in the development of this. And now I think we’re in this other new moment where people are really raising questions: What are the implications of this? In the short term, that means jobs, cybersecurity, and bio-threats, but also the longer-term unknown of what happens when AI keeps improving. Are we going to have, this artificial superintelligence? What does that mean for humanity overall? And these are not new conversations, but they’re new conversations to be had at this national level. And so, we’ve taken what were once conversations among very small groups of people and expanded them to the national level in a way that I think is raising a lot of questions, fears, and anxieties. And of course, we’ve seen this in the past with many technologies, whether it’s the printing press. We can go through the long list there, and I think probably people are familiar with many of these. The question put to policymakers is: how do you respond? And there’s a lot at stake here. And one of the arguments that I’m really trying to make is that it’s not just the risk that something goes wrong, it’s the risk that we don’t make sure it goes right. It’s the lost opportunity – the opportunity cost of not using AI to significantly reduce healthcare costs and improve healthcare outcomes. It’s the opportunity cost of knowing we need to improve our public schools and not using this amazing technology to provide the highest-quality tutor that’s available in the world to every student in America and really think about how we can improve their educational opportunities. It would be a missed opportunity to not use AI to think about autonomous or semi-autonomous vehicles, where we’re talking about how do we reduce roadway deaths and accidents. There are many areas like that where AI presents enormous opportunities. And I think the thinking right now, both in Washington and throughout the states, among many policymakers, is we just need to make it safe and then everything else will fall into place. And I think the most important historical lesson we can learn is that’s never the case. Adoption is incredibly hard. We had a bill a few years ago, the HITECH Act, part of the health IT reforms, that spent $40 billion to get hospitals and physicians to adopt electronic health records. The technology for digital health records has been around since the 1990s, and the internet was widely adopted by the early 2000s. The technology itself wasn’t the hard part. It’s changing human behavior. Getting us to do these types of Zoom meetings took a pandemic to get people to really learn how to change their behavior and engage in this type of new use of technology. And so with AI, it’s going to be the same thing. How do we actually get adoption? And that needs to be something that government works hand in hand with industry and academia, so we can make that happen. And again, that’s something where we talk about the race with China. There are national security implications and other considerations, but the adoption of AI throughout the economy, in every sector, that’s the race that I’m really worried about. It’s not just who’s leading in the development of AI, who’s leading in the adoption, because the lead adopters of AI, whether it’s in manufacturing or financial services, or healthcare services, all these areas that are traded sectors in the economy. If we don’t lead in the adoption of that, we’re going to fall behind in terms of competitiveness, and that’s going to have long-term implications for the U.S. economy, GDP, and economic growth.
Should we pump the brakes on AI?
Ryan Clancy: Daniel, how should we think about the many voices out there who are, in different ways, saying, “Pump the brakes on this”? Those voices are not just coming from Congress or outside activists. In several cases, they’re coming from senior people inside major AI companies themselves – Dario Amodei, Elon Musk, and Sam Altman – all of whom in the last couple of weeks have said some version of “We are worried about some risks here, and we need to think about some sensible guardrails for how we move forward.” What are we to make of this?
Daniel Castro: Yeah. One thing that’s important to recognize is that, of course, it’s not the whole tech industry saying “pump the brakes” – it’s only some of them. Sometimes it’s presented as a unanimous view, and that’s really not the case. I was talking to some researchers at a major frontier lab this morning, and their message was: no, we don’t need to pump the brakes. We want to do more work. We see the risk out there, but we do want to see more understanding of what’s going on. One of the problems I think we have right now is, every day there’s a new headline about new cyberattacks or models or agents escaping the lab, engaging in some type of cyber attack. We don’t actually have yet good language to distinguish between a minor incident and a major incident, or everything in between. And so, when we see this, it’d be like if a bank couldn’t account for $10 versus $10 billion, and it really matters what you’re talking about in terms of scale, and that’s the same thing with a lot of these AI incidents. There will be AI incidents, there are going to be many AI incidents. That’s the history of technology. Things don’t work right the first time or the second time, and there’s a lot of improvement and evolution. But we need to know when we should worry and when we should not. We need better evidence, metrics, and assessments of this, that can be cross-industry, so it’s not just about taking one company’s word for it. And this is where some of the institutions that have been created, started in the Biden administration, now in the Trump administration, the Center for AI Standards and Innovation, which is in NIST at the Department of Commerce. There’s bipartisan support for that work – measuring and understanding the benchmarks. How should these different models be tested? How should the agents be tested? There’s so much innovation happening in that space, and there’s so much cross-lab learning that should take place. And government has an important role to invest in that and help accelerate that. And we’re still waiting on some bills that would actually give permanent funding and expand that. So that’s one way to look at the solutions in this space. The other thing is to understand that there are real risks, but the real risks probably aren’t that AI might end humanity in the next decade. We know these systems are very good at identifying and exploiting cybersecurity vulnerabilities. We know that today, based on what the capabilities are of the Claude-class models from Anthropic and the latest models from OpenAI. And so, we also know that we have a lot of vulnerabilities in our digital systems throughout the economy, whether it’s financial systems, critical infrastructure, or systems across the board. And we know they’re being exploited today by human actors. Those human actors abroad are going to have new tools in 6 to 12 months, coming from China, coming from open models that are going to be accessible to everyone. And we have to close those vulnerabilities. So we hear about this call for a pause, but we, at best, we could pause what’s happening in the United States, if that were even possible, I’m not sure it is. You can’t pause what’s happening everywhere else. Those tools are going to be out there. And so, to me, again, the real question is how do we prepare ourselves for that? How do we make our systems secure so it doesn’t matter if we’re being hacked by an individual or an individual using an AI model, we’re still secure. But to really get to your point about whether we can pause it, the challenge that I see in that discussion is it’s never clear exactly what they’re talking about pausing. You have to who are you telling to stop doing what? Are you telling AI labs to stop doing research? Are you telling them to stop doing research on safety? Because that’s part of the frontier as well. Are you telling them that they can’t have larger models? Well, a lot of the models, they get better not by getting larger, but by getting more optimized, or through chain-of-reasoning approaches, which means that instead of spending more time training the model, you’re spending more time executing the model, and running it over time, and linking them together. Or you see work with agentic AI, where multiple smaller agents come together, and it’s forming collective intelligence and action. So pausing research doesn’t really even compute in that sense. It’s not clear what they want, they’re just, I think, saying, well, something’s wrong, and please stop. And that’s not a viable solution.
What should the federal government do about AI?
Ryan Clancy: Anybody who has a question for Daniel, please raise your hand. I’ll be happy to call on you. I have one or two more in the meantime as we wait for folks to come up in the queue. Daniel, how should we be thinking about what the federal government should do here? I think one of the things that’s hard for people to get their arms around is that everybody has a sense there are some things government should be doing, but there’s also some cynicism that some of the companies asking for regulation happen to be incumbent companies. The concern is that maybe they’re trying to put up firewalls to prevent competition. There was an article in The Wall Street Journal the other day about the early days of electricity. There was a debate over whether alternating current or direct current should be the standard, and Thomas Edison, an advocate of direct current, was going around electrocuting animals with alternating current to show, “Hey, this is really dangerous.” Some people thought he had legitimate concerns, but others thought he was just trying to talk his own book – he wanted his thing to be the standard. So how do we disentangle some of the self-interested stuff from the constructive things the federal government could be doing right now? What helpful guardrails could ensure innovation moves in the right direction? What should that actually look like?
Daniel Castro: It’s a great question, and the analogy to electricity is great. If you remember, the electric chair was also part of Edison’s effort to make alternating current sound incredibly dangerous. And, again, there’s some validity to it, but that wasn’t really the root cause. I think with AI safety, we should always be evidence-driven. Right? That’s the most important thing, is you follow the evidence. So, you say, what are the risks that we can see today? What are the harms that we see today? And how do we respond to those? How do we mitigate those? And some of that isn’t about actually changing the model. Again, if you look at cybersecurity, for example, we could you could I guess one solution, in theory, is make these models not available so that they can’t exploit vulnerabilities that are out there, or you could say fix the vulnerabilities. Now that’s not always the solution, and that won’t always be the case. In biotech, for example, there’s the risk of using AI to create bio-threats. We do want better AI, but we also probably want to try to identify if we have specialized models that are highly capable of helping synthesize potential biological agents, then maybe those are limited, and there’s export controls on those, or there’s access restrictions on those, or there’s know-your-customer rules. And, we think about how do we also improve those models to be able to distinguish between a dangerous pathogen, or someone that’s trying to create a dangerous pathogen, or something that’s not. We think about supply chains, and we say, well, you might be able to have the know-how to do it, but you’re not going to have access to the materials. Or if you start getting access to the materials, there’s going to be some red flags that are raised. So there’s a lot that we can do when we look at very specific problems and say, well, how do we address that problem that’s not about pausing the AI, it’s about, getting to that piece of it. And I think that’s how we should pursue most of this. Now, broadly, what the government can do, one, it needs to fund more AI safety research. There’s just so much that’s being done in terms of benchmarking and testing that isn’t being done today, and it won’t be done as long as we, don’t have a government agency that’s really focused on that. Two, on cybersecurity in the short term, CISA or another agency should prioritize that rapidly. Three, this isn’t an American-only problem. Other major labs are in China. Obviously, we have relationships with Cohere in Canada and some of the European labs, but we need to figure out how the U.S. is going to work with China on these issues, because when these agents are operating on the internet, which they already are today. These are going to be American agents, Chinese agents. We need to understand how they work together, and we need to be able to share information about risks, vulnerabilities, incidents, and ways that are useful. And so, that’s a really good starting place for how we can start moving forward in this area.
Ryan Clancy: Okay. We have some questions. Go ahead.
How soon could AI and robotics transform the economy?
No Labels Member: I’ve read a number of things about the positive benefits of the future of AI – that the technology will allow things that today are almost mind-boggling, like robots building housing to eliminate homelessness or economic activity increasing to the point where the total U.S. debt can be eliminated by the middle of the 2030s. How feasible is that in your mind?
Daniel Castro: Yeah, I think it’s pretty far off. I don’t see that around the corner, especially in robotics. I think we’re really behind. You look at where it is today, the U.S. Has had a declining robotics industry for quite a while, and a lot of it has to do with the supply chain. We need to rebuild that robotics supply chain so we can have an American robotics industry again. I think there are some short-term protectionist steps that might help. We don’t want to just maybe let in all the competing Chinese robots everywhere if we’re serious about building it back up. But we are a very long way off, and the reason I the technical reason I say we’re a long way off is that when you look at the current wave of AI, it’s based on, the large language models that most people are familiar with. There are limits to what those models can do, and so there’s another whole strain of research right now around world models, which are trying to understand, basically, the physics of the world. So instead, if you look at a picture of a tree and you want to understand what’s happening, you could maybe assume that if it’s windy, that a tree, the next image in a movie frame might show the leaves moving. But a world model would actually understand the physics of it, and it would understand exactly what’s happening in that world. And that’s what I think we have to get to in order to have a lot of this more, advanced robotics, that could build things, that they can interact more seamlessly with people safely. And those companies are in their infancy right now. There’s some great researchers doing that work, but they’re still in the infancy. And so, whether that’s, 2 years or 20 years, we don’t know. But we still have a lot of problems to solve, and the current wave of AI is likely going to hit some scaling limit at some point, and we’re going to have to have a new paradigm. That happens all the time in technology, and I’m sure that will happen again with this as well.
Ryan Clancy: Go ahead.
How much water and electricity do data centers really use?
No Labels Member: There seems to be so much disinformation coming from China, Russia, and apparently now Iran about how bad building these facilities is. Where do you get good information? How do you know if they’re really using so much water and electricity to build these facilities? What information can be used, and what can’t be used?
Daniel Castro: Yeah, that’s a great question. I’ve looked at various data center projects, and the problem is, of course, some don’t disclose all the information, so there’s only so much you can know. One thing that has been clear is that the large hyperscalers have made very strong commitments in terms of reducing energy use and getting to net zero on water – they want to be water positive. But of course, there’s more than just the hyperscalers out there, those large tech companies, there’s also, other operators, and not everyone has made those commitments. So I think we have to evaluate each project on an individual basis. We have to look and say, I’m very pro-data center in the sense that, if AI is the industry of the future, data centers are the factories of the future, and we have to have those factories, we have to have that infrastructure. That doesn’t mean I want a factory next to my house. It doesn’t mean that a factory should appear
Daniel Castro: everywhere in the world or in every community, but we need them, and we need to find places for them, and we need to evaluate each project and look at the net costs and benefits. And in many cases, they’re appropriate. And so some of the things, end up being very specific because they’re based on the environment. So, for example, in one location, there might not be a stressed watershed, so it actually makes sense to use water for cooling. Because that’s the most efficient way of doing it. In another place, it might not make sense to use water, but they’re going to be using more energy, but it’s because they have a lot of solar, they have a lot of wind, and that energy may not come at a significant environmental cost. And so, you have to look at the project, I think, in context. And so, in all these cases, I think the key is to have effective local leaders who are saying, one, we want to see the details, we want to make it open, we want public input, but also, not just saying, well, we don’t want any development, because, we’ve seen communities get some real benefit out of the investments from these data centers.
Ryan Clancy: Great. Go ahead.
What do data centers give back to local communities?
No Labels Member: My question was along the lines of the last one about data centers. It’s pretty much what you hear from neighbors and friends: “We don’t want a data center in our area.” You mentioned there were some real benefits to data centers. Could you expand on what the benefits are to the community? And why is it such a distributed model? Why couldn’t we build a few large data centers – say three in Pennsylvania, one each in the east, central, and west?
Daniel Castro: Yeah, well, I’ll take the second part of your question first. Some of the big companies are probably building somewhere between three and 10, and but then there are five companies that want to do that, so it starts adding up. And then, of course, there are a lot of smaller data centers. There’s the neoclouds – the smaller ones that, they’re going to have a smaller footprint, and they’re going to matter as well. A lot of the energy generation is happening, separately. So it’s behind the meter, they’re basically adding on new capacity. And so, they’re looking for sites that they can do that in, and there’s a lot of factors that come into play. Obviously, they’re looking for, maybe tax incentives, but they’re also looking for, again, where is the land cheapest? Where is the best energy based on their specific needs? So, there’s a lot of factors there. In terms of the community benefits, there are a few, and it depends, of course, on how you define the community. If you talk about, the immediate community, where it’s built, or the state, or, of course, the whole country, immediately, the biggest is, of course, the tax revenue. We see, we see incentives to bring the companies there, but that’s usually a reduction in the sales taxes that they’re going to be paying on the equipment they’re buying to put in place there. And so, there’s some reduction, but they’re still paying a lot of money. And they’re also usually paying taxes on the land. There will also be tax revenue from the construction, all the people working on it that are there. So there’s immediate financial benefits. If you look at Loudoun County in Virginia, that’s the one, about 38% of their general revenue is coming from the data centers. Now, not every community is going to be that big, but you have other, installations. There’s one in, I think it was Alabama, they’re getting $50,000 bonus checks for their teachers. Another place they’re exploring, basically, reducing property taxes on everyone, because of this. Now, where the benefits, and how the benefits manifest, of course, also depends on local leadership, it depends on some of the tax codes. Sometimes it’s going more to the state, and then the community has to say, “Wait a second. We’re the ones allowing the data center here, we want to make sure we get a cut of that.” So, a lot of it depends on, the specifics there, and that’s where it’s important, again, to have, local community leaders who can engage on that. A lot of these companies are also doing community benefit programs, so in addition to what they’re paying in taxes, they’re saying, give us your list of what you’d like, and let’s talk. What can we do to help you? Whether it’s, investments and workforce training, sometimes it’s as simple as, we’re going to build some firehouses and make other investments of what you need. A lot of times it’s also about improving the infrastructure. They need the roads to get to the data center that they’re building. But of course, they’re also improving the electric grid, they’re improving the water facilities, sometimes they’re also making significant investments in water processing plants, because they know they’ll be using water, but they’re providing much more than is needed, and that’s going back to the community. And then, there’s the resiliency. When these companies are building all this additional capacity, especially for the grid, that’s adding resilience to the network that benefits everyone. There’s also the jobs piece. There are permanent jobs. It’s not a job creation program, but there, of course, are jobs at these data centers. There are a lot of jobs in the construction part of it. And then, there’s a lot of jobs in making the equipment that’s actually going in the data centers, and so that’s not always, again, right there in that community, but these are American jobs, and that part is important as well.
Ryan Clancy: Go ahead.
Is AI being used to monitor other AI?
No Labels Member: I recently read Henry Kissinger and others’ book Genesis about AI. One of the things mentioned in the book was using AI to monitor AI.
Daniel Castro: Yep.
No Labels Member: Are we doing that at all yet?
Daniel Castro: Yes, absolutely. That’s definitely happening, especially with these AI agents where you have autonomous code programs running. They have what they call supervisory agents that monitor them. They have coordinating agents that, try to manage their tasks, and then, of course, they’re also doing a lot of research to see, well, has something gone wrong? And they’re often using AI for that. That’s because it’s generating a massive volume of code. The interesting challenge they’re running into is, of course, they have these concerns about, well, what if the AI agent cheats, or what if it knows it’s being monitored? So, it’s like the idea that when I’m driving and I see the sign that says, speed camera ahead, well, I better slow down. So, it’s the same principle with these AI agents. They may be behaving when they’re being watched, and so there are questions about whether we can create these sandboxes so that there’s a fully monitored environment, like a Truman Show for AI agents, where you can see everything that’s happening. And so there’s a lot of creativity around how to do this, and this is why I think we have to continue to build, because if you pause the research, you’re also pausing the research on how to make it safer. All of that comes hand in hand. I recently read part of a biography of the Wright brothers, and that was their whole point about how they designed the plane. They said one way to design a plane is by sitting in the house, and the analogy was to riding a horse. If you wanted to ride a horse, you could sit in the house and watch horses for a couple years, and then decide you know everything about them and jump on. Or you could just jump on and start figuring it out. And their point for how they built the plane was to just jump on and figure it out, that they had to figure out how to fly at the same time they were figuring out how to build an airplane. And I think that’s the same thing with this AI safety. We have to figure out those safety measures as we’re figuring out how to build them. You can’t really do one without the other.
Ryan Clancy: Thanks. Go ahead.
Can we program morals into AI?
No Labels Member: This may sound simplistic, but one idea I’ve heard is to program AI to love humans, or to program morals into it. How feasible or realistic is that as a possible safety measure?
Daniel Castro: Yeah, there’s a lot of interesting research on that. AI alignment is the whole field. How do you get the AI to do the things you want it to do, and behave as expected? And there are different theories. One is that you raise it like a child. So you try to ingrain a certain moral philosophy in it, over time, and you train it, but even with that model, there are a lot of questions that are raised. Do you teach a child that something is bad and tell them not to do it, or do you avoid exposing them to it so they don’t even know how to do it? And so, there’s that conflict that arises. The problem with AI and even the “love humans” concept or something like that is, it comes into conflict, of course, with other values or goals, or it becomes unclear: what does it mean to love humans? Do you prevent them from doing something harmful? Well, that means we give up free will. Those types of challenges that always arise. And so, we often think about, giving AI a task and expecting it to execute for us. It’s like if you have an assistant, or you ask a friend to do something for you. In some ways, you’re trusting them to do something. Over time, you develop trust based on whether they consistently follow through on what they say. Are they reliable? I think we’re going to see something similar with AI. We will likely develop trust for it over time based on continued performance and expectations being met. But in the short term, there are going to be limits and mistakes, and I think that’s where we have to be cautious about how we choose to use it, but that doesn’t mean we should be cautious about innovating the technology.
Ryan Clancy: Go ahead.
Could there be a “mutually assured destruction” system for AI?
No Labels Member: Also related to the earlier comment about Kissinger’s book: At the conclusion of that book, if I remember correctly, he suggested that what was going to be needed was the equivalent of a mutually assured destruction system for AI. We’re probably not even close to that now, but under what circumstances would that be possible?
Daniel Castro: Well, I’m not sure. One of the challenges, when people are talking about the risk of AI, there’s often an assumption that the more capable an AI system is, and we talk about it being powerful, but it’s really about capability. Being able to make better predictions and assessments, that doesn’t necessarily translate into tangible power, right? It doesn’t mean that you could have the smartest person or the smartest AI system in the world, and it’s probably not going to convince you to do something that you’re sure you don’t want to do, right? You still have free will. And we know that because, we’re in election season right now, right? There are tons of campaigns with really smart people trying to convince us how to vote. There are limits to that, and we know there are limits to that. That happens in, complex systems like stock markets and other systems, there’s an incentive to try and get something to move one way, but there are lots of incentives to try and get it to move a different way. And so, even with AI, even really powerful systems, there are limits to what it can do. And so, I don’t think it’s quite the same as the nuclear situation, where the system comes to a stable state because of the fact that, there are different forces, and we agree this is where we stop. I think with AI, it’s, just another part of a very complex ecosystem that’s continuously changing. And sometimes it’s going to skew a little one way, sometimes it’s going to skew a little another way, just like we see changes in human behavior and actions, fads and trends that come and go. I think that will be true with AI. But it’s not going to be about controlling. It’s going to be, I think, ideally about empowering individuals. With that empowerment, we’re gonna see individuals that want very different things, and that’s going to continue pushing in different directions, and hopefully over the long arc of history, that’s moving us in a direction that’s positive.
Ryan Clancy: Just two more questions. Go ahead.
Is my money safe as AI moves into banking?
No Labels Member: Daniel, thanks for being with us today. I was wondering about the implications of AI with respect to the banking industry, and specifically the advancing fintech industry. Are we safe? Will I wake up one day and my money will be gone?
Daniel Castro: That’s another great question. I think there are a few things there. We’re already seeing AI systems and agents that you can give access to your bank accounts and credit cards, to make purchases on your behalf. I think that can be reasonable. I think we’re also going to see incidents occur because of that. If you think about the early days of mobile payments, when people first had the iPhone and the App Store there, people gave their iPads to their kids, and then the kids realized that they could buy a golden apple or some other virtual item on the app, and they can run up all these expenses. So I think those are the things that are more likely to happen in the short term. Mistaken purchases – and we have systems in place generally for refunds of that type of activity, and sometimes the refund decision will be correct, and sometimes it won’t, and there might be some liability in lawsuits. But I think most of that will pan out. On the bigger question about fraudulent activity or attacks on people’s banking systems, again, I do think there are risks there, but I don’t think the risks are necessarily very different than what they are today. If you are banking with a trusted financial service provider, they’re stress testing their systems a lot. They do have access to some of these unreleased frontier models, and they should be using it specifically for that purpose of identifying vulnerabilities. But I know one of my banks still uses your voice as your passcode, and I was shocked by that, because we’ve had over a year and a half of being able to do voice cloning, and my voice is on the internet. I don’t want that. So I think there are still some big holes there, but again, over time, I think there are improvements to security that happen that AI can be part of even if it’s identifying patterns of activity that look suspicious and intervening.
Ryan Clancy: And final question.
Are AI company leaders right to be worried?
No Labels Member: Just to get back to some of these company leaders who have expressed concerns: For a layperson, we know the president came out and said there’s nothing to see here, and there are other people who have said maybe these people are overreacting. But it’s very unusual for people to be building something, putting so much money into it, and then all of a sudden saying, “Oh my goodness, what are we doing?” As Ryan alluded to, maybe somebody is just posturing to sideline or slow down competitors. But are there really legitimate concerns? People like yourself in the field – do you think they’re right to be concerned? For a lot of us, it’s a little unsettling.
Daniel Castro: Yeah, thank you for the question, and I’ll try and give a condensed answer to this. There are a lot of true believers who believe artificial superintelligence, smarter-than-human, highly capable AI is on the horizon. They believe they’re here to make it happen, or want to be here to make it happen, and it’s almost a religious conviction that this will occur, and that it will be the end of humanity as we know it. And that has nothing to do with any particular moment or what they’ve achieved in a lab. That’s just their fundamental worldview about the technology and what it’s capable of. Where human history will take us. It’s almost like believing in the Rapture or, some other event. They’ve built their companies around that goal, and I try to take people at their word on this. If you take them at their word on this, you get to the rationale for what they’re doing, because you arrive to a really important question, if they believe that this technology is going to destroy the world, why are they building it? And the reason they believe it, and still go to work every day and try to build it, is they think there is an endpoint to this race, that they can build an AI within their lifetime that is so capable it keeps self-improving, and whoever gets there first wins. The company that does that effectively owns everything. To the earlier point, it can create enormous wealth, it can control entire economies, basically, the current political systems will go away because we don’t need them. We just need these powerful companies, and whoever controls them controls the world. So they’re seeing this as: well, who better to do that than themselves? And also, better us than China, better a democracy than an authoritarian system. They don’t see it as the U.S. Winning versus China. It’s more about an authoritarian or controlling system versus them, because they’re not winning it for the United States; they’re winning it for this worldview. There are a lot of people who strongly believe this. If you look at the effective altruist movement, it’s been based on this idea that there are certain big things you can do to improve welfare, and one of the big things they want to do is create this safe AI that will prevent the disastrous AI. And so, that’s also part of it. Or, sometimes they’re not opposed to maybe an authoritarian Chinese AI, they’re just opposed to someone building an unsafe AI, and so they think if they build the safe AI fastest that we’ll get there. And so maybe their point of slowing it down is to say, make sure we can actually build safe AI before people build dangerous AI. So there’s a lot of different ways you might look at it, but that’s really their belief, and again, I think it’s always important to listen to all those views, but when you make such a monumental claim that the technology we’re building is going to kill us all, or potentially kill us all, you have to provide evidence for that. I think the burden of proof is on them to actually give tangible evidence of the risk and the harm, and a clear pathway of how that harm could be realized. And I think we’re still waiting on that evidence. I remain open-minded about it, but it’s like an alien invasion – within the realm of possibility. I don’t think it’s going to happen tomorrow. I don’t think we should design our space policy or our telecom policy around that potential threat.
Ryan Clancy: Daniel, thank you for that, and thank you for trying to bring some light to a conversation that is too often just heat. This was really illuminating. Thank you, Daniel.
What are AI companies actually afraid of when they talk about AI “going rogue”?
No Labels Member: May I ask one question? Just to understand: These are large language models, and AI is really running based on data. So the data that goes in drives it. When companies are worried about it going rogue, are they worried about an employee inside the company programming in some information to tell it to do something bad? It seems AI is driven by the information it receives. We were talking earlier about whether you teach it like a child and give it good information versus bad information, but it seems to come down to control of the data that’s put into it. If companies want to slow down, is that about slowing down what their employees are programming? I don’t understand that risk aspect of it.
Daniel Castro: Yeah, some of it is the data that goes into it, but as you said, there is the risk of a rogue employee. I think they’ve been trying to put more controls around that. They’re also concerned about IP theft, so they’re trying to lock down some of it from that perspective. And they’re worried about unauthorized access because unauthorized access is a monetary threat to them. But I think their primary concern is that they may think it’s doing something good when it isn’t. How do they make sure that doesn’t happen? This alignment problem is very hard to solve, because we’re talking about non-deterministic systems. So, with older systems, you gave them two options. You could do this, or you could do that, and so you knew what the results or potential results were. But with these large language models, the results can be much more diffuse, and you don’t always know what the result will be when you run the system. That’s the hard challenge here. These companies are going to continue to do different testing, and I think that’s where we can, again, make significant improvements, so that we know, okay, if we’re releasing this model, it’s met this threshold of testing that we think is best in class. I think that’s what we can do, again, in the short term, and over time, I think we’ll get better at doing that. But we also have to think about the idea that, well, let’s just put it under human control, and that will solve everything. Look at the Volkswagen emissions-cheating scandal. This was a company-wide effort to say, we’re going to figure out how to cheat the test. We had standards, we had human control, and they were still cheating. So, yeah, you’re right: what are we learning from? Well, we’re learning from humans, and humans do things wrong all the time. So, we’re really trying to design a system that acts like a human, except better. And that’s a good thing, but it’s difficult, and there’s a reality that it will never be perfect. And I’ll end here: There’s a good paper I was reading recently that talks about the emphasis on making sure we get one thing right, maybe accuracy, or confidence in that accuracy, but that means that if you’re only somewhat sure, you’re not giving an answer to something. For example, if you don’t have access to a doctor, and you need to use an LLM to get an answer to a medical question. One view of the world could be that, that chatbot can never give you an answer unless it’s 100% sure. But the alternative for you is that you get no healthcare, because maybe you don’t have access to a doctor, or you’re in a rural area, and you need immediate help. A lot of our laws and regulations that are so focused on safety are saying, well, it can only give you the answer if it’s 100% right. In some cases, that might be appropriate, but in other cases, you would rather have the answer that’s 90% right while knowing it might be 10% wrong, but you need an answer. That’s one of the tensions we see here as well.
Ryan Clancy: Thanks, Daniel, and thank you for that final question. So, Daniel Castro, president of the Information Technology and Innovation Foundation. Congratulations, Daniel, on your recent promotion, and thank you for sharing all these perspectives with us today.
Daniel Castro: Appreciate everyone’s time and great questions. Thank you.
Ryan Clancy: Thank you, Daniel. Thank you, everybody, for joining. Bye-bye.
