Proof of Concept: Opening Up the AI 'Black Box'

Transcript

Anna Delaney: Hello, this is Proof of Concept, a talk show where we invite security leaders to discuss the cybersecurity and privacy challenges of today and tomorrow, and how we can potentially solve them. We are your hosts. I'm Anna Delaney, director of productions here at ISMG.

Tom Field: I'm Tom Field, I'm senior vice president of editorial at ISMG. Anna, good to be back.

Delaney: Good to be back yet again for another episode on the huge topic that is AI.

Field: I believe we call this rounding up the usual suspects. We've got some friends here that we've been continuing this conversation for some weeks now.

Delaney: Yeah, excellent. And just to recap, on the last discussion that we had with them, we explored two crucial aspects of AI development and deployment, the vulnerabilities of AI models to potential threats and the challenge of biases in AI training data and model predictions. And this week, we're looking at the important themes of trusts and explainability in AI, with a focus on black box models. So these models, particularly in sensitive areas like healthcare, finance, pose challenges due to their lack of transparency, affecting both professionals' and the public's trust in AI decisions. So there's definitely a need to develop strategies for making AI accessible and trustworthy, particularly in these sensitive fields.

Field: As I thought about this topic in preparation for our conversation, reminded me of being back in school, I don't know if it was the same where you grew up. But when I was being taught arithmetic, the teachers' reminder was always, "show your work, show your work," - didn't matter how you got to the solution and the right one, show what you've done. Transparently and explainability is a lot like that. And it's something I don't hear being discussed enough by security and technology leaders when it comes to AI. The notion that yes, you do have to show how AI has helped you arrive at your decisions. And you've got to explain this process, increasingly, not just to external stakeholders, but to internal as well. So while everybody is out there talking about policies, and proofs of concepts - not our show - but their practices, I hope that they're thinking a lot more about transparency and explainability. And I bet our guests will help them to be asking the right questions.

Delaney: Absolutely. I like the analogy with maths there. And we're also looking at beyond explainability, looking for transparency in the AI development process, and the humans behind the AI models, and maybe we can debate the definition of transparency in AI later. But we're also here to ask to what extent can explainable AI enhance trust across critical sectors? And how can a broader approach to transparency address ethical concerns and improve AI development? So lots of deep questions.

Delaney: Why don't we bring on our guests, our guides, to help us through these complexities of AI. They are proof of concept regulators and AI experts Heather West, senior director of cybersecurity and privacy services at Venable LLP, and Sam Curry, CISO at Zscaler.

Field: Let's kick this off. Question I have for each of you. Heather, Sam, just teeing off what Anna said a few minutes ago, how do you ensure the AI models are understandable in critical scenarios? I'm thinking like those that affect human lives or vital systems in plenty of applicability there.

Sam Curry: Do you want to go first, Heather?

Heather West: I think it's useful to talk through a few terms here. Because you're right, AI feels like this black box. And you type in a question to the chatbot and it gives you an answer. And you can guess at why it gave you the answer it did. But there's a lot that's going on there that you don't necessarily understand. And so when you're working with any of these advanced AI models, you have an input, and you have an output, and then somewhere in the middle of really good stuff happens, you hope. But that's part of the point is how do we figure out and make sure that it's actually happening? And there's two or three different pieces here. One is interpretability. And that is can you understand why a decision or prediction or ranking or scoring or whatever the output of the system is, why it got to that conclusion? And that can be fuzzy. My favorite example is the first chatbot prompt I've ever put in which was asking it to write a poem about my dog. And I could see where it got the structure, I could see why it talked about the dog barking, but I didn't know how it worked. And I'm fine with that in a lot of contexts. In the same way that when I drive a car, I'm confident it's not going to blow up. It's been well maintained. It's from a good manufacturer, but I don't know how it works. Plenty of people do, but that's not my forte. And that success rate of I understand why the model is doing what it's doing at a high level is one piece of this. And then you dive a little bit deeper into explainable AI, which is a set of tools and frameworks that really help you dive in and think about how you interpret those predictions and where you might need to think extra hard about what might go wrong. And that's the tools that you're using for debugging and improving performance. And thinking about model behavior at a deeper level. Both of those are really useful pieces, and I think that it's important to think about transparency on a spectrum, because no end user needs perfect transparency. I don't think perfect transparency is useful in any way, shape or form. I can print out an algorithm and put it in front of you, it'd be a lot of paper, and it still wouldn't help you understand what's going on. And so we need different tools to think about what we want our models doing. And some of those are existing out there. And I think we're going to talk about some of them, and the ways that you can be confident in the AI that you're using.

Curry: I want to build on what Heather said. So the first thing is, there is no such thing as perfect transparency. And I fully agree with that. And the word "ensure" was in the question, which is difficult, because even in the case, or perhaps especially in the case of carbon-based decisions, when humans decide, we don't have that. So even now, humans are very poor witnesses. When it comes to things like legal cases, we're bad at understanding our own logic and rationale. Now, that doesn't mean that we should give up, by the way, when it comes to machines at all. We have an opportunity, in fact, to be far better with machines than we are with understanding how humans come to decisions. The first additional point that I would make is that not all AI is equal. And so we need to stop talking about AI as if it's one thing. In the case of something like machine learning, it is much more understandable. And this notion of a black box is very different when we're talking about generative AI and LLM or deep learning or other forms. But some of the technology in the AI toolkit we have understood how to use for a very long time, some of it is novel. And some of it we haven't even haven't even begun to fathom how to use. There are things we can look at output. And we can both manage the output and we can tune based on the output. But beware the outliers, beware when the input to the box is not what we expect, what will happen with the output. And this notion of traceability. So let's imagine version one of a technology comes along and it says, "Look what it can do. Let's rush into production," Slow down! And we say, well, we don't know why it made that decision. Okay, well, technology actually serves us. It's not like we suddenly discovered a new life form, and it can't be altered. You know, some folks will say, "I can't do that," well, you're not trying hard enough. There's both a science and understanding artificial intelligence as it emerges in the engineering or technology development, as we find new applications for it. And I think that, one thing we can do is in the next version, and in future versions, we can put requirements for traceability, and what it might mess up, how it performs for a version or two, but we can perfect and we can improve the ability to understand the inner workings of these black boxes through multiple versions. And we can say, when it is minimally viable from an internal tracing point of view, then it's okay to start considering it for primetime. We don't have to rush the first, second or third version into production. Or we might say that for some uses, it's okay and not for others. So this is about many different applications of different types of technology. And it might be okay, when making a decision on whether to do something like recommend something for purchase versus driving a car. Those are not the same thing. And it's not the same applications of technology. So we can change those over time. And I think we need to start tearing it apart like that, because we talk in far too many generalities here. And so blackbox is one thing, but it doesn't have to stay opaque.

Field: Sam, this conversation brings me back to when you and I on the road probably six or seven years ago, talking with security leaders about predictive AI. And you are fond of going up to a solution provider and saying how does this work? And they will say, "well, AI?" And you would say, "What about AI?" And the response was, "Well, AI." We're kind of back there.

Curry: I'm not interested in pulsating brains. Don't just tell me that it's a large data set that even you don't understand because now I have no confidence. I much prefer the application of technology, where you sell me on the output. It'll help me with anti-fraud. It'll help me with detecting malware. It'll help me with identifying variations in data for classification. That's interesting. I understand how to deal with that. I am not interested in you saying, "Just send me all your data and I'll figure out cool things about it." No, thanks. I'm good with that. And that's academic, by the way, that's interesting from an academic perspective, but not from a business applications.

Delaney: It's true. I just want to look at sensitive areas like healthcare, or finance, how do you recommend we manage that delicate balance between keeping information private, but also being transparent about how AI works from start to finish - the whole life cycle?

West: I'm laughing a little bit because I feel like Sam and I have this theme throughout all these conversations, you should do it the careful way. We know that's not as fun. But I'm going to say be doubly sure when you're talking about sensitive areas, but there's a flip side of that, too, when you're talking about healthcare, or finance or other decisions that really impact someone's life. And that is that we've been thinking harder about those decisions for a long time. And we have some of those tools. And that's interesting to me, how did we deal with this when it was a human? How do we think about bias when it was a human? How do we think about healthcare decisions being wrong when it was a human or a set of humans or a system? And how are we translating that into these situations where we're using a tool that might feel like a black box? I think that one of the things that the industry is doing - and I'm glad they are - is it being really careful about putting some of these harder-to-explain models into use in those contexts. But it goes to what Sam was saying, too, is they're looking at outputs, and they're starting to compare, if I have a human making this assessment, what does it look like, if I have an AI system making this assessment, and as that system gets more complicated, or as that system learns, how does those outputs change? And why do we like how they're changing, because there are examples of where these systems, I think we talked about them last time, are actually improving on human decision-making. They're more fair, they're more neutral, they're more clear. But all of that takes a lot of engineering and a lot of thinking to make sure the system is working the way that we want it to. And so on top of just being really careful about those more sensitive higher-risk situations, there's this concept of just making sure that you're continuously monitoring, and you're continuously thinking about what your system is doing, so that you have your antenna up, even though you're using this really cool AI system that you understand, and you're pretty confident and doesn't mean it's going to be perfect forever. And especially when you're thinking about healthcare, finance, or public safety, or any of these systems or Sam, you mentioned autonomous driving. I think all of those are situations where the bar is simply higher. And there's more diligence and more care that needs to be taken, more testing, more understanding of what the system is doing. And simply more time invested.

Curry: I agree with you that it's less fun. It's very fun to sit in an audience and have someone sell you on a vision that enchants you and makes you float away on what the art of the possible is. But it's also very often unrealistic. It's often Dunning-Kruger, you sort of sit there and go, "Okay, so that seems like it's right around the corner," but it's not. So the other thing is it's less scary when you do it this way. And potentially research may be faster. And it may be more practical, because if you put the boundaries on how information is shared and put to use, you'll find more people will share it, and you'll find that more people will have scrutiny, and the state of the art will advance faster. So this is the irony that when you put the boundaries in place, and you put what the guidelines are for how to advance the boxes, whether they're opaque boxes, or transparent boxes, how to verify the outputs, how to look at the transparencies, we start to build a whole new set of technology on it, and the APIs: who can plug into it, who can share it, and what are the business contracts you put around that? What are the federation models around data, you're no longer sitting in an audience listening to a visionary telling you the amazing things they're going to do and then find out that they have no ability to make it happen. And you don't feel like you were enchanted. And then it was disillusionment, right, you get this hype cycle where you go into a trough of disillusionment. Instead, the rate of advancement of actual practical applications of this stuff improves. And if we look to the model of, as Heather was saying, what do we do with human beings? If you were doing medical research previously, there were medical review boards, and they're not there, even though it may feel like it takes a long time to do medical research to slow things down. They actually established a means by which tens of thousands of researchers can do things like oncology research, and the data can be made available to them in a predictable way that patients don't feel bad about sharing their information on what it means for their families, or whether they can get insurance or medicine. And so that's much more interesting. Just because it is a more powerful silicon brain doesn't mean we should suddenly throw out all the rules and go, "Send all your data to us. And we'll figure it out later." No, in fact, it's the reverse. Doing this correctly opens up far more potential than sitting at a visionary show and hearing some entrepreneurs say, "Hey, I'm working to change the human experience," and then they've never surfaces. That's not interesting.

West: I think it's also useful to say that this transparency doesn't necessarily require transparency of the data itself. And so if I am using some health AI, transparency doesn't mean that everyone who looks at the information about how it works says, "Heather West has seasonal allergies." And it's DC, it's springtime. So it's a very good assumption. However, it doesn't need to be a part of that transparency, because it's not actually useful to the people who are looking at and testing this in most circumstances. And then in those circumstances, where you are doing the research and in situations like those medical review boards, then yes, there's protections on that that we should be pulling over into the AI world, as we are pulling over the technology into the same kinds of domains.

Field: I'm thinking about as this technology evolves quicker than anything we've ever seen before. Globally, we have a regulatory landscape that's shifting quickly as well. Maybe not as quick in the U.S. as elsewhere. But even so, how do we align our AI model so that they do comply with evolving regulations, particularly when it comes to transparency and explainability? Heather, I bet this is something you talk about every day.

West: I do talk with folks about this a lot. And it's a really good example of where your upfront investment and feeling like you're moving slow at the frontend, is going to enable you to move quickly later. Because these regulatory, these new rules are based in those existing rules that I was talking about. And so compliance is going to look very similar. And making sure that you have your governance programs in place looks very similar, with some tweaks for the things that are truly unique about these new tools. And part of that is making sure that you have a team or folks that you can talk to about the things you need to be thinking about and folks who can step up and say, "Wait a second, let's move slow on this piece until we really figure it out." Whether that's how it works, or how you can do compliance or maybe even future looking regulatory compliance. Thinking about here's how we can figure out a long-term sustainable strategy to feel good about these tools. Because I do think that one of the things that's true of regulation that's moving right now is it is not there to slow you down. It is not there to say you shouldn't use AI, or we're going to make it really hard to use AI. These legislators and regulators and standards bodies are thinking about these same questions and saying, "How do we create trust? How do we create confidence in these systems?" and we may or may not always love their approach for it. But the end goals are shared. And so I think that there's actually potentially room for really helpful collaboration partnership, but I do talk to companies a lot about how they can best position themselves for that kind of sustainable long-term compliance. Even if we don't know the final shape that a lot of this AI legislation and regulation is going to be because we kind of know what it's going to look like.

Curry: I think the most important thing to realize is that regulatory waves accelerate, when you look at at the wider context. So what happens is the first to regulate something, they say, "Well, if something needs to be done, if you look at what happened with FFIEC and then PCI, and then you look at what happened internationally around something as simple as logging in cybersecurity." And what happens is they tend to copy the same language. And they tend to have less forgiveness, and they tend to do it more quickly. And so what happens is the very first regulatory wave comes down, the next one is faster. And so you wind up with a uniformity over time, and they all tend to the same thing. So if you were to actually chart any given regulatory wave, you would see it accelerate and tend to a similar sort of language instead of principles. And that's a gross abstraction, that somebody out there is probably screaming "I can think of exceptions" and that's fine. But what we're seeing right now, globally, is a tendency to data sovereignty and you can't go outside our borders. And part of that is because data is a strategic asset in a geopolitical sense. Part of it is because rights of citizens and other entities like corporations are being thought of as national importance. And what you'll notice is the language that's being used, while not identical, is being copied among nations and other jurisdictions. And so I would encourage folks, as you look at this, realize that tendency is going to continue. So even if we are a little slower here in the United States, to start, that's not going to last. And we already see it with things like California and some other states that are leaning into it and perhaps a federal level soon, or more aggressively, as maybe the better way to put it. So don't take refuge from those laws in the jurisdictions that have yet to be legislated. Instead, lean in and make sure that you meet the requirements of the most aggressive ones. But you're thinking about how do we get the ability to leverage all of our data while maintaining the sovereignty and all the requirements of the most restrictive data regulation. Because imagine a world 10 years from now, where everyone has similar rules and requirements, regardless of the penalties or the specificity, the spirit will be the same. And the general requirements will be the same, even if it's different in the letter in each country, or perhaps it's evolving slightly slower by that point. So my advice is assume that this is going to be an ongoing trend. And pay attention to it, sure, but start thinking now about how you make your technology able to thrive in that world. Instead of saying, "It hasn't happened yet, in this place, we'll manage our data there." And, "We're not going to pay attention to it because there'll be new rules." You're not going to hide out in the last few places that don't have the strong regulations on data. That's my personal advice. So pay attention to this stuff, because it's a trend that's not going away.

West: 100% right. And actually, there are interesting data points around that convergence on language, because, you know, like anyone else, legislators and policymakers are influenceable. And they take the best ideas from different places, and they may customize them a bit. But you can see that even in the definition of AI, I think a year ago, definitions of AI were all over the place. And everyone's kind of converged on the OECD definition, which is very broad, but it is a starting point. And the fact that we all kind of converge on the same starting point. And that legislation around the world has that common reference is really helpful. But I will also 100%, don't try to hide from the regulation. It may feel like a good short-term strategy, but it's an incredibly problematic one long-term that will make it a lot harder.

Curry: And I have a suspicion as well, that all those staffers and people that write the language of the policy are probably using gen AI and LLM to help get the language and guess where that's pulled from. It's pulled from everybody else's language too. So guess what? It's going to look similar.

Delaney: This is fascinating stuff as always. Okay, so we've got that we might not need complete transparency in AI. But I think we agree we need better understanding or better grasp on how AI makes decisions. So what strategies can you both recommend to help stakeholders get a handle on these decisions?

Curry: So I have a recommendation, which is I think we should be embracing some of humanities a bit more, I think we should be thinking about, let's have a discussion with the departments, we don't normally talk about our emphasis on STEM has been crucial. But I think we should be talking to philosophy departments and ethics. I think it's a passion of mine, that we think about what our ethical frameworks are, and not ignore the conversations that we should be conscious about it. And we should be open about what those ethical frameworks are. And we should state them. And it's okay to disagree over them, because but it should be an open discussion. And we should say, what are the values that we are using? And I know some people out there who that's dangerous, because certainly in a lot of modern democracies with things like freedom of religion, we kind of avoid discussions of values. And it's dangerous. But I think if we approach it correctly, we'll say, let's have a difficult discussion about this. Because if we don't, the AIs are going to have baked in de facto values that aren't well reasoned, and that aren't something that we can question and examine and have discussion about. So I would encourage folks to do that openly now. And to this is not about the politics or left or right or anything like that. It's about actually having a discussion about how we want them to behave. How will the AI behave in absentia when we're not there to help them with the decision? So one part of it is accountability and being able to understand how they made a decision, but that before any of that we should be deciding ahead of time how we want them to behave. That's a different kind of discussion than just a technical one.

West: I love that. Starting there with your actual goals sitting down and saying, "How do I want to think about this?" and then turning that what is my goal with this system, and then you figure out how to measure it. And then you figure out where the data is. And you marched forward through this process, to decide whether or not your system, as you're implementing it, actually fulfills those goals, and actually is in line with your values and ethics and that you have these accountability measurements in place and metrics is really important. And I do think that folks who are rushing forward without taking those steps are really going to have a hard time with it.

Curry: I'd like to see more psychologists and sociologists, more philosophers, more awareness, especially as a lot of the arts are going to be impacted. These things are going to be writing poetry for us and music and graphic design. So I think we should be having a conversation with those people too.

Delaney: Well, I always love how much philosophy and arts and humanities comes into these conversations. Thank you so much, both of you. The wealth of knowledge and education you always share. So thank you.